Intelligent garbage recognition system and method based on multi-modal fusion

Abstract

The invention relates to the technical field of garbage recognition, and particularly discloses an intelligent garbage recognition system and method based on multi-modal fusion, a hardware sensing unit is composed of a three-dimensional vision module, a touch sensing module and a near infrared spectrum module, and point cloud data, a pressure distribution matrix and spectrum data of garbage are obtained respectively; and an edge computing unit is equipped to carry an NPU acceleration chip for data processing. And the software module realizes multi-modal feature fusion by using ResNet-50, LSTM (Long Short Term Memory), 1D-CNN (Convolutional Neural Network) and Transform Encoder, and is also provided with a multi-sensor timestamp synchronous controller. And the dynamic feature database starts online learning to update the material feature library when the garbage recognition confidence coefficient is lower than 85%. During feature fusion, a cross-modal feature mapping relation is established, the weight is adjusted through an attention mechanism, and dynamic learning adopts a model lightweight method based on knowledge distillation. And starting an arbitration mechanism to correct the classification result when different modal identification results conflict. According to the method, the advantages of different modal data are fully utilized, and the recognition capability in a complex environment is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of garbage identification, and in particular to a garbage intelligent identification system and method based on multimodal fusion. Background Art

[0002] In today's society, waste sorting and processing are crucial for environmental protection and resource recycling. However, existing waste identification technologies have many drawbacks:

[0003] Single-vision recognition system: In complex scenarios such as lighting changes and occlusion, the misjudgment rate exceeds 40%. This is because single-vision recognition relies primarily on image information. When lighting conditions change, image features such as brightness and contrast will change, making it difficult for the recognition model to accurately extract features. When garbage is partially obscured, key recognition information is lost, further reducing recognition accuracy.

[0004] Limitations of Traditional Pressure Sensors: Traditional pressure sensors are used solely for weighing and are not integrated with material identification. Garbage is made of a wide variety of materials, each with significantly different physical properties. Weighing alone cannot capture effective information about the material and provide comprehensive data support for waste sorting.

[0005] Limitations of static feature databases: Static feature databases cannot adapt to new waste types. With the development of technology and changes in people's lifestyles, new types of waste continue to emerge. Static databases are difficult to update in real time, resulting in the recognition system being unable to accurately judge new types of waste, affecting the efficiency and accuracy of waste classification.

[0006] Therefore, an intelligent garbage identification system and method based on multimodal fusion has become an urgent problem to be solved. Summary of the Invention

[0007] The purpose of the present invention is to provide a garbage intelligent identification system and method based on multimodal fusion to solve the problems of accurate identification of multi-material mixed garbage in complex environments, rapid feature learning of adaptive new types of garbage, and temporal alignment and conflict resolution of multimodal sensor data.

[0008] To achieve the above objectives, the present invention provides a technical solution: a garbage intelligent identification system based on multimodal fusion, comprising:

[0009] Hardware sensing unit, used to obtain point cloud data, pressure distribution matrix and spectral data of garbage;

[0010] A software module used in the multimodal feature extraction layer to extract visual features using ResNet-50, process tactile pressure time series data using LSTM, process spectral features using 1D-CNN, and then perform feature fusion using the Transformer Encoder.

[0011] The dynamic feature database is used to start online learning when the confidence level of garbage identification is lower than 85%. It continuously updates the material feature library by extracting contour features, adding them to the material feature library, and retraining the classifier.

[0012] Furthermore, the hardware sensing unit includes a three-dimensional vision module, a tactile perception module, and a near-infrared spectroscopy module. The three-dimensional vision module uses a dynamic range 3D camera and is equipped with 940nm infrared fill light; the tactile perception module uses an array tactile sensor with a resolution of 0.1N; the near-infrared spectroscopy module uses a near-infrared spectrometer with a wavelength range of 900-1700nm.

[0013] Furthermore, the tactile perception module adopts an array layout to obtain a pressure distribution matrix when contacting garbage. The pressure distribution matrix is ​​used for subsequent tactile feature extraction to assist in determining the material of the garbage.

[0014] Furthermore, the hardware sensing unit also includes an edge computing unit equipped with an NPU acceleration chip for processing data collected by the three-dimensional vision module, tactile perception module, and near-infrared spectroscopy module.

[0015] Furthermore, the software module also includes a multi-sensor timestamp synchronization controller for achieving timestamp synchronization of multi-sensor data.

[0016] Furthermore, when performing feature fusion, a cross-modal feature mapping relationship Φ = α·V + β·T + γ·S is established; where V, T, and S are the visual, tactile, and spectral feature weights, respectively, α+β+γ = 1, and the weight coefficient is dynamically adjusted through the attention mechanism.

[0017] The present invention also provides a method for intelligent garbage identification based on multimodal fusion, comprising the following steps:

[0018] S1. Data acquisition: Trigger the 3D camera in the 3D vision module to acquire point cloud data, which contains RGB-D information. When contacting garbage, the tactile perception module collects the pressure distribution matrix. Simultaneously, the near-infrared spectrum module is activated to perform spectral scanning with a scanning cycle of 50ms.

[0019] S2. Feature Fusion: Based on the collected data, visual, tactile, and spectral features are extracted in the multimodal feature extraction layer, and then fused through the Transformer Encoder. A cross-modal feature mapping relationship Φ = α·V + β·T + γ·S is established. Where V, T, and S are the weights of visual, tactile, and spectral features, respectively, and α + β + γ = 1. The weight coefficients are dynamically adjusted using the attention mechanism.

[0020] S3. Dynamic Learning: When the confidence level of garbage identification is lower than 85%, online learning is initiated. Specifically, the system extracts garbage contour features, adds the new features to the material feature library, and then retrains the classifier. A knowledge distillation-based model lightweighting method is used to optimize model performance while ensuring recognition accuracy.

[0021] Furthermore, during the spam identification process, when conflicts arise in the identification results of different modalities, an arbitration mechanism is initiated to modify the classification results based on the arbitration results.

[0022] The advantages of the present invention over the existing technology are: by integrating multimodal information such as vision, touch and spectrum, the present invention fully utilizes the advantages of different modal data, reduces the impact of factors such as lighting changes and occlusion on the recognition results, and improves the system's recognition ability in complex environments.

[0023] The present invention enables the system to quickly learn the characteristics of newly emerging garbage categories through a dynamic incremental learning module, timely update the material feature library, and enhance the adaptability and scalability of the system.

[0024] The NPU acceleration chip and optimized algorithm architecture carried by the edge computing unit of the present invention ensure that the system can quickly process the collected data and meet the time requirements of real-time garbage sorting. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a system block diagram of a garbage intelligent identification system based on multimodal fusion in the present invention.

[0026] Figure 2 This is a flow chart of a garbage intelligent identification method based on multimodal fusion of the present invention. DETAILED DESCRIPTION

[0027] Various exemplary embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangements of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0028] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0029] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0030] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0031] The following is a further detailed description of a garbage intelligent identification system and method based on multimodal fusion according to the present invention with reference to the accompanying drawings.

[0032] Combined with attachment Figure 1-2 , the present invention is introduced in detail.

[0033] A garbage intelligent identification system based on multimodal fusion, comprising:

[0034] Hardware modules:

[0035] High Dynamic Range 3D Camera (with 940nm infrared fill light): Capable of capturing point cloud data of garbage, including RGB-D information. RGB information provides the garbage's color characteristics, while D (depth) information reflects its shape and spatial position. The 940nm infrared fill light function ensures the camera captures clear image data even in low-light environments, improving the system's adaptability to varying lighting conditions.

[0036] Array tactile sensors (0.1N resolution, covering the end of the robotic arm): This sensor uses a high resolution of 0.1N to detect the pressure distribution of the robotic arm when it contacts the garbage, generating a pressure distribution matrix. By analyzing the differences in pressure distribution, the hardness and other characteristics of different parts of the garbage can be determined, providing a powerful basis for material identification.

[0037] Near-infrared spectrometer (wavelength range 900-1700nm): This scans the garbage spectrum and acquires spectral data. Different garbage materials have unique absorption and reflection characteristics within the near-infrared spectrum. By analyzing the spectral curve, the material composition of the garbage can be accurately identified.

[0038] Edge Computing Unit (equipped with an NPU accelerator chip): Responsible for real-time processing of data collected by the 3D vision module, tactile perception module, and near-infrared spectroscopy module. The NPU accelerator chip has powerful parallel computing capabilities and can quickly process large amounts of data, meeting the system's real-time sorting requirements for multimodal data fusion with a latency of less than 300ms.

[0039] Software Architecture:

[0040] Multimodal feature extraction layer: This layer utilizes multiple deep learning models for feature extraction. Visual features are extracted using the ResNet-50 model, which has excellent performance in image feature extraction and can effectively capture spatial features such as the shape and color of garbage. Tactile pressure time series data is processed using an LSTM (Long Short-Term Memory) network, which effectively handles long-term dependencies in time series, thereby extracting key information from pressure changes. Spectral curves are processed using a 1D-CNN (one-dimensional convolutional neural network), which can automatically extract local features from spectral data. Finally, feature fusion is performed using the TransformerEncoder, fully integrating features from different modalities to improve recognition accuracy.

[0041] Dynamic Incremental Learning Module (Continuously Updates the Material Feature Library): When the confidence level in waste identification falls below 85%, the system initiates online learning. It first extracts the waste's contour features, then adds these new features to the material feature library and retrains the classifier. This allows the system to continuously learn the characteristics of newly emerging waste categories, enabling adaptive learning for these new categories. This adaptive learning time is reduced to 15 minutes, compared to over 72 hours with traditional methods, significantly improving learning efficiency.

[0042] Multi-sensor timestamp synchronization controller (μs-level accuracy): realizes timestamp synchronization of multi-sensor data, ensures the temporal consistency of data of different modalities, avoids data fusion errors caused by time differences, and ensures the accuracy and stability of the system.

[0043] Based on the above system, the present invention also provides a method for intelligent garbage identification based on multimodal fusion, comprising the following steps:

[0044] During the data acquisition phase, the 3D camera is triggered to acquire point cloud data (including RGB-D information), providing a foundation for subsequent visual feature extraction. When the robotic arm contacts the garbage, the tactile sensor collects the pressure distribution matrix and acquires tactile information. Simultaneously, a near-infrared spectral scanner is activated with a 50ms scan cycle to acquire spectral data. This multi-sensor synchronized acquisition method enables comprehensive, multimodal information on the garbage.

[0045] Feature fusion stage:

[0046] A cross-modal feature mapping relationship Φ = α·V + β·T + γ·S is established; where V, T, and S are the weights of visual, tactile, and spectral features, respectively, and α+β+γ = 1. In this way, the features of different modalities are organically combined.

[0047] The attention mechanism is used to dynamically adjust the weight coefficient. The attention mechanism can automatically assign weights based on the importance of different data, so that more critical features can receive greater weights during the feature fusion process, thereby improving the quality of the fused features and enhancing recognition accuracy.

[0048] Dynamic Learning Phase: Online learning begins when the confidence level of garbage identification falls below 85%. This involves extracting garbage profile features, adding these features to the material feature library, and then retraining the classifier, enabling the system to continuously adapt to emerging garbage categories. Furthermore, a model lightweighting method based on knowledge distillation (Teacher→Student network) is employed to optimize model performance while maintaining recognition accuracy, reducing computational workload and storage requirements, and improving system efficiency.

[0049] The specific implementation process of the garbage intelligent identification system and method based on multimodal fusion of the present invention is as follows:

[0050] Take a resident delivering a takeaway lunch box (including a plastic box, metal lid, and food residue) as an example:

[0051] Data collection: The 3D camera identifies the general outline of the takeout box, measuring its length, width, and height to 200×150×80mm, capturing its shape and dimensions. When the robotic arm grasps the box, the tactile sensor detects a local stiffness of 0.5N / mm in the plastic area. 2 , the metal area is 2.3N / mm 2 The material was preliminarily determined by the pressure difference; spectral analysis found that the reflectivity of the metal cover dropped sharply at 1350nm. This characteristic peak matched that of aluminum, further confirming the material of the metal cover.

[0052] Identification and Decision-Making: The system initially identifies the waste as "recyclables - metal," but this result may conflict with the visual recognition result. At this point, an arbitration mechanism is activated to correct the classification result based on more discriminative features, such as spectral characteristics, to ensure classification accuracy. In practical applications, this multimodal fusion approach can effectively improve the efficiency and accuracy of waste sorting.

[0053] The present invention and its embodiments are described above. Such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by the above, and does not deviate from the purpose of the present invention, without inventive design, a structure and embodiment similar to the technical solution should fall within the scope of protection of the present invention.

Claims

1. A garbage intelligent identification system based on multimodal fusion, characterized by: include: Hardware sensing unit, used to obtain point cloud data, pressure distribution matrix and spectral data of garbage; A software module used in the multimodal feature extraction layer to extract visual features using ResNet-50, process tactile pressure time series data using LSTM, process spectral features using 1D-CNN, and then perform feature fusion using the Transformer Encoder. The dynamic feature database is used to start online learning when the confidence level of garbage identification is lower than 85%. It continuously updates the material feature library by extracting contour features, adding them to the material feature library, and retraining the classifier.

2. The intelligent garbage identification system based on multimodal fusion according to claim 1 is characterized by: The hardware sensing unit includes a three-dimensional vision module, a tactile perception module, and a near-infrared spectrum module. The three-dimensional vision module uses a dynamic range 3D camera equipped with 940nm infrared fill light; the tactile perception module uses an array tactile sensor with a resolution of 0.1N; the near-infrared spectrum module uses a near-infrared spectrometer with a wavelength range of 900-1700nm.

3. The intelligent garbage identification system based on multimodal fusion according to claim 2 is characterized by: The tactile perception module adopts an array layout to obtain a pressure distribution matrix when contacting garbage. The pressure distribution matrix is ​​used for subsequent tactile feature extraction to assist in determining the material of the garbage.

4. The intelligent garbage identification system based on multimodal fusion according to claim 3 is characterized by: The hardware sensing unit also includes an edge computing unit equipped with an NPU acceleration chip for processing data collected by the three-dimensional vision module, the tactile perception module, and the near-infrared spectroscopy module.

5. The intelligent garbage identification system based on multimodal fusion according to claim 4 is characterized by: The software module also includes a multi-sensor timestamp synchronization controller for achieving timestamp synchronization of multi-sensor data.

6. The intelligent garbage identification system based on multimodal fusion according to claim 5 is characterized by: When performing feature fusion, a cross-modal feature mapping relationship Φ = α·V + β·T + γ·S is established; where V, T, and S are the weights of visual, tactile, and spectral features, respectively, α+β+γ = 1, and the weight coefficient is dynamically adjusted through the attention mechanism.

7. A method for intelligent garbage identification based on multimodal fusion, implemented based on the system according to any one of claims 1 to 6, characterized in that: The following steps are involved: S1. Data acquisition: Trigger the 3D camera in the 3D vision module to acquire point cloud data, which contains RGB-D information. When contacting garbage, the tactile perception module collects the pressure distribution matrix. Simultaneously, the near-infrared spectrum module is activated to perform spectral scanning with a scanning cycle of 50ms. S2. Feature Fusion: Based on the collected data, visual, tactile, and spectral features are extracted in the multimodal feature extraction layer, and then fused through the Transformer Encoder. A cross-modal feature mapping relationship Φ = α·V + β·T + γ·S is established. Where V, T, and S are the weights of visual, tactile, and spectral features, respectively, and α + β + γ = 1. The weight coefficients are dynamically adjusted using the attention mechanism. S3. Dynamic Learning: When the confidence level of garbage identification is lower than 85%, online learning is initiated. Specifically, the system extracts garbage contour features, adds the new features to the material feature library, and then retrains the classifier. A knowledge distillation-based model lightweighting method is used to optimize model performance while ensuring recognition accuracy.

8. The intelligent garbage identification system based on multimodal fusion according to claim 7 is characterized by: During the garbage identification process, when the identification results of different modalities conflict, the arbitration mechanism is activated and the classification results are corrected based on the arbitration results.

Images