Multi-mode garbage station garbage detection method and system

Abstract

The invention discloses a multi-mode garbage station garbage detection method and system, and the method comprises the following steps: S1, data collection: capturing real-time image data of a garbage station through an image sensor, collecting smell data of the garbage station through a smell sensor, and recording sound wave data of mosquito activities of the garbage station through a sound wave sensor; s2, data preprocessing: carrying out denoising and enhancement processing on the image data, carrying out standardization processing on the smell data, and carrying out spectrum analysis on the sound wave data; s3, feature extraction: carrying out image feature extraction on the image data through a residual network ResNet model, carrying out voice feature extraction on the sound wave data through a Whisper model, and carrying out odor analysis on the odor data through an MLP learning model to obtain odor concentration and components; and S4, feature fusion: carrying out weighted integration on the multi-modal features obtained in the step S3 through a self-attention mechanism by using a Transform model so as to realize multi-modal feature fusion, and generating a comprehensive judgment result. According to the invention, image, smell and mosquito sound wave information can be comprehensively utilized, and overflow and throwing conditions of the garbage station can be accurately judged.

Description

Technical Field

[0001] The present invention relates to the technical field of spam site garbage detection, and in particular to a multi-modal spam site garbage detection method and system. Background Art

[0002] As urbanization accelerates, urban waste management faces challenges such as inefficiency and environmental pollution. Traditional methods rely on manual inspections, which make it difficult to detect garbage overflow and spillage in a timely manner.

[0003] In patent CN202310257458.4, a camera is used to monitor the trash can in real time, capture the image of the trash can, and use an image processing algorithm to analyze the status of the trash can. However, this prior art has the following disadvantages: (1) Environmental dependence: image recognition is easily affected by environmental factors such as lighting, weather, and obstructions, resulting in inaccurate recognition results; (2) Single information source: relying only on visual information, it is impossible to judge the smell of garbage and mosquito activity, resulting in incomplete detection information.

[0004] In patent CN201911272971.0, volatile organic compounds (VOCs) and other odor components in the air are analyzed by odor sensors to determine the degree of corruption and overflow of garbage. However, this prior art has the following disadvantages: (1) Environmental dependence: The detection range of the odor sensor is limited and is easily affected by environmental factors such as wind direction and temperature, resulting in unstable detection results; (2) Single information source: Relying only on odor information, it is unable to provide the visual status of the garbage bin and the activity of mosquitoes, making it difficult to fully judge the actual situation of the garbage station.

[0005] In patent CN114711207A, an acoustic sensor is installed at a garbage station to monitor the acoustic signals of mosquito activities in real time to determine the status of the garbage. However, this prior art has the following disadvantages: (1) Environmental dependence: Acoustic wave detection is easily affected by environmental noise, such as vehicles, wind noise, etc., which may lead to false alarms or missed alarms; (2) Single information source: Relying only on acoustic wave information, it is impossible to fully judge the smell and visual status of the garbage, and the information is not comprehensive enough; (3) Simple frequency detection: The sound frequency of mosquitoes of different species and genders fluctuates greatly. Judging mosquito activities only by detecting a specific frequency threshold may not accurately distinguish different types of mosquito activities, resulting in inaccurate detection results. Summary of the invention

[0006] In response to the above technical problems, the present invention aims to provide a multimodal garbage detection method and system, which can comprehensively utilize images, odors and mosquito acoustic wave information to accurately determine the overflow and spillage conditions of garbage sites, and promptly notify the platform to request vehicles or personnel to handle them.

[0007] In order to achieve the above-mentioned purpose, the technical solution of the present invention provides a multimodal garbage site garbage detection method, which includes the following steps: S1 data acquisition: capturing real-time image data of the garbage site through an image sensor, collecting odor data of the garbage site through an odor sensor, and recording the acoustic wave data of mosquito activities at the garbage site through an acoustic wave sensor; S2 data preprocessing: denoising and enhancing the image data, standardizing the odor data, and performing spectral analysis on the acoustic wave data; S3 feature extraction: performing image feature extraction on the image data through the residual network ResNet model, performing speech feature extraction on the acoustic wave data through the Whisper model, and performing odor analysis on the odor data through the MLP learning model to obtain the odor concentration and composition; S4 feature fusion: using the Transformer model to perform weighted integration of the multimodal features obtained in step S3 through the self-attention mechanism to achieve multimodal feature fusion and generate a comprehensive judgment result.

[0008] Furthermore, in step S1, the multimodal data collected by the sensor are all accompanied by a timestamp, and in step S4, before multimodal feature fusion, the feature data of different modalities are time-synchronized by using a timestamp alignment technique to ensure that the analysis is performed within the same time window.

[0009] Furthermore, in step S4, the dimensions of the output results of the three models are (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ), N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dim wave 、Dim vis They are the latent space dimensions of the MLP learning model, the Whisper model, and the ResNet model, respectively. First, the Concat operator is used to merge along the second dimension to obtain the dimension (N, Dim VOC +Dim wave +Dim vis ); and then the Transformer model will merge the features to create a (N, Dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

[0010] Furthermore, in step S4, when making a comprehensive judgment, the multimodal feature fusion result output by the Transformer model is input into the sigmoid function: Thereby, the probability of garbage overflow is obtained, that is, the comprehensive judgment result.

[0011] Furthermore, in step S4, based on the comprehensive determination result, the state of the garbage site is determined, and when overflowing or spilling of garbage is detected, an alarm message is sent to the management platform via a wireless network to trigger the dispatch of vehicles or personnel.

[0012] The technical solution of the present invention also provides a multimodal garbage detection system for garbage sites, which includes the following modules: a data acquisition module, which is used to capture real-time image data of garbage sites through image sensors, collect odor data of garbage sites through odor sensors, and record acoustic wave data of mosquito activities at garbage sites through acoustic wave sensors; a data preprocessing module, which is used to perform denoising and enhancement processing on image data, standardize odor data, and perform spectrum analysis on acoustic wave data; a feature extraction module, which is used to perform image feature extraction on image data through a residual network ResNet model, perform speech feature extraction on acoustic wave data through a Whisper model, and perform odor analysis on odor data through an MLP learning model to obtain odor concentration and composition; a feature fusion module, which is used to use a Transformer model to perform weighted integration of the multimodal features obtained in step S3 through a self-attention mechanism to achieve multimodal feature fusion and generate a comprehensive judgment result.

[0013] Furthermore, in the data acquisition module, the multimodal data collected by the sensor are all accompanied by a timestamp, and in the feature fusion module, before multimodal feature fusion, the feature data of different modalities are time-synchronized through the timestamp alignment technology to ensure that the analysis is performed within the same time window.

[0014] Furthermore, in the feature fusion module, the dimensions of the output results of the three models are (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ), N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dim wave 、Dim vis They are the latent space dimensions of the MLP learning model, the Whisper model, and the ResNet model, respectively. First, the Concat operator is used to merge along the second dimension to obtain the dimension (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ); and then the Transformer model will merge the features to create a (N, Dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

[0015] Furthermore, in the feature fusion module, when making a comprehensive judgment, the multimodal feature fusion result output by the Transformer model is input into the sigmoid function: Thereby, the probability of garbage overflow is obtained, that is, the comprehensive judgment result.

[0016] Furthermore, in the feature fusion module, the status of the garbage site is judged based on the comprehensive judgment results, and when overflowing or spilling garbage is detected, an alarm message is sent to the management platform via the wireless network to trigger the dispatch of vehicles or personnel.

[0017] By combining three existing detection methods based on mosquito acoustic wave frequency, the present invention further improves the system's multimodal data fusion capability, overcomes the limitations of a single information source, and achieves more comprehensive and accurate garbage detection. By combining image and odor data, the present invention can effectively improve the robustness and adaptability of the system and reduce detection errors caused by frequency fluctuations and simple frequency detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a schematic diagram of extracting image features through ResNet (residual network) of the present invention;

[0020] Figure 2 is a schematic diagram of fusing multimodal features using a Transformer model in the present invention;

[0021] Figure 3 is a schematic diagram of time synchronization of multimodal data according to the present invention;

[0022] Figure 4 It is a flow chart of the multimodal garbage detection method of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The present invention proposes a multimodal garbage detection method and system, including an image sensor, an odor sensor and an acoustic sensor. Figure 4 As shown, the method of the present invention mainly adopts the following specific process:

[0025] (1) Data collection: The image sensor captures real-time images of the trash can, the odor sensor collects the odor concentration and composition of the garbage station, and the acoustic sensor records the sound of mosquito activities.

[0026] (2) Data preprocessing: denoising and enhancing image data, standardizing odor data, and performing spectral analysis on sound wave data.

[0027] (3) Feature extraction: Figure 1 As shown in the figure, the image data is extracted through ResNet (residual network). ResNet solves the gradient vanishing problem in deep networks by introducing residual modules, allowing the network to learn image features more deeply and effectively, thereby improving the accuracy of image recognition.

[0028] The acoustic data is feature extracted through Whisper technology to detect the frequency and intensity of mosquito activity. Whisper technology can effectively identify and classify different sound sources in the environment through fine-grained analysis of acoustic signals. Only the Encoder part of Whisper is used to extract and embed features of acoustic data.

[0029] The odor data collected by the odor sensor is analyzed through the MLP learning model to obtain the odor concentration and composition.

[0030] (4) Feature fusion: Figure 2 As shown in Figure 1, the Transformer model is used to fuse multimodal features. Transformer uses the self-attention mechanism to weightedly integrate data from different modalities, capturing the interactive relationship between modalities, thereby improving the accuracy of comprehensive judgment.

[0031] Among them, Figure 3 As shown in the figure, the timestamp synchronization technology is used to align the data of different modes in time. The Transformer model is used to perform feature fusion and generate comprehensive judgment results. All data collected by sensors are accompanied by timestamps to ensure the time consistency of the data. The timestamp alignment technology is used to synchronize the data of different modes in time to ensure that the analysis is performed in the same time window. In the fusion stage, the output results of the three models are merged, and their dimensions are respectively

[0032] (N,Dim VOC ),(N,Dim wave),(N,Dim vis ),

[0033] N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dim wave 、Dim vis They are the latent space dimensions of the MLP learning model, Whisper model, and ResNet model respectively.

[0034] After the Concat operator merges along the second dimension, its dimension is (N, Dim VOC +Dim wave +Dim vis ). Then, the merged features are used by Transformer to build a matrix consisting of (N, Dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

[0035] Decision and notification: The status of the garbage station is determined through comprehensive judgment results. If overflowing or spilling of garbage is detected, the system will send an alarm message to the management platform via the wireless network to trigger the dispatch of vehicles or personnel. The judgment method uses the sigmoid function

[0036]

[0037] The output result of the processing model is the output value of the Transformer model. After being processed by σ(x), the output value is the probability of garbage overflow.

[0038] Understandably, in the case of using only image data, as an image enhancement solution, the accuracy of image recognition can be improved by using a higher resolution camera or a more advanced image processing algorithm. In addition, the use of a multi-angle camera can be considered to reduce the occlusion problem.

[0039] If only odor sensors are relied upon, as an odor detection enhancement solution, a multi-sensor network can be used to cover a larger area, or more sensitive sensors can be used to improve the accuracy of odor detection. Machine learning models can also be used to perform more complex pattern recognition on odor data to enhance the judgment of different garbage states.

[0040] As an enhanced solution for sound wave detection, array microphones can be used to improve the directionality and sensitivity of the signal and reduce the interference of environmental noise. More complex signal processing algorithms can also be used.

[0041] In addition, it can be understood that in addition to images, smells and sound waves, the present invention can also consider introducing other types of data sources, such as temperature sensors, humidity sensors or infrared sensors, to provide more dimensional information for garbage status detection.

[0042] In specific implementations, if cost is a key factor, consider using low-cost sensors and edge computing devices to reduce the overall cost of the system without significantly reducing performance.

[0043] In addition, a distributed detection system can be developed to utilize the existing urban monitoring network or other public facilities to assist in the monitoring of garbage sites and reduce the installation and maintenance costs of independent systems.

[0044] In an embodiment of the present invention, a multimodal garbage site garbage detection method is provided, which includes the following steps: S1 data acquisition: capturing real-time image data of the garbage site through an image sensor, collecting odor data of the garbage site through an odor sensor, and recording the acoustic wave data of mosquito activities at the garbage site through an acoustic wave sensor; S2 data preprocessing: denoising and enhancing the image data, standardizing the odor data, and performing spectral analysis on the acoustic wave data; S3 feature extraction: performing image feature extraction on the image data through a residual network ResNet model, performing speech feature extraction on the acoustic wave data through a Whisper model, and performing odor analysis on the odor data through an MLP learning model to obtain odor concentration and composition; S4 feature fusion: using a Transformer model to perform weighted integration of the multimodal features obtained in step S3 through a self-attention mechanism to achieve multimodal feature fusion and generate a comprehensive judgment result.

[0045] Furthermore, in step S1, the multimodal data collected by the sensor are all accompanied by a timestamp, and in step S4, before multimodal feature fusion, the feature data of different modalities are time-synchronized by using a timestamp alignment technique to ensure that the analysis is performed within the same time window.

[0046] Furthermore, in step S4, the dimensions of the output results of the three models are (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ), N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dim wave 、Dim vis They are the latent space dimensions of the MLP learning model, the Whisper model, and the ResNet model, respectively. First, the Concat operator is used to merge along the second dimension to obtain the dimension (N, Dim VOC +Dimwave +Dim vis ); and then the Transformer model will merge the features to create a matrix consisting of (N, dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

[0047] Furthermore, in step S4, when making a comprehensive judgment, the multimodal feature fusion result output by the Transformer model is input into the sigmoid function: Thereby, the probability of garbage overflow is obtained, that is, the comprehensive judgment result.

[0048] Furthermore, in step S4, based on the comprehensive determination result, the state of the garbage site is determined, and when overflowing or spilling of garbage is detected, an alarm message is sent to the management platform via a wireless network to trigger the dispatch of vehicles or personnel.

[0049] In another embodiment of the present invention, a multimodal garbage site garbage detection system is also provided, which includes the following modules: a data acquisition module, which is used to capture real-time image data of the garbage site through an image sensor, collect odor data of the garbage site through an odor sensor, and record the acoustic wave data of mosquito activities at the garbage site through an acoustic wave sensor; a data preprocessing module, which is used to denoise and enhance the image data, standardize the odor data, and perform spectral analysis on the acoustic wave data; a feature extraction module, which is used to extract image features from the image data through a residual network ResNet model, extract voice features from the acoustic wave data through a Whisper model, and perform odor analysis on the odor data through an MLP learning model to obtain odor concentration and composition; a feature fusion module, which is used to use a Transformer model to perform weighted integration of the multimodal features obtained in step S3 through a self-attention mechanism to achieve multimodal feature fusion and generate a comprehensive judgment result.

[0050] Furthermore, in the data acquisition module, the multimodal data collected by the sensor are all accompanied by a timestamp, and in the feature fusion module, before multimodal feature fusion, the feature data of different modalities are time-synchronized through the timestamp alignment technology to ensure that the analysis is performed within the same time window.

[0051] Furthermore, in the feature fusion module, the dimensions of the output results of the three models are (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ), N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dimwave 、Dim vis They are the latent space dimensions of the MLP learning model, the Whisper model, and the ResNet model, respectively. First, the Concat operator is used to merge along the second dimension to obtain a dimension of (N, dim VOC +Dim wave +Dim vis ); and then the Transformer model will merge the features to create a matrix consisting of (N, dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

[0052] Furthermore, in the feature fusion module, when making a comprehensive judgment, the multimodal feature fusion result output by the Transformer model is input into the sigmoid function: Thereby, the probability of garbage overflow is obtained, that is, the comprehensive judgment result.

[0053] Furthermore, in the feature fusion module, the status of the garbage site is judged based on the comprehensive judgment results, and when overflowing or spilling garbage is detected, an alarm message is sent to the management platform via the wireless network to trigger the dispatch of vehicles or personnel.

[0054] By combining three existing detection methods based on mosquito acoustic wave frequency, the present invention further improves the system's multimodal data fusion capability, overcomes the limitations of a single information source, and achieves more comprehensive and accurate garbage detection. By combining image and odor data, the present invention can effectively improve the robustness and adaptability of the system and reduce detection errors caused by frequency fluctuations and simple frequency detection.

[0055] Beneficial technical effects of the technical solution of the present invention:

[0056] The present invention improves the accuracy and reliability of garbage detection through comprehensive analysis of multimodal data, and can promptly detect garbage overflow and spillage problems, thereby reducing environmental pollution and public health risks. Compared with single-modal detection methods, the present invention can more comprehensively obtain status information of garbage sites through cross-modal information fusion. The comprehensive analysis of images, odors, and sound waves enables the system to more accurately determine the degree of garbage spillage and reduce false alarm and missed alarm rates. At the same time, the system has a high degree of automation, which reduces the workload of manual inspections and improves the efficiency of garbage management. By introducing the Transformer model, the effect of multimodal data fusion is further improved, and the intelligence level of the system is enhanced.

[0057] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multimodal spam site spam detection method, characterized in that: The steps include: S1 Data Collection: Capture real-time image data of garbage sites through image sensors, collect odor data of garbage sites through odor sensors, and record acoustic wave data of mosquito activities at garbage sites through acoustic wave sensors; S2 data preprocessing: denoising and enhancing image data, standardizing odor data, and performing spectrum analysis on sound wave data; S3 feature extraction: Image features are extracted from image data through the residual network ResNet model, speech features are extracted from sound wave data through the Whisper model, and odor data is analyzed through the MLP learning model to obtain odor concentration and composition; S4 feature fusion: The Transformer model is used to perform weighted integration of the multimodal features obtained in step S3 through the self-attention mechanism to achieve multimodal feature fusion and generate a comprehensive judgment result.

2. The method according to claim 1, characterized in that In step S1, the multimodal data collected by the sensor are all accompanied by a timestamp, and in step S4, before multimodal feature fusion, the feature data of different modalities are time-synchronized by using the timestamp alignment technology to ensure that the analysis is performed within the same time window.

3. The method according to claim 2, characterized in that In step S4, the dimensions of the output results of the three models are (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ), N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dim wave 、Dim vis are the latent space dimensions of the MLP learning model, Whisper model, and ResNet model, respectively. First, the Concat operator is used to merge along the second dimension to obtain a dimension of (N, Dim VOC +Dim wave +Dim vis ); and then the Transformer model will merge the features to create a (N, Dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

4. The method according to claim 3, characterized in that In step S4, when making a comprehensive judgment, the multimodal feature fusion result output by the Transformer model is input into the sigmoid function: Thereby, the probability of garbage overflow is obtained, that is, the comprehensive judgment result.

5. The method according to claim 4, characterized in that In step S4, based on the comprehensive determination result, the state of the garbage site is determined, and when overflowing or spilling garbage is detected, an alarm message is sent to the management platform via a wireless network to trigger the dispatch of vehicles or personnel.

6. A multimodal spam site spam detection system, characterized in that: Includes the following modules: A data acquisition module is used to capture real-time image data of the garbage site through an image sensor, collect odor data of the garbage site through an odor sensor, and record sound wave data of mosquito activities at the garbage site through an sound wave sensor; Data preprocessing module, used to perform denoising and enhancement on image data, standardize odor data, and perform spectrum analysis on sound wave data; The feature extraction module is used to extract image features from image data through the residual network ResNet model, to extract speech features from sound wave data through the Whisper model, and to analyze odor data through the MLP learning model to obtain odor concentration and composition; The feature fusion module is used to use the Transformer model to perform weighted integration on the multimodal features obtained in step S3 through a self-attention mechanism to achieve multimodal feature fusion and generate a comprehensive judgment result.

7. The system according to claim 6, characterized in that In the data acquisition module, the multimodal data collected by the sensor are all accompanied by timestamps, and in the feature fusion module, before multimodal feature fusion, the feature data of different modalities are time-synchronized through timestamp alignment technology to ensure that the analysis is performed within the same time window.

8. The system according to claim 7, characterized in that In the feature fusion module, the dimensions of the output results of the three models are (N, Dim VOC ),(N,Dim wave ),(N,Dim vis ), N is the batch number, that is, the number of different garbage stations processed at one time, Dim voc 、Dim wave 、Dim vis are the latent space dimensions of the MLP learning model, Whisper model, and ResNet model, respectively. First, the Concat operator is used to merge along the second dimension to obtain a dimension of (N, Dim VOC +Dim wave +Dim vis ); and then the Transformer model will merge the features to create a (N, Dim VOC +Dim wave +Dim vis )→(N,), (N,) represents the dimension of a one-dimensional tensor.

9. The system according to claim 8, characterized in that In the feature fusion module, when making a comprehensive judgment, the multimodal feature fusion result output by the Transformer model is input into the sigmoid function: Thereby, the probability of garbage overflow is obtained, that is, the comprehensive judgment result.

10. The system according to claim 9, characterized in that In the feature fusion module, the status of the garbage station is judged based on the comprehensive judgment results, and when garbage overflow or spillage is detected, an alarm message is sent to the management platform via the wireless network to trigger the dispatch of vehicles or personnel.

Images