Multi-modal model city solid waste high-precision extraction method based on multi-source data

Abstract

The application discloses a kind of multi-modal model city solid waste high-precision extraction methods based on multi-source data, including collecting city solid waste multi-modal data, including city solid waste remote sensing optical image, remote sensing spectral data and surrounding ground object data, the data collected are preprocessed;Based on the data after pre-processing, construct sample library, train optical recognition model, spectral recognition model and ground object recognition model respectively through sample library, the optical recognition model after training is used for the preliminary screening of solid waste area, obtains the positioning and classification result of solid waste point, the spectral recognition model after training is used for further screening to the solid waste point screened by optical recognition model, the ground object information around the solid waste point screened by optical recognition model and spectral recognition model is input into ground object recognition model, and ground object recognition model outputs solid waste category;The application is suitable for city solid waste monitoring, environmental protection and land use planning, and is suitable for solid waste information extraction in complex urban environment.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of ecological environment protection, in particular to a multi-modal model urban solid waste high-precision extraction method based on multi-source data. BACKGROUND

[0002] With the continuous advancement of urbanization, the generation and accumulation of urban solid waste have become one of the main sources of environmental pollution. Traditional solid waste identification methods rely on manual investigation and on-site measurement, which are labor-intensive, low-efficiency, and limited by human resources and time costs. Remote sensing technology, as a non-contact ground observation method, can obtain large-scale, high-resolution image data through satellite, unmanned aerial vehicle and other platforms, and is an important technical means to solve the problem of urban solid waste monitoring.

[0003] At present, the solid waste identification method based on remote sensing image mainly relies on a single data source, such as optical remote sensing image or spectral data, but these methods have certain limitations in complex urban environments and are difficult to cope with different scales and diversified solid waste types. Therefore, how to combine multiple data sources and effectively integrate their advantages to improve the precision and robustness of solid waste extraction has become a hot issue in the research of remote sensing solid waste identification technology. SUMMARY

[0004] The application aims to provide a multi-modal model urban solid waste high-precision extraction method based on multi-source data, which combines remote sensing optical image, remote sensing spectral data and surrounding feature data, and uses deep learning and machine learning algorithms to jointly train multiple models to efficiently and accurately extract urban solid waste areas.

[0005] To achieve the above purpose, the technical scheme of the application is as follows:

[0006] The multi-modal model urban solid waste high-precision extraction method based on multi-source data comprises:

[0007] Data collection and preprocessing, collecting multi-modal data containing urban solid waste, including remote sensing optical image, remote sensing spectral data and surrounding feature data of urban solid waste, and preprocessing the collected data;

[0008] Constructing a sample library based on the preprocessed data, comprising:

[0009] Labeling the preprocessed remote sensing optical image of the solid waste area, and labeling the remote sensing image of the solid waste area as a solid waste area category;

[0010] Classifying the preprocessed remote sensing spectral data, and labeling the remote sensing spectral data as a solid waste area category;

[0011] Extracting the feature of the preprocessed feature data, and labeling the feature data as a solid waste category;

[0012] Model building and training, including:

[0013] Optical recognition model construction and training: remote sensing optical images with solid waste area categories marked are input into the constructed optical recognition model for model training. The trained optical recognition model is used for preliminary screening of solid waste area categories to obtain the positioning and classification results of solid waste points;

[0014] Spectral recognition model construction and training: inputting remote sensing spectral data with labeled solid waste area categories into the constructed spectral recognition model for model training. The trained spectral recognition model is used to further screen the solid waste points screened by the optical recognition model to determine the accuracy of the solid waste point classification results determined by the optical recognition model;

[0015] Building and training a feature recognition model: Different solid waste categories correspond to corresponding surrounding feature data. The feature data labeled with solid waste categories are input into the constructed feature recognition model for model training. The trained feature recognition model is used to determine the solid waste categories of the solid waste points screened by the spectral recognition model.

[0016] Solid waste category extraction and result generation: the ground feature information around the solid waste points screened by the optical recognition model and the spectral recognition model is input into the trained ground feature recognition model. The trained ground feature recognition model outputs the solid waste category based on the ground feature information around the solid waste points.

[0017] Furthermore, the pre-processing of the collected data includes:

[0018] Atmospheric correction: perform atmospheric correction processing on remote sensing images and spectral data to eliminate the influence of the atmosphere on remote sensing data;

[0019] Geometric correction: correct image distortion and projection errors to make the image coordinates consistent with the geographic coordinate system;

[0020] Processing of surrounding feature data, unified processing of surrounding feature data and coordinates;

[0021] Multimodal data registration uses image registration technology to spatially register optical images, spectral data, and ground feature information, so that data from different modalities are in the same coordinate system.

[0022] Furthermore, the construction and training of the optical recognition model includes using the YOLOv8 model, inputting the remote sensing optical image with the solid waste area category marked into the YOLOv8 model for training, and the trained YOLOv8 model is used to preliminarily determine the solid waste area category corresponding to the remote sensing optical image, and obtain a solid waste point list, which includes the location coordinates and solid waste category of the solid waste point.

[0023] Further, the trained YOLOv8 model comprises a local feature extractor ƒ loc (*), a surrounding context extractor ƒ sur (*), a joint feature extractor ƒ joi (*) and a global context extractor ƒ glo (*) ; the local feature extractor is used to extract features from a local area of the solid waste landfill site, and after extracting features, the remote sensing optical image has a local area receptive field; the surrounding context extractor is used to expand the receptive field to capture the surrounding context information of the site; the joint feature extractor combines the local features and the surrounding context features; the global context extractor is used to aggregate the global information of the entire input site image, and combines the global information with the joint features;

[0024] The local feature extractor uses a 3x3 convolution layer to learn local features, and processes eight adjacent feature vectors;

[0025] The surrounding context extractor uses a 3x3 dilated convolution layer to capture larger surrounding context;

[0026] The joint feature extractor ƒ joi (*) combines the output features of ƒ loc (*) and ƒ sur (*), and performs batch normalization and parameterization after the connection operation;

[0027] The global context extractor ƒ glo (*) extracts global context, and ƒ glo (*) aggregates the global context of the image through a global average pooling layer.

[0028] Further, the spectral recognition model construction and training comprises using a random forest model, inputting the labeled solid waste region category remote sensing spectral data into the constructed random forest model, and the trained random forest model judging whether the classification result of the solid waste point determined by the optical recognition model is accurate according to the spectral features.

[0029] Further, the random forest model determines whether the classification result of the solid waste point determined by the optical recognition model is accurate according to the spectral features, including: after the YOLOv8 model is trained, the solid waste area type corresponding to the remote sensing optical image is determined, the solid waste point is obtained, the remote sensing spectral data of the area where the solid waste point is located is input into the trained random forest model, the random forest model determines the solid waste area type corresponding to the remote sensing spectral data according to the spectral features, if the solid waste area type determined by the random forest model is consistent with the classification result of the solid waste point obtained by the YOLOv8 model, the solid waste area type determined by the optical recognition model is accurate, and the solid waste point is screened by the YOLOv8 model and the random forest model; if the solid waste area type determined by the random forest model is inconsistent with the classification result of the solid waste point determined by the YOLOv8 model, the solid waste point determined by the optical recognition model is inaccurate, and the solid waste point obtained by the optical recognition model is not treated as a solid waste area.

[0030] Further, the ground object recognition model construction and training includes using a TPOT model, inputting the ground object data labeled with the solid waste category into the constructed TPOT model for training, and the trained TPOT model is used to determine the solid waste category of the solid waste point screened by the spectral recognition model.

[0031] Further, the TPOT model used to determine the solid waste category of the solid waste point screened by the spectral recognition model includes: inputting the ground object information around the solid waste point screened by the YOLOv8 model and the random forest model into the trained TPOT model, and the TPOT model determines the solid waste category according to the ground object information features around the solid waste point.

[0032] Further, the trained TPOT model includes feature selection, feature preprocessing and feature reconstruction, the feature selection is to input the site location information and the predicted site category of the solid waste point obtained by the spectral recognition model as a part of the input, and collect a series of ground object information data around the solid waste landfill site as another part of the input; the feature preprocessing includes missing value processing, filling missing data or deleting samples with too many missing values; data standardization uniformly converts the feature values to a preset range, and data coding converts certain classification features into a digital expression coding format; the feature reconstruction includes original feature combination, conversion or extension, and generates features expressing the characteristics of the solid waste landfill site.

[0033] A city solid waste high-precision extraction system based on multi-modal data, the system is used to realize the city solid waste high-precision extraction method based on multi-source data multi-modal model.

[0034] A computing device includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the multi-source data-based multi-modal model city solid waste high-precision extraction method.

[0035] The present application has the following advantages:

[0036] The present application is suitable for city solid waste monitoring, environmental protection, land use planning and other fields, and can effectively support solid waste extraction tasks in complex urban environments, and the specific advantages include:

[0037] 1. High-precision solid waste extraction: The present application combines optical remote sensing images, spectral data and surrounding ground object information, and through multi-modal data fusion, it can perform high-precision solid waste positioning and classification on multiple data scales, especially suitable for solid waste identification tasks in complex urban environments.

[0038] 2. Enhanced adaptability: The data enhancement method of the present application enables the model to adapt to different scenes, perspectives and scales, enhancing the generalization ability of the solid waste extraction model.

[0039] 3. Efficient identification model: The optical, spectral and ground object identification model based on YOLOv8, random forest and TPOT method of the present application can efficiently and accurately extract solid waste areas and improve the precision of screening.

[0040] 4. Joint training optimization: All models of the present application are based on deep learning and machine learning algorithms for joint training and optimization, using an end-to-end training method to ensure smooth information flow between steps, ultimately achieving optimal solid waste extraction results. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1 The present application is a city solid waste high-precision extraction method flowchart.

[0042] Figure 2 The present application is a Context Guided Block schematic diagram.

[0043] Figure 3 The present application is a Low-GD detail diagram.

[0044] Figure 4 The present application is a High-GD detail diagram. DETAILED DESCRIPTION

[0045] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with the drawings and examples. It should be understood that the specific examples described herein are part of the embodiments of the present application, but not all the embodiments, and are not intended to limit the present application. Based on the examples in the present application, all other examples obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0046] The present application provides a multi-modal model city solid waste high-precision extraction method based on multi-source data. By combining remote sensing optical images, remote sensing spectral data and surrounding feature data, multiple models are jointly trained using deep learning and machine learning algorithms, which can efficiently and accurately extract urban solid waste areas. The method comprises the following steps:

[0047] Step a: data collection and preprocessing

[0048] Data collection: collect multi-modal data containing urban solid waste, including urban solid waste remote sensing optical images such as Sentinel-2, WorldView, etc., urban solid waste remote sensing spectral data such as spectral band data such as NDVI, NDSI, etc., and urban solid waste surrounding feature data such as urban infrastructure, land use data, etc. These data sources can include satellite remote sensing, unmanned aerial vehicle images, ground monitoring stations, etc.

[0049] Data preprocessing:

[0050] Atmospheric correction: atmospheric correction is performed on remote sensing images and spectral data to eliminate the influence of the atmosphere on remote sensing data. Common atmospheric correction models such as 6S model, Flaash model are used for radiation calibration to ensure the physical consistency of the data.

[0051] Geometric correction: geometric correction is performed to ensure the spatial accuracy of remote sensing images, especially to correct image distortion, projection error, etc., so that the image coordinates are consistent with the geographic coordinate system.

[0052] Surrounding feature data processing: data cleaning and coordinate unification processing are performed on surrounding feature data such as urban infrastructure, land use division, etc. to ensure that the surrounding data correspond to each other.

[0053] Multi-modal data registration: spatial registration is performed on different modal data such as optical images, spectral data and feature information through image registration technology to ensure their consistency in the same coordinate system. Common registration methods include feature-based registration such as SIFT, ORB, or pixel-based registration such as cross-correlation method.

[0054] Step b: preliminary interpretation and sample library construction

[0055] Remote sensing image interpretation:

[0056] Pixel-level interpretation is performed on the remote sensing optical image, and the urban solid waste region category is labeled, including dividing the remote sensing optical image into tiles and image recognition, and through artificial labeling, each tile is corresponded to a solid waste region category one by one to obtain a preliminary labeling map of urban solid waste based on remote sensing optical image. In the present application, the solid waste region category includes a domestic waste region, a construction waste region, an industrial solid waste region and the like.

[0057] Spectral data interpretation:

[0058] The spectral data are classified, and the solid waste region category is labeled on the remote sensing spectral data through artificial labeling to obtain a labeling map of urban solid waste based on remote sensing spectral data.

[0059] Feature data interpretation:

[0060] The surrounding feature data are interpreted, and the feature characteristics such as buildings, roads and greenery around the point are extracted in combination with the ground investigation data. Spatial analysis methods such as buffer analysis are used to classify the urban solid waste points, record the feature information near the points, and construct a multi-modal data sample library. Buffer analysis is one of the spatial analysis tools for solving the proximity problem, which is to automatically establish a buffer polygon layer within a certain width range around a point, line or surface entity, and then establish the overlay of the layer and the target layer to obtain the required results. In the present application, the feature characteristics of the pre-processed feature data are extracted, and the feature data characteristics are corresponded to the solid waste category, and the solid waste category includes domestic waste, construction waste, industrial solid waste and the like.

[0061] Sample library construction:

[0062] The urban solid waste labeling information extracted from the optical image, spectral data and feature data are integrated to construct a multi-modal data sample library. The sample library will be used for subsequent model training to provide high-quality training data for solid waste identification.

[0063] Step c: optical recognition model construction and preliminary screening

[0064] The YOLOv8 model is used for urban solid waste detection in remote sensing optical images. Through its powerful target detection capability, it quickly identifies solid waste areas. YOLOv8 uses an improved CSPDarknet backbone network, which can extract efficient features. Through convolutional layers, batch normalization (BN), and activation functions, spatial features of remote sensing images are extracted. YOLOv8 can perform feature fusion at different scales, allowing the model to accurately predict the bounding boxes and classes of solid waste areas at multiple scales. YOLOv8 includes efficient detection heads that combine convolutional layers and detection algorithms to accurately locate solid waste areas.

[0065] YOLOv8 model construction:

[0066] The YOLOv8 (You Only Look Once Version 8) model is used for preliminary screening of urban solid waste areas. YOLOv8 is a real-time target detection model based on convolutional neural networks (CNN), with efficient feature extraction capabilities and fast inference speed.

[0067] Network architecture: YOLOv8 uses an improved CSPDarknet backbone network, which extracts features through convolutional layers and predicts the location and class of solid waste areas through efficient detection heads. This model can perform target detection at multiple scales and accurately locate the bounding boxes of solid waste areas.

[0068] Training process: Data augmentation such as rotation, cropping, flipping, and multi-scale training strategies are used to increase the diversity of training data and improve the robustness of the model. The model is trained at different scales to improve its detection capabilities for solid waste areas of different sizes. During training, the cross-entropy loss function (Cross-Entropy Loss) is used to optimize the model, ensuring the accuracy of solid waste area classification detection.

[0069] The trained optical recognition model is used for preliminary screening of solid waste areas, obtaining the positioning and classification results of solid waste points. The classification results can be residential waste areas, construction waste areas, industrial solid waste areas, etc.

[0070] Preliminary screening results:

[0071] The preprocessed remote sensing optical images are input into the trained YOLOv8 model, and target detection is performed through the trained YOLOv8 model to obtain preliminary solid waste points. Each solid waste area class corresponds to specific optical image features, such as specific pixel value intervals, optical image colors, site shapes, optical image texture smoothness, etc. The trained YOLOv8 model performs preliminary screening of solid waste areas based on the obtained optical image features, obtaining the positioning and classification results of solid waste points.

[0072] In the present application, a solid waste point list is obtained through the trained optical recognition model, which includes the solid waste area type of a certain solid waste point, the position coordinates of the solid waste point and other information. According to the bounding box predicted by the YOLOv8 model, a preliminary detection result of the solid waste area is obtained. Through threshold filtering, low-confidence detection results are removed, and high-confidence solid waste area candidate points are obtained. The threshold is the percentage of a certain solid waste point belonging to a specific solid waste area category calculated by the YOLOv8 model. When the percentage of a certain solid waste point belonging to a specific solid waste area category is lower than the set threshold, the certain solid waste point is removed and no longer calculated.

[0073] In the present application, YOLOv8 includes three parts, namely the backbone network, the neck network and the detection head. Each part has a different role. The backbone network is mainly responsible for extracting low-level to high-level features from the input image, such as edges, textures, shapes, etc. Through phased feature extraction, it ensures that the model can obtain target information at different scales. The neck network is mainly responsible for fusing the multi-scale features extracted by the backbone network to enhance the detectability of the target. The detection head is mainly responsible for the final target classification and bounding box regression and outputs the detection result, which is the final result output of the overall recognition task. YOLOv8 includes the three parts working together, making the overall model have the advantages of high efficiency, accuracy and light weight. Under the main framework of the three parts, improved modules are introduced to make the overall architecture of YOLOv8 more suitable for the recognition task of solid waste landfill sites at the city scale

[0074] The first improvement is that the solid waste landfill site recognition not only requires the model to focus on the inherent characteristics of the site itself, but also requires the model to accurately capture the information around the site, i.e. the context information, to help with the recognition of the site. In order to help the model better understand the context information around the site and further help the model locate and classify the solid waste site, the Context Guided Block (CG-Block) (Wu et al., 2021) is introduced. The basic principle of CG-Block is to simulate the human visual system, capture local features, surrounding context and global context of the site, and fuse these information to help understand the site and improve the accuracy of recognition. It contains a local feature extractor ƒ loc (*), a surrounding context extractor ƒ sur (*), a joint feature extractor ƒ joi (*) and a global context extractor ƒ gloThe local feature extractor focuses on extracting features from a local region of the solid waste landfill site. After extracting the features, the remote sensing optical image starts to have a local receptive field in the region, which can be identified by a red box in specific applications. The surrounding context extractor expands the receptive field to capture more extensive context information around the site. In practical applications, the original local region receptive field can be expanded to the size of the outer red box in the entire image. The joint feature extractor combines local features and surrounding context features to enhance the representation ability of the features. The global context extractor aggregates global information of the entire input site image and combines it with the joint features to further optimize the representation of the features.

[0075] In combination Figure 2 The module diagram further analyzes the implementation of the above-mentioned processes from the network architecture level. The module mainly includes two steps: feature learning and feature improvement. In the feature learning step, the local feature extractor f loc The standard 3x3 convolution layer is used to learn local features, processing eight adjacent feature vectors. sur The 3x3 dilated convolution layer is used to capture larger surrounding context. Then, the joint feature extractor f joi combines the output features of f loc and f sur , and performs batch normalization and parameterized ReLU processing after the connection operation to simplify the calculation complexity. In the feature improvement step, the global context extractor f glo extracts the global context to improve the joint features. f glo aggregates the global context of the image through the global average pooling layer, further extracts the global context through the multi-layer perception, and finally applies it to the joint features through the scaling layer, which is equivalent to the introduction of the attention mechanism. The improvement of this part of the module is introduced to the Backbone position of the overall network structure, and the feature layer changes are used for feature extraction.

[0076] The second improvement is that the current traditional YOLO Neck framework, such as FPN and its related variants, can achieve cross-scale feature information fusion of feature maps, but this method also has a significant defect: when fusing cross-scale feature information, it cannot guarantee lossless information transmission for FPN and similar structures. In the structure of FPN, only the information of adjacent layers can be completely fused, and for the information of other layers, only indirect "recursion" can be obtained. This transmission mode may cause information loss in the calculation process, which has become a key factor hindering the improvement of network model performance. In this improvement, aiming at the problems existing in the FPN structure, combined with the characteristics of solid waste landfill sites, a new Gather-Distribute mechanism is introduced based on the theory of TopFormer, which realizes more efficient information exchange in the network model by fusing multi-layer features and injecting global information into higher layers, significantly increasing the information fusion capability of the Neck structure (Wang et al., 2023). The GD mechanism gives up the recursive method and constructs a new information fusion, which collects and fuses information from each layer using a unified module, and then distributes it to different layers. In specific implementation, the aggregation and distribution processes correspond to three modules: Feature Alignment Module (FAM), Information Fusion Module (IFM), and Information Injection Module (Inject). Among them, the aggregation process uses the Feature Alignment Module (FAM) to collect and align features from different layers, and then uses the Information Fusion Module (IFM) to fuse the aligned features to obtain global information. The distribution process is after collecting global information, using the Information Injection Module (Inject) to distribute these information to different levels, and using simple attention operations for injection, so as to improve the detection ability of the branch. The specific network structure is as follows:

[0077] To adapt to different scales of solid waste landfill sites and increase the model's ability to detect different sizes of solid waste site targets, two branches, Low-GD and High-GD, are proposed as follows Figure 3The input of Low-GD is the B2, B3, B4 and B5 feature layers extracted in Backbone. Its main purpose is to enhance the ability to detect small target solid waste sites, and it pays more attention to high-resolution features, that is, the input low-level features B2 and B3. B2 and B3 have higher spatial resolution and can clearly retain the edge information and details of small target sites. Small target features are often weak, and if information is directly extracted from high-level features (such as B4, B5), it is easy to cause the features of small targets to be submerged. Therefore, Low-GD mainly extracts features from B2 and B3, and aligns the local information through Low-FAM to obtain a more detailed target representation. But it is not difficult to find that in Low-GD, B5 features and B4 features are also introduced through Bilinear interpolation up-sampling, which is to let the low-level features obtain certain high-level semantic information, help the model to distinguish small target sites and background, and play a role of “information supplement”. After the aligned and supplemented feature information, it is fused through Low-IFM, and finally distributed to P3 and P4 by Inject module.

[0078] On the contrary, High-GD is mainly responsible for the detection of large target solid waste sites, such as Figure 4 As shown, the main input of High-GD is P3, P4 and P5, which are low-resolution and high-semantic information features. The structure of High-FAM is consistent with that of Low-FAM, which is to align the features. The High-IFM is quite different from the Low-IFM. Unlike the Low-IFM, which uses convolution to simply fuse features, the High-IFM adds a multi-head attention mechanism architecture in the Transformer to calculate attention, realize global information interaction, and further extract deep information through a feedforward neural network to improve the non-linear expression ability. Finally, the Inject module is used to distribute to N4 and N5. In summary, the biggest difference between High-GD and Low-GD is that each module focuses on different levels of features, and they are designed to focus on solid waste sites of different sizes.

[0079] Finally, combined with the above two improvements, the Context-GD Net suitable for solid waste site recognition is developed.

[0080] Step d: Spectral recognition model construction and screening

[0081] Random forest model construction:

[0082] A Random Forest model is constructed based on spectral data for spectral feature selection and classification. Random Forest is an ensemble learning method that makes classification decisions through multiple decision trees. Each tree avoids overfitting and improves the model's generalization ability by randomly selecting a subset of features during training. The Random Forest model is used to screen and classify solid waste areas based on remote sensing spectral data. Random Forest is composed of multiple decision trees, each tree splits based on a randomly sampled subset of features, and the final classification result is obtained by integrating the prediction results of all trees. The model classifies solid waste areas based on spectral band features, and uses feature importance calculation methods to select bands related to solid waste areas. Since Random Forest makes decisions through the integration of multiple decision trees, it has strong noise resistance and can handle complex spectral data.

[0083] Model training: The Random Forest model is trained using the spectral data obtained from step b. The model classifies solid waste based on different spectral band features and outputs the classification result of each point. During training, cross-validation is used to optimize hyperparameters, such as the number of trees and depth, to select the best.

[0084] Feature subset sampling: During the training process of each tree, a randomly sampled subset of features is used to avoid overfitting.

[0085] Cross-validation: Cross-validation is used to optimize the model's hyperparameters (such as the number of trees and depth) to improve the model's generalization ability.

[0086] Band selection: By calculating the feature importance of different bands, select the bands highly related to solid waste areas to optimize the classification accuracy of the model.

[0087] Output results: The Random Forest model classifies solid waste areas based on spectral information from the training data set and outputs the classification result of each solid waste point. In this invention, the trained spectral recognition model is used to further screen the solid waste points screened by the optical recognition model to determine the accuracy of the classification result of the solid waste points determined by the optical recognition model.

[0088] Spectral screening:

[0089] In this invention, the solid waste points screened by the trained YOLOv8 model are a list of solid waste points, and the spectral recognition model is used to further screen the list to prevent the optical recognition model from confusing the site, and to prevent water bodies, wastelands, etc. from being misidentified as solid waste areas.

[0090] The remote sensing spectral data of the region where the solid waste points selected by the trained YOLOv8 model are located are input into the trained random forest model. Different types of solid waste regions correspond to different band features. The trained random forest model obtains the band features and determines the type of solid waste region based on the band. The trained random forest model is further screened, and the solid waste region is finely classified based on the spectral features. The screening process optimizes the classification decision process based on the feature importance in the spectral data, improving the recognition accuracy. For example, if the trained YOLOv8 model selects 100 solid waste points, the remote sensing spectral data of the region where the 100 solid waste points are located is input into the trained random forest model. The random forest model determines the type of solid waste region corresponding to the remote sensing spectral data based on the spectral features. The random forest model determines the result as follows:

[0091] First, if the type of solid waste region determined by the random forest model is consistent with the type of solid waste point obtained by the YOLOv8 model, the optical recognition model determines the solid waste point accurately. The solid waste point passes the screening of the YOLOv8 model and the random forest model, and the solid waste point that passes the screening of the YOLOv8 model and the random forest model is retained.

[0092] Second, if the type of solid waste region determined by the random forest model is inconsistent with the type of solid waste point obtained by the YOLOv8 model, it is determined that the solid waste point determined by the optical recognition model is inaccurate. The solid waste point does not pass the screening of the YOLOv8 model and the random forest model. The solid waste point that does not pass the screening of the YOLOv8 model and the random forest model is not processed as a solid waste region.

[0093] Step e: ground object recognition model construction and fine screening

[0094] TPOT model construction:

[0095] Based on the TPOT (Tree-based Pipeline Optimization Tool) method, a ground object recognition model is constructed. TPOT is an automatic machine learning tool based on tree structure, which optimizes feature selection and model configuration to improve model performance. TPOT is used to finely screen solid waste regions based on ground object features. TPOT method automatically selects the best feature engineering and model configuration to optimize feature selection and classification process. TPOT can optimize multiple feature engineering and machine learning models through tree structure pipeline.

[0096] Different types of solid waste correspond to corresponding surrounding ground object data. The ground object data labeled with the solid waste category is input into the constructed TPOT model for model training. The trained TPOT model is used to determine the solid waste category of the solid waste point selected by the spectral recognition model.

[0097] Feature extraction: Extract typical features from surrounding feature data (such as roads, buildings, green belts, etc.), combine the feature information around the point, and build a feature recognition model to support the final screening of solid waste areas.

[0098] Feature selection: TPOT automatically selects the best feature related to solid waste recognition, and through continuous optimization of feature engineering, extracts features that help solid waste screening.

[0099] Model configuration optimization: Through the optimization of tree structure, TPOT can select the most suitable model configuration to make the feature recognition process automatic and maximize the model performance.

[0100] Fine screening:

[0101] The feature data around the solid waste point screened by the optical and spectral model is input into the feature recognition model for further screening. The feature model combines feature information (such as land division, terrain features, etc.) to fine-tune the solid waste area, ensuring the spatial accuracy and classification accuracy of the screening results. TPOT model further screens the solid waste area screened by the optical and spectral recognition model based on feature characteristics, and finally obtains high-precision solid waste extraction results.

[0102] As the last step of the entire city-scale solid waste landfill site recognition, the role of feature information screening is to determine the final site category based on the point screened in the previous step. In the entire recognition process, the first step of optical screening and the second step of spectral screening are essentially based on the characteristics of solid waste landfill sites, using advanced models to extract features and locate sites. The purpose of designing the third step of feature model information screening is not to be limited to the characteristics of solid waste landfill sites, but to consider the actual process of recognizing and classifying solid waste landfill sites. Assuming that we determine the location and category of a site through human recognition, in the entire process, in addition to investigating the site itself, we will also investigate the surrounding feature information of the site, including the land use type of the site, the typical features around the site, the distance between the typical features, and the population density of the region to assist in the final recognition and classification of the site. Therefore, the third step of feature information screening is designed for this purpose, based on the point results obtained in the previous two steps, using surrounding feature information to make the final judgment of the point category, fully ensuring the rationality and completeness of the solid waste landfill site recognition task process at the city scale, and ensuring the accuracy of the final site point recognition and classification results.

[0103] The present application adopts a TPOT (Tree-based Pipeline Optimization Tool) AutoML framework in the ground object information screening stage. The framework is based on a genetic algorithm (Genetic Algorithm, GA) and can automatically complete feature selection, model selection and hyperparameter optimization, thereby improving the accuracy and efficiency of the classification task. Traditional machine learning modeling usually requires manual design of features, selection of appropriate models, and complex hyperparameter tuning. TPOT automatically builds the optimal machine learning pipeline through evolutionary computation, effectively reducing the impact of human intervention on model performance.

[0104] Unlike the second part of the spectrum screening, although both essentially use a machine learning framework to train the model, the two are different in terms of data characteristics, model complexity, computational efficiency, and automation requirements, which ultimately lead to different methods. The spectral data in the second step of spectral screening is relatively simple, with features mainly concentrated in multiple bands and strong spatial characteristics. Traditional machine learning TabPFN performs well in this high-dimensional, low-interactive task, effectively capturing the relationship between features and categories. In the third step of ground object information screening, more diversified features are introduced, including roads, population, and construction land, with strong interactivity and complexity between features. TPOT processes these complex features automatically and optimizes multiple models to adapt to the needs of this complex type of data. In addition, from the perspective of model performance and computational efficiency, single multi-spectral data can achieve efficient and accurate classification using traditional machine learning algorithms, but complex ground object information data also requires increasing the complexity of the model to ensure the model's ability to handle complex data. Therefore, the selection of different algorithms is also the result of considering data, models, efficiency and other aspects.

[0105] In the above TPOT flowchart, the most important part of TPOT is the feature engineering part, including feature selection, feature preprocessing and feature reconstruction. The site location information of the solid waste point obtained in the second step and the predicted site category are input as part of the input, and a series of solid waste landfill site surrounding ground object information data collected are input as another part of the input. First, the data is cleaned and preprocessed to structure and standardize the data features, remove redundant information, including missing value processing, i.e. filling missing data or deleting samples with too many missing values, data standardization, i.e. converting feature values to a unified range, such as standard normal distribution, to avoid some feature values being too large or too small, data encoding, i.e. converting some classification features to digital expression encoding format, etc., to provide sufficient and reliable data support for the subsequent process. Feature selection in feature engineering is to select the most useful features for site recognition and classification from the input features, such as some key ground objects in typical ground objects, such as waste disposal facilities, industrial buildings, etc., and threshold values between traffic networks, etc. These need to be selected through feature selection. In addition, there is feature reconstruction, which takes into account the interaction between features, combines, transforms or extends the original features to generate advanced features that can better represent the characteristics of solid waste landfill sites, such as distance feature cross index, i.e. composite distance feature considering road network distance and water system distance, regional index, i.e. regional composite index combining regional attributes and population density index, etc. The above feature engineering not only studies and expresses the data features at multiple levels, but also further provides important value for the selection of subsequent models and the optimization of parameters.

[0106] In the present application, the ground object recognition model further confirms the solid waste type corresponding to the solid waste point according to the site features around the solid waste point screened by the spectrum recognition model. For example, the trained YOLOv8 model screens 100 solid waste points, and the remote sensing spectrum data of the region where the 100 solid waste points are located is input into the trained random forest model. Among them, 80 solid waste points, the solid waste type judged by the random forest model is consistent with the solid waste type judged by the YOLOv8 model, then the ground object data around the 80 solid waste points is input into the ground object recognition model. Different solid waste categories correspond to different ground object features, such as industrial waste around industrial plants, and domestic waste around residential areas, etc. The solid waste categories of the 80 solid waste points are confirmed through the ground object features around the 80 solid waste points. When the judgment result of the ground object model is inconsistent with the judgment result of the random forest model, the judgment result of the ground object model is used as the standard.

[0107] The multiple models (YOLOv8 optical recognition model, random forest spectrum recognition model, and TPOT ground feature recognition model) each have their own advantages in different modal data. Through joint training and end-to-end optimization, the extraction accuracy of urban solid waste can be effectively improved, and high-precision positioning and classification of solid waste areas in complex urban environments can be ensured. In addition, data enhancement techniques and automated optimization methods make the entire model have good adaptability and generalization ability, and can cope with changes in different scenarios and data characteristics.

[0108] Step f: final solid waste extraction and result generation

[0109] Remote sensing image input and block cropping:

[0110] After preprocessing the remote sensing image of the region to be identified, block cropping operation is performed to divide the image into blocks of appropriate size, which are input into the optical recognition model one by one to obtain preliminary solid waste point detection results.

[0111] Multi-stage screening and result generation:

[0112] According to the preliminary screening results, the spectral band data of the solid waste point is obtained and input into the spectrum recognition model for screening. According to the screening results, the ground feature information around the solid waste point is obtained and input into the ground feature recognition model for final screening.

[0113] High-precision solid waste extraction results:

[0114] Finally, according to the joint results of the optical recognition, spectral recognition and ground feature recognition models, high-precision urban solid waste recognition results are obtained. The spatial position, category information and related features of the solid waste area are output for subsequent analysis and decision-making.

[0115] The application also discloses a multi-modal model urban solid waste high-precision extraction system based on multi-source data.

[0116] A computing device includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the multi-modal model urban solid waste high-precision extraction method based on multi-source data as described.

[0117] A readable storage medium, the readable storage medium having a computer program stored thereon, the computer program being executed to implement the multi-modal model urban solid waste high-precision extraction method based on multi-source data.

[0118] The above embodiments are merely exemplary but not limiting on the implementation. Based on the above description, those skilled in the art can make further changes or modifications to the embodiments in different forms. Here, it is not necessary or possible to enumerate all the embodiments. The obvious changes or modifications derived from the above description are still within the scope of the present application.

Claims

1. A high-precision extraction method for municipal solid waste based on a multimodal model of multi-source data, characterized by: The method comprises: Data collection and preprocessing: collecting multimodal data on municipal solid waste, including remote sensing optical images, remote sensing spectral data, and surrounding ground feature data, and preprocessing the collected data; Build a sample library based on preprocessed data, including: Label the pre-processed remote sensing optical images of the solid waste area and label the remote sensing images of the solid waste area as solid waste area categories; Classify the pre-processed remote sensing spectral data and mark the remote sensing spectral data as solid waste area categories; Extract the features of the pre-processed feature data and label the feature data as solid waste categories; Model building and training, including: Optical recognition model construction and training, including using the YOLOv8 model to input remote sensing optical images labeled with solid waste area categories into the YOLOv8 model for training. The trained YOLOv8 model is used to preliminarily determine the solid waste area categories corresponding to the remote sensing optical images and obtain a list of solid waste locations, including the location coordinates and solid waste categories of the solid waste locations; Spectral recognition model construction and training, which includes using a random forest model to input remote sensing spectral data labeled with solid waste area categories into the constructed random forest model. The trained random forest model determines whether the classification results of solid waste points determined by the optical recognition model are accurate based on spectral characteristics; ground feature recognition model construction and training, different solid waste categories correspond to corresponding surrounding ground feature data, and the ground feature data labeled with solid waste categories are input into the constructed ground feature recognition model for model training. The trained ground feature recognition model is used to determine the solid waste categories of solid waste points selected by the spectral recognition model; Solid waste category extraction and result generation: the remote sensing spectral data corresponding to the area where the solid waste points are located screened out by the YOLOv8 model is input into the trained random forest model for further screening; the ground feature information around the solid waste points screened out by the random forest model is input into the trained ground feature recognition model; the trained ground feature recognition model outputs the solid waste category based on the ground feature information around the solid waste points.

2. The high-precision extraction method for municipal solid waste based on a multi-modal model of multi-source data according to claim 1 is characterized in that: The preprocessing of the collected data includes: Atmospheric correction: perform atmospheric correction processing on remote sensing images and spectral data to eliminate the influence of the atmosphere on remote sensing data; Geometric correction: correct image distortion and projection error to make the image coordinates consistent with the geographic coordinate system; surrounding feature data processing: unify the surrounding feature data and coordinates; Multimodal data registration uses image registration technology to spatially register optical images, spectral data, and ground feature information, so that data from different modalities are in the same coordinate system.

3. The high-precision extraction method for municipal solid waste based on a multi-modal model of multi-source data according to claim 1 is characterized by: The trained YOLOv8 model includes a local feature extractor f loc (*), surrounding context extractor f sur (*), Joint feature extractor f joi (*) and the global context extractor f glo (*); The local feature extractor is used to extract features from the local area of ​​the solid waste landfill. After feature extraction, the remote sensing optical image has a local area receptive field; the surrounding context extractor is used to expand the receptive field and capture the surrounding context information of the site; The joint feature extractor combines local features with surrounding context features; the global context extractor is used to aggregate the global information of the entire input site image and combine the global information with the joint features; The local feature extractor uses a 3×3 convolutional layer to learn local features and process eight adjacent feature vectors; the surrounding context extractor uses a 3×3 dilated convolutional layer to capture a larger surrounding context; Joint feature extractor f joi (*) f loc (*) and f sur The output features of (*) are combined, batch normalized and parameterized after concatenation operation; Global context extractor f glo (*) Extract global context, f glo (*) Aggregate the global context of the image through a global average pooling layer.

4. The high-precision extraction method for municipal solid waste based on a multi-modal model of multi-source data according to claim 1 is characterized in that: The random forest model judges whether the classification result of the solid waste point determined by the optical recognition model is accurate based on the spectral characteristics, including: after judging the solid waste area type corresponding to the remote sensing optical image based on the trained YOLOv8 model, the solid waste point is obtained, and the remote sensing spectral data of the area where the solid waste point is located is input into the trained random forest model. The random forest model judges the solid waste area type corresponding to the remote sensing spectral data based on the spectral characteristics. If the solid waste area type judged by the random forest model is consistent with the classification result of the solid waste point obtained by the YOLOv8 model, the solid waste area category determined by the optical recognition model is accurate, and the solid waste point passes the screening of the YOLOv8 model and the random forest model; if the solid waste area type judged by the random forest model is inconsistent with the classification result of the solid waste point judged by the YOLOv8 model, the solid waste point judged by the optical recognition model is inaccurate, and the solid waste point obtained by the optical recognition model is not treated as a solid waste area.

5. The high-precision extraction method for municipal solid waste based on a multi-modal model of multi-source data according to claim 1 is characterized by: The construction and training of the land feature recognition model includes using the TPOT model, inputting land feature data marked with solid waste categories into the constructed TPOT model for training, and the trained TPOT model is used to determine the solid waste categories of the solid waste points screened by the spectral recognition model.

6. The high-precision extraction method for municipal solid waste based on a multi-modal model of multi-source data according to claim 5 is characterized in that: The TPOT model is used to determine the solid waste category of the solid waste points screened out by the spectral recognition model, including: inputting the ground feature information around the solid waste points screened out by the YOLOv8 model and the random forest model into the trained TPOT model, and the TPOT model judges the solid waste category based on the ground feature information characteristics around the solid waste points.

7. The high-precision extraction method for municipal solid waste based on a multi-modal model of multi-source data according to claim 5 is characterized in that: The trained TPOT model includes feature selection, feature preprocessing and feature reconstruction. The feature selection is to use the site location information around the solid waste point obtained by the spectral recognition model and the predicted site category as one part of the input, and a series of collected land feature information data around the solid waste landfill as another part of the input; the feature preprocessing includes missing value processing, filling missing data or deleting samples with too many missing values; data standardization uniformly converts feature values ​​into a preset range, and data encoding converts classification features into a coding format for digital expression; the feature reconstruction includes combining, converting or expanding original features to generate features that express the characteristics of the solid waste landfill.

8. A high-precision extraction system for municipal solid waste based on a multi-modal model of multi-source data, characterized by: The system is used to implement the high-precision extraction method for urban solid waste based on a multi-modal model of multi-source data as described in any one of claims 1-7.

Images