A method for extracting agricultural plastic cover by integrating multimodal remote sensing and deep learning

Abstract

This invention relates to the field of intelligent extraction of agricultural plastic mulch, and discloses a method for extracting agricultural plastic mulch by integrating multimodal remote sensing and deep learning. The method includes acquiring multi-temporal multispectral optical remote sensing images, SAR remote sensing images, and stereo image pairs of remote sensing images during key phenological periods of the plastic mulch crop, and processing them accordingly. A corresponding training sample set is constructed based on the processed remote sensing images. A deep semantic segmentation model containing four multi-temporal multimodal feature fusion methods is trained to obtain a target plastic mulch and plastic greenhouse extraction model, extracting the spatial distribution of plastic mulch and plastic greenhouses in the region. This method comprehensively considers the differences in spectral and structural features of ground objects and three-dimensional height information. By utilizing multimodal remote sensing and deep learning, it fully explores the complementary information and spatiotemporal correlations between multi-temporal and multimodal data, improving the extraction accuracy of regional plastic mulch and plastic greenhouses. It requires less manual intervention, has a high degree of automation, and has strong universality, making it easy to promote and apply at the regional scale.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent extraction of agricultural plastic mulching, in particular to an agricultural plastic mulching extraction method combining multi-modal remote sensing and deep learning, belonging to the application fields of agricultural ecological environment safety, agricultural precision management, remote sensing intelligent monitoring, etc. BACKGROUND

[0002] As an important technical carrier of modern agricultural intensification, agricultural plastic mulching (plastic film and plastic greenhouse) plays a key role in improving multiple cropping index and ensuring food production, and has been widely used worldwide. However, long-term mulching and lack of residual film recycling system have led to soil structure destruction, biodiversity degradation, and microclimate imbalance, etc. Therefore, accurately grasping the spatial distribution of agricultural plastic mulching is helpful to improve agricultural management efficiency and scientifically assess the potential risk area of agricultural plastic pollution, which is crucial for balancing high crop yield and ecological environment protection.

[0003] At present, China's agricultural plastic mulching monitoring system mainly relies on statistical data and field investigation. Statistical data mainly focus on the use scale of plastic film, lacking key information such as spatial distribution and temporal variation; although field investigation method can obtain spatial distribution, it is time-consuming, labor-intensive, and has small survey range, lacking timeliness, which is difficult to meet the needs of modern agricultural precision and efficient management.

[0004] In recent years, remote sensing technology has provided a new path for precise monitoring of agricultural plastic mulching due to its objective and large-scale dynamic monitoring advantages. However, existing research mainly relies on single modal remote sensing data such as optical remote sensing and synthetic aperture radar (SAR) to extract target ground objects. Although optical satellite data has advantages such as multi-scale spatial resolution and spectral resolution, it is often difficult to obtain effective images in key phenological periods due to cloud and fog obstruction; SAR images have strong penetration ability, but their scattering intensity is affected by complex interactions of multiple physical properties such as surface roughness, geometric structure (orientation, slope), dielectric constant (mainly affected by water content), etc., resulting in serious "same object different spectrum" and "same spectrum different object" phenomena. Therefore, by combining the advantages of optical and SAR data, the information limitations of single modal can be effectively broken through, and the extraction accuracy of agricultural plastic mulching can be improved. However, existing researches mainly focus on the fusion of single-phase optical and SAR images, and pay insufficient attention to the time dynamic characteristics of mulched crops, and there are limitations in dealing with complex interactions among multi-temporal and multi-modal data. In addition, plastic greenhouses and plastic films have obvious three-dimensional elevation differences from other ground objects, but existing methods generally ignore the unique and key feature factor of nDSM elevation information, which limits the model's ability to distinguish complex ground cover types.

[0005] In addition to the characteristics of remote sensing data, image processing methods are also crucial in high-precision mapping of agricultural plastic mulching. Current mainstream methods include the pixel method, the object-oriented method, and traditional machine learning algorithms (such as support vector machines and random forests), but they are all limited by the manual feature design paradigm. With the rapid development of computer technology, deep learning can autonomously capture complex spatial-spectral features in remote sensing images through end-to-end feature learning and powerful nonlinear modeling capabilities, improving the recognition accuracy of target ground objects. Semantic segmentation, as an important branch of deep learning in the field of computer vision, has shown significant advantages in target recognition due to its pixel-level classification, automated feature learning, and strong spatial context modeling capabilities. SUMMARY

[0006] To address the deficiencies of the prior art, the present application proposes a method for extracting agricultural plastic mulching by fusing multi-modal remote sensing and deep learning, which fully considers the key phenological period of mulched crops, as well as the influence of ground object spectral and structural differences and three-dimensional height information, and proposes a deep semantic segmentation model containing four multi-temporal multi-modal fusion methods, which fully exploits and utilizes the features of multi-temporal multi-modal remote sensing images, significantly improving the extraction accuracy of existing mulching films and plastic greenhouses. Moreover, this method has high automation, less human intervention, and is easy to apply on a large scale.

[0007] The application provides a method for extracting agricultural plastic mulching by fusing multi-modal remote sensing and deep learning, which adopts the following technical scheme:

[0008] A method for extracting agricultural plastic mulching by fusing multi-modal remote sensing and deep learning, comprising the following steps:

[0009] S1, obtaining multi-temporal multi-modal remote sensing images of the area to be studied, the multi-temporal being the key phenological period of mulched crops; the multi-modal remote sensing images including multi-spectral optical remote sensing images, SAR remote sensing images, and stereo pair remote sensing images;

[0010] The stereo pair remote sensing images are any phase remote sensing images during the phenological period of mulched crops, including panchromatic forward-looking remote sensing images and panchromatic rear-looking remote sensing images;

[0011] S2, pre-processing each phase of the multi-spectral optical remote sensing images in S1 to obtain multi-temporal multi-spectral optical image result maps representing the true reflectivity of ground objects;

[0012] S3, performing multi-view, registration, filtering, geographic coding, and radiation calibration on each phase of the SAR remote sensing images in S1 to obtain multi-temporal SAR image result maps representing the backscattering coefficient of ground objects;

[0013] S4. Perform stereo image pair processing on the stereo image pair remote sensing images in S1 to obtain an nDSM image result map containing information on the vertical structure of ground features.

[0014] S5, based on the multispectral optical image result map of any time phase in S2, performs geometric registration, resampling and overlap range extraction on the multispectral optical image result map of the remaining time phase in S2, the multi-time phase SAR image result map in S3 and the nDSM image result map in S4, to obtain a multispectral optical image atlas, SAR image atlas and nDSM image map that match in geospatial coordinates and have consistent spatial resolution and range.

[0015] S6, based on the multispectral optical image atlas, SAR image atlas and nDSM image atlas in S5, constructs a multi-temporal, multimodal remote sensing image training sample set;

[0016] S7, based on the multi-temporal, multi-modal remote sensing image training sample set constructed in S6, trains the deep semantic segmentation model until the loss function converges to the preset value, and obtains the target mulch film and plastic greenhouse extraction model;

[0017] S8, based on the target mulch film and plastic greenhouse extraction model in S7, extracts mulch film and plastic greenhouses from the multi-temporal, multi-modal remote sensing images of the area under study, and obtains the spatial distribution of mulch film and plastic greenhouses in the area under study.

[0018] Preferably, the preprocessing in S2 includes: radiometric calibration, atmospheric correction, and orthorectification.

[0019] Preferably, the stereo image pair processing process in S4 is as follows:

[0020] Stereo matching is performed on panchromatic forward-looking remote sensing images and panchromatic backward-looking remote sensing images to generate three-dimensional surface point cloud data. The point cloud data is then interpolated and smoothed to generate a continuous surface elevation model (DSM).

[0021] A terrain filter is used to separate the ground and features from the DSM, resulting in a ground DEM and a feature DEM.

[0022] A smoothing filtering algorithm is used to eliminate local outliers and high-frequency noise in the ground DEM, resulting in a noise-reduced ground DEM.

[0023] Spatial interpolation tools were used to fill the voids in the noise-reduced ground DEM to obtain the ground DTM;

[0024] Spatial difference calculations are performed between the DSM and the ground DTM to obtain an nDSM image result map containing vertical structure information of ground features.

[0025] Preferably, the multi-temporal, multi-modal remote sensing image training sample set in S6 includes multi-temporal, multi-modal remote sensing images and corresponding mulch film label images and plastic greenhouse label images.

[0026] Preferably, the deep semantic segmentation model in S7 includes an encoder, a multi-temporal multimodal feature fusion module (MTMF), a decoder, and skip connections;

[0027] a. Encoder;

[0028] The encoder includes a time phase. to Optical coding features, temporal phase to SAR coding features and temporal phase The nDSM encoding features, each branch contains one patch embedding layer and four encoder sub-modules. The first three encoder sub-modules each include two visual state space modules and one downsampling module, and the fourth encoder sub-module includes two visual state space modules.

[0029] The time phase Any phase within the phenological period of plastic film-covered crops;

[0030] Where n is greater than or equal to 2;

[0031] b. Multi-temporal and multi-modal feature fusion module (MTMF);

[0032] The multi-temporal multimodal fusion module (MTMF) uses a cross-attention mechanism to enhance features and allocate weights across different temporal phases and modalities. The MTMF module incorporates four multi-temporal multimodal feature fusion methods:

[0033] All features are directly superimposed: Remote sensing data features of all modes across all time phases are directly superimposed to obtain temporal-modal fusion features;

[0034] Phase first, then mode: First, consider the phase separately. to Optical coding features and temporal phase to Adaptive temporal enhancement is performed on the SAR coding features to obtain multi-temporal optical fusion features and multi-temporal SAR fusion features; then the multi-temporal optical fusion features, multi-temporal SAR fusion features, and temporal enhancement are further performed. Adaptive modality enhancement is performed on the nDSM encoded features to obtain temporal-modality fusion features;

[0035] Modality first, then phase: First, separately... to Optical coding features and SAR coding features of the same phase and phase Adaptive modal enhancement is performed on the nDSM encoded features to obtain temporal phases. to The multimodal fusion features, and then the temporal phase to Adaptive temporal enhancement is performed on the multimodal fusion features to obtain temporal-modal fusion features;

[0036] Pre-phase modal fusion feature constraints followed by post-phase modal fusion: First, the temporal phases are constrained... Optical coding features and SAR coding features and temporal phase Adaptive modal enhancement is performed on the nDSM encoded features to obtain the temporal phase. Multimodal fusion features, and then temporal phase Multimodal fusion features, temporal phase Optical coding features and SAR coding features and temporal phase Adaptive modal enhancement is performed on the nDSM encoded features until the temporal phase is optimized. Multimodal fusion features, temporal phase Optical coding features and SAR coding features and temporal phase Adaptive modality enhancement is performed on the nDSM encoded features to obtain temporal-modality fusion features;

[0037] Where n is greater than or equal to 2;

[0038] The calculation method for the cross-attention mechanism is as follows:

[0039] First, cross-attention calculation is performed for each modality. The characteristics of this feature are used, with residual modes as keys and values ​​for attention computation; for temporal... The feature is that the remaining time phase is used as the key and value for attention calculation, and the calculation formula is:

[0040] ;

[0041] In the formula, For the first The results of cross-attention calculation for each modality or phase It is modal or phase The query, and It is modal or phase The transpose and values ​​of the key matrix, and It is modal or phase The transpose and values ​​of the key matrix, For the dimension of the key, It is a normalization operation, in which It is an integer greater than or equal to 1, and , It is an integer greater than or equal to 2;

[0042] Secondly, the cross-attention calculation results of each modality or time phase are fused using a weighted average method to obtain fused features;

[0043] c. Decoder;

[0044] The decoder comprises four decoder sub-modules, wherein the first decoder sub-module includes two visual state space modules, and the remaining three decoder sub-modules each include two visual state space modules and one upsampling module.

[0045] d. Jump links;

[0046] The skip connection adopts an additive operation.

[0047] Preferably, the target mulch film and plastic greenhouse extraction model in S7 is a model constructed based on the multi-temporal multi-modal feature fusion method with the highest accuracy selected after comparing four multi-temporal multi-modal feature fusion methods.

[0048] Preferably, the loss function in S7 is:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] in, The total loss for the two categories, Let be the binary cross-entropy, which is the category loss function for binary classification; For multi-class total loss, Cross-entropy is the category loss function for multi-class classification; The Dice loss represents the similarity between the segmentation result and the true label; N represents the total number of samples, and C represents the total number of categories. and These represent the true label and the predicted label, respectively. It is an identifier, if the sample If it belongs to category c, the value is 1; otherwise, the value is 0. The model predicts the sample. The probability of belonging to category c; and These represent the true labels and the predicted results, respectively. , Each term controls the relative importance of different loss terms in the formula; the default value is 1.

[0055] The beneficial effects of this invention are as follows:

[0056] 1. This invention fully considers the key phenological stages of plastic film crops, as well as the influence of differences in ground object spectral and structural information and three-dimensional height information. It proposes a deep semantic segmentation model that includes four multi-temporal and multi-modal fusion methods, which fully explores and utilizes the features of multi-temporal and multi-modal remote sensing images, significantly improving the extraction accuracy of existing plastic film and plastic greenhouses. Furthermore, this method has a high degree of automation, requires little manual intervention, and is easy to promote and apply on a large scale.

[0057] 2. This invention significantly improves the recognition accuracy of agricultural plastic mulch in complex scenarios. It fully considers the key phenological stages of plastic film crops, as well as the influence of differences in the spectral and structural characteristics of ground objects and three-dimensional height information. By introducing a multi-temporal multimodal feature fusion module (MTMF), it can adaptively learn and fuse the spectral information of multispectral images, the geometric structure of SAR images, and the three-dimensional height information provided by nDSM. Moreover, the deep dynamic fusion effectively overcomes the limitations of single-modal data sources (such as spectral confusion, cloud and fog obstruction, and surface structure interference), significantly improving the recognition accuracy of plastic film and plastic greenhouses in complex farmland scenarios (such as crop growth period changes and soil background interference).

[0058] 3. This invention breaks through the limitations of traditional fusion methods and achieves optimal feature representation. Its proposed MTMF module can deeply mine the complementary information and spatiotemporal correlation between multi-temporal and multi-modal data, and dynamically find and realize the optimal feature fusion method. Compared with traditional fixed rule fusion methods such as band stacking or shallow feature fusion methods, this invention can more effectively extract the most discriminative comprehensive features for identifying plastic film and plastic greenhouses, suppress redundant and interference information, and thus achieve significant optimization at the feature level, laying a solid foundation for high-precision identification.

[0059] 4. This invention provides a new, efficient, and reliable method for precise monitoring of agricultural plastic mulch. By offering an end-to-end solution based on deep learning and advanced fusion strategies, it can automatically and intelligently process multi-source heterogeneous remote sensing data to generate high-precision agricultural plastic mulch distribution maps. This method significantly reduces reliance on manual interpretation and improves monitoring efficiency. At the same time, its excellent accuracy and robustness enable it to adapt to a wide range of complex and changing agricultural environments, providing strong technical support and a novel approach for agricultural plastic pollution supervision, environmental assessment, and precision agriculture management. Attached Figure Description

[0060] Figure 1 This is a flowchart illustrating an embodiment of this application;

[0061] Figure 2 This is a schematic diagram of a deep semantic segmentation model according to an embodiment of this application;

[0062] Figure 3 These are schematic diagrams illustrating four multi-temporal and multi-modal feature fusion methods according to embodiments of this application;

[0063] Figure 4 The image is a Sentinel-2 visible-near-infrared remote sensing image from an embodiment of this application.

[0064] Figure 5 This is a Sentinel-1 VV polarimetric SAR remote sensing image from an embodiment of this application;

[0065] Figure 6 The three images are the front-view and rear-view stereo image pairs of Resource No. 3 in this application embodiment. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0067] This embodiment uses Sentinel-2 visible-near-infrared remote sensing images, Sentinel-1 SAR remote sensing images, and ZY-3 stereo image pairs from two key phenological periods—the bare mulch period and the vigorous vegetation period—as examples to illustrate the extraction of agricultural plastic mulch using the Vision Mamba UNet (VM-UNet) deep semantic segmentation model. Sentinel-2 is a multispectral optical satellite with 10m resolution visible-near-infrared remote sensing images. Sentinel-1 is a C-band synthetic aperture radar capable of generating 10m resolution result maps. ZY-3 carries a three-line array mapping camera with 2.5m resolution panchromatic stereo images (forward-looking and backward-looking), capable of generating 10m resolution nDSM. The VM-UNet model captures extensive contextual information by introducing Visual State Space (VSS) blocks as base blocks and constructs an asymmetric encoder-decoder structure. The Sentinel-2, Sentinel-1, ZY-3, and VM-UNet models mentioned above can meet the requirements for accurate and intelligent extraction of mulch film and plastic greenhouses using multimodal remote sensing images and deep semantic segmentation models.

[0068] Reference Figure 1 This invention discloses a method for extracting agricultural plastic cover by integrating multimodal remote sensing and deep learning, which includes the following steps:

[0069] S1, acquire multi-temporal, multimodal remote sensing images of the area under study, where the multi-temporal phases are the key phenological periods of the mulched crop; the multimodal remote sensing images include multispectral optical remote sensing images, SAR remote sensing images, and stereo image pairs remote sensing images.

[0070] Specifically, the multispectral optical remote sensing images used Sentinel-2 L2A level data from February 16 and April 26, 2024, included 13 bands and three spatial resolutions: 10m, 20m, and 60m.

[0071] The SAR remote sensing images used were GRD data in Sentinel-1IW imaging mode taken on February 18 and April 28, 2024.

[0072] The stereo image pair remote sensing image is a remote sensing image of any time phase within the phenological period of the mulched crop, including panchromatic forward-looking remote sensing image and panchromatic backward-looking remote sensing image;

[0073] The stereo image pairs were selected from ZY-3 stereo remote sensing images dated December 2, 2023 and January 25, 2024.

[0074] S2, preprocess the multispectral optical remote sensing images of each time phase in step S1 to obtain a multispectral optical image result map representing the true reflectance of ground objects.

[0075] The preprocessing includes: radiometric calibration, atmospheric correction, and orthorectification.

[0076] Specifically, the Sentinel-2 L2A data has been radiometrically calibrated, atmospherically corrected, and orthorectified. In this embodiment, ENVI 6.0 is used to open MTD_MSIL2A.xml and output a multispectral optical image of the visible-near-infrared band with a resolution of 10m.

[0077] S3, perform multi-view, registration, filtering, geocoding and radiometric calibration on the SAR remote sensing images of each time phase in step S1 to obtain multi-time phase SAR image result map representing the backscattering coefficient of ground objects.

[0078] Specifically, the Sentinel-1IW imaging mode GRD data is data that has undergone multi-view processing. During implementation, the ENVI 6.0 SARscape software is mainly used for registration, filtering, geocoding, and radiometric calibration.

[0079] S4. Perform stereo image pair processing on the stereo image pair remote sensing images in step S1 to obtain an nDSM image result map containing information on the vertical structure of ground features.

[0080] Stereo image pair processing is a commonly used process in this field. Software and methods for stereo image pair processing can be selected according to actual needs. Here, this embodiment presents a commonly used stereo image pair processing process, as follows:

[0081] The stereo image pair processing procedure is as follows:

[0082] First, the Generate Point Clouds and DSM by Dense ImageMatching module of ENVI software was used to perform stereo matching, point cloud generation, interpolation and smoothing on panchromatic forward-looking remote sensing images and panchromatic backward-looking remote sensing images to generate a continuous surface elevation model (DSM).

[0083] Secondly, the terrain filter of SAGA GIS software was used to separate the ground and features in the DSM to obtain the ground DEM and feature DEM.

[0084] A smoothing filtering algorithm is used to eliminate local outliers and high-frequency noise in the ground DEM, resulting in a noise-reduced ground DEM.

[0085] Spatial interpolation tools were used to fill the voids in the noise-reduced ground DEM to obtain the ground DTM;

[0086] Finally, the Band Math tool in ENVI software was used to perform spatial difference calculations between the DSM and the ground DTM to obtain an nDSM image result map containing vertical structure information of ground features.

[0087] S5. Based on the multispectral optical image result map of any time phase in step S2, perform geometric registration, resampling and overlap range extraction on the multispectral optical image result map of the remaining time phase in step S2, the SAR image result map of multiple time phases in step S3 and the nDSM image result map in step S4 to obtain a multispectral optical image map set, SAR image map set and nDSM image map that match in geospatial coordinates and have consistent spatial resolution and range.

[0088] S6. Based on the multispectral optical image atlas, SAR image atlas and nDSM image atlas in step S5, construct a multi-temporal, multimodal remote sensing image training sample set.

[0089] Specifically, the multi-temporal, multi-modal remote sensing image training sample set includes multi-temporal, multi-modal remote sensing images and their corresponding mulch film label images and plastic greenhouse label images. In the implementation process, ArcGIS software is mainly used for interpretation, delineation, and annotation, while Python is used for slicing.

[0090] S7. Based on the multi-temporal, multi-modal remote sensing image training sample set constructed in step S6, the deep semantic segmentation model is trained until the loss function converges to a preset value, thus obtaining the target mulch film and plastic greenhouse extraction model; wherein, the preset value can be set according to actual needs.

[0091] Specifically, the deep semantic segmentation model includes modules such as an encoder, a multi-temporal multimodal feature fusion module (MTMF), a decoder, and skip connections. The architecture of each module is as follows:

[0092] a. Encoder;

[0093] The encoder includes time phase to Optical coding features, temporal phase to SAR coding features and temporal phase The nDSM encoding features, each branch contains one patch embedding layer and four encoder sub-modules. The first three encoder sub-modules each include two visual state space modules and one downsampling module, and the fourth encoder sub-module includes two visual state space modules.

[0094] Phase Any phase within the phenological period of plastic film-covered crops;

[0095] Where n is greater than or equal to 2.

[0096] b. Multi-temporal and multi-modal feature fusion module (MTMF);

[0097] The Multi-Temporal Multimodal Fusion (MTMF) module utilizes a cross-attention mechanism to enhance features and allocate weights across different temporal phases and modalities. MTMF incorporates four multi-temporal multimodal feature fusion methods:

[0098] All features are directly superimposed: Remote sensing data features of all modes across all time phases are directly superimposed to obtain temporal-modal fusion features;

[0099] Phase first, then mode: First, consider the phase separately. to Optical coding features and temporal phase to Adaptive temporal enhancement is performed on the SAR coding features to obtain multi-temporal optical fusion features and multi-temporal SAR fusion features; then the multi-temporal optical fusion features, multi-temporal SAR fusion features, and temporal enhancement are further performed. Adaptive modality enhancement is performed on the nDSM encoded features to obtain temporal-modality fusion features;

[0100] Modality first, then phase: First, separately... to Optical coding features and SAR coding features of the same phase and phase Adaptive modal enhancement is performed on the nDSM encoded features to obtain temporal phases. to The multimodal fusion features, and then the temporal phase to Adaptive temporal enhancement is performed on the multimodal fusion features to obtain temporal-modal fusion features;

[0101] Pre-phase modal fusion feature constraints followed by post-phase modal fusion: First, the temporal phases are constrained... Optical coding features and SAR coding features and temporal phase Adaptive modal enhancement is performed on the nDSM encoded features to obtain the temporal phase. Multimodal fusion features, and then temporal phase Multimodal fusion features, temporal phase Optical coding features and SAR coding features and temporal phase Adaptive modal enhancement is performed on the nDSM encoded features until the temporal phase is optimized. Multimodal fusion features, temporal phase Optical coding features and SAR coding features and temporal phase Adaptive modality enhancement is performed on the nDSM encoded features to obtain temporal-modality fusion features;

[0102] Where n is greater than or equal to 2.

[0103] The calculation method for the cross-attention mechanism is as follows:

[0104] First, cross-attention calculation is performed for each modality. The characteristics of this feature are used, with residual modes as keys and values ​​for attention computation; for temporal... The feature is that the remaining time phase is used as the key and value for attention calculation, and the calculation formula is:

[0105] ;

[0106] In the formula, For the first The results of cross-attention calculation for each modality or phase It is modal or phase The query, and It is modal or phase The transpose and values ​​of the key matrix, and It is modal or phase The transpose and values ​​of the key matrix, For the dimension of the key, It is a normalization operation, in which It is an integer greater than or equal to 1, and , It is an integer greater than or equal to 2;

[0107] Regarding the example given in this embodiment, the calculation processes for the features of mode 1, mode 2, and mode 3 are as follows:

[0108] ;

[0109] ;

[0110] ;

[0111] In the formula, , , The results of cross-attention calculations for mode 1, mode 2, and mode 3 are shown below. It is a query of mode 1. and It is the transpose and sum of the bond matrix of mode 1. It is a query in mode 2. and It is the transpose and sum of the bond matrix of mode 2. Querying in mode 3 and These are the transpose and sum of the bond matrix for mode 3. For the dimension of the key, It is a normalization operation.

[0112] The calculation methods for the two temporal cross-attention mechanisms are as follows:

[0113] ;

[0114] ;

[0115] In the formula, , The results of cross-attention calculations for time phase 1 and time phase 2 are shown below. This is a query for phase 1. and It is the transpose and sum of the bond matrix of phase 1. It is a query for phase 2. and It is the transpose and sum of the bond matrix of phase 2. For the dimension of the key, It is a normalization operation.

[0116] Secondly, the cross-attention calculation results of each modality (phase) are fused using the weighted average method to obtain the fused features.

[0117] c. Decoder;

[0118] The decoder contains four decoder submodules. The first decoder submodule includes two visual state space modules, while the other three decoder submodules each include two visual state space modules and one upsampling module.

[0119] d. Jump links;

[0120] The jump connection uses the simplest additive operation.

[0121] The loss function is:

[0122] ;

[0123] ;

[0124] ;

[0125] ;

[0126] ;

[0127] in, The total loss for the two categories, Let be the binary cross-entropy, which is the category loss function for binary classification; For multi-class total loss, Cross-entropy is the category loss function for multi-class classification; The Dice loss represents the similarity between the segmentation result and the true label; N represents the total number of samples, and C represents the total number of categories. and These represent the true label and the predicted label, respectively. It is an identifier, if the sample If it belongs to category c, the value is 1; otherwise, the value is 0. The model predicts the sample. The probability of belonging to category c; and These represent the true labels and the predicted results, respectively. , Each term controls the relative importance of different loss terms in the formula; the default value is 1.

[0128] Specifically, during the implementation process, modeling is performed using 4 / 5 of the data from the multi-temporal, multi-modal remote sensing image training sample set obtained in step S6, and validation is performed using 1 / 5 of the data.

[0129] The model was constructed by selecting the most accurate multi-temporal multi-modal feature fusion method from four different time-phase multimodal feature fusion methods after comparison, and finally obtaining the target mulch film and plastic greenhouse extraction model.

[0130] S8. Based on the target mulch film and plastic greenhouse extraction model in step S7, extract the mulch film and plastic greenhouse from the multi-temporal and multi-modal remote sensing images of the area to be studied, and obtain the spatial distribution of mulch film and plastic greenhouse in the area to be studied.

[0131] Although the invention has been described herein with reference to several illustrative embodiments, it should be understood that many other modifications and implementations can be devised by those skilled in the art, which will fall within the scope and spirit of the principles disclosed herein. More specifically, various variations and modifications can be made to the components and / or layout of the subject matter arrangement within the scope of the disclosure, drawings, and claims. Besides variations and modifications to the components and / or layout, other uses will be apparent to those skilled in the art.

Claims

1. A method for extracting agricultural plastic mulch by fusing multi-modal remote sensing and deep learning, characterized in that, The method comprises the following steps: S1, obtaining multi-temporal multi-modal remote sensing images of a region to be studied, the multi-temporal being key crop phenological periods of mulching; the multi-modal remote sensing images comprising multi-spectral optical remote sensing images, SAR remote sensing images and stereo image pair remote sensing images; The stereo image pair remote sensing images are any temporal remote sensing images in the mulching crop phenological period, including panchromatic forward-looking remote sensing images and panchromatic backward-looking remote sensing images; S2, pre-processing the multi-spectral optical remote sensing images of each temporal in S1 to obtain multi-temporal multi-spectral optical image result maps representing the real reflectivity of ground objects; S3, performing multi-view, registration, filtering, geographic coding and radiation calibration on the SAR remote sensing images of each temporal in S1 to obtain multi-temporal SAR image result maps representing the backscattering coefficients of ground objects; S4, performing stereo image pair processing on the stereo image pair remote sensing images in S1 to obtain nDSM image result maps containing vertical structure information of ground objects; The stereo image pair processing process comprises stereo matching, interpolation and smoothing processing, terrain filter for ground and object separation, smoothing filter algorithm for removing outliers and high-frequency noise, spatial interpolation tool for hole filling, spatial difference operation; S5, based on the multi-spectral optical image result maps of any temporal in S2, performing geometric registration, resampling and overlapping range extraction on the multi-spectral optical image result maps of the remaining temporals in S2, the multi-temporal SAR image result maps in S3 and the nDSM image result maps in S4 to obtain multi-spectral optical image atlases, SAR image atlases and nDSM images that are matched in geographic spatial coordinates, consistent in spatial resolution and range; S6, based on the multi-spectral optical image atlases, SAR image atlases and nDSM images in S5, constructing a multi-temporal multi-modal remote sensing image training sample set; S7, based on the multi-temporal multi-modal remote sensing image training sample set constructed in S6, training a deep semantic segmentation model until the loss function converges to a preset value to obtain a target mulch and plastic greenhouse extraction model; The deep semantic segmentation model comprises an encoder, a multi-temporal multi-modal feature fusion module MTMF, a decoder and a skip connection; The multi-temporal multi-modal fusion module MTMF performs feature enhancement and weight distribution between different temporals and modalities through a cross-attention mechanism, and is provided with four multi-temporal multi-modal feature fusion methods, which are: all features are directly superimposed, temporal first and then modal, modal first and then temporal, and pre-temporal and modal fusion feature constraint post-temporal and modal fusion; S8, based on the target mulch and plastic greenhouse extraction model in S7, extracting mulch and plastic greenhouses from input multi-temporal multi-modal remote sensing images of the region to be studied to obtain the spatial distribution of mulch and plastic greenhouses in the region to be studied.

2. The method according to claim 1, wherein the method comprises the steps of: The pre-processing in S2 comprises radiation calibration, atmospheric correction and orthorectification.

3. The method of claim 1, wherein the method comprises: The stereo image pair processing process in S4 is: The full-color forward-looking remote sensing image and the full-color backward-looking remote sensing image are stereomatched to generate three-dimensional ground point cloud data, and the point cloud data is interpolated and smoothed to generate a continuous ground elevation model DSM; The terrain filter is used to separate the ground and the object from the DSM to obtain a ground DEM and an object DEM; The smoothing filter algorithm is used to eliminate local outliers and high-frequency noise of the ground DEM to obtain a denoised ground DEM; The spatial interpolation tool is used to fill the holes in the denoised ground DEM to obtain a ground DTM; The DSM and the ground DTM are subjected to spatial difference operation to obtain an nDSM image result containing vertical structure information of the object.

4. The method of claim 1, wherein the method comprises: The multi-temporal multi-modal remote sensing image training sample set in S6 includes multi-temporal multi-modal remote sensing images and film label images and plastic greenhouse label images corresponding to the multi-temporal multi-modal remote sensing images.

5. The agricultural plastic mulch extraction method of claim 1, wherein, a. an encoder; The encoder includes a temporal phase to optical encoding features, a temporal phase to SAR encoding features and a temporal phase nDSM encoding features, each branch includes one Patch embedding layer, four encoder sub-modules, the first three encoder sub-modules each include two visual state space modules and one down-sampling module, and the fourth encoder sub-module includes two visual state space modules; the time phase any time phase within the phenological phase of the mulch crop; wherein n is greater than or equal to 2; b. a multi-temporal multi-modal feature fusion module MTMF; All features are directly superimposed: all modal remote sensing data features of all phases are directly superimposed to obtain phase-modal fusion features; First, the temporal phases are respectively enhanced by adaptive temporal enhancement on the optical encoding features of the temporal phases to and the SAR encoding features of the temporal phases to to obtain multi-temporal optical fusion features and multi-temporal SAR fusion features respectively; then, the temporal-phase-modal fusion features are obtained by adaptive modal enhancement on the multi-temporal optical fusion features, the multi-temporal SAR fusion features and the nDSM encoding features of the temporal phase . First modality, then phase: first, respectively, to to the optical encoding features of the same phase, SAR encoding features and the phase nDSM encoding features are adaptively modally enhanced, respectively, to obtain the multi-modal fusion features of the phase to to to the multi-modal fusion features of the phase are adaptively temporally enhanced to obtain the phase-modality fusion features; Pre-phase modal fusion feature constraints followed by post-phase modal fusion: First, the temporal phases are constrained... Optical coding features and SAR coding features and temporal phase Adaptive modal enhancement is performed on the nDSM encoded features to obtain the temporal phase. Multimodal fusion features, and then temporal phase Multimodal fusion features, temporal phase Optical coding features and SAR coding features and temporal phase Adaptive modal enhancement is performed on the nDSM encoded features until the temporal phase is optimized. Multimodal fusion features, temporal phase Optical coding features and SAR coding features and temporal phase Adaptive modality enhancement is performed on the nDSM encoded features to obtain temporal-modality fusion features; wherein n is greater than or equal to 2; The calculation method of the cross-attention mechanism is as follows: First, cross-attention computation is performed, for features of a modality using the remaining modalities as keys and values for the attention computation; for features of a time phase using the remaining time phases as keys and values for the attention computation, with the following computation formula: ； wherein is the cross-attention computation result for the th modality or time phase, is the query for the modality or time phase , and are the transpose and values of the key matrix for the modality or time phase , and are the transpose and values of the key matrix for the modality or time phase , is the dimension of the key, is a normalization operation, wherein is an integer greater than or equal to 1, and , is an integer greater than or equal to 2; Secondly, the weighted average method is used to fuse the cross-attention calculation results of each modality or phase to obtain fusion features; c. a decoder; The decoder includes four decoder submodules, wherein the first decoder submodule includes two visual state space modules, and the remaining three decoder submodules each include two visual state space modules and one up-sampling module; d. a skip connection; The skip connection adopts an addition operation form.

6. The method of claim 5, wherein the method further comprises: The target film and plastic greenhouse extraction model in S7 is a model constructed according to the multi-temporal multi-modal feature fusion method with the highest precision selected after comparing four multi-temporal multi-modal feature fusion methods.

7. The method of claim 1, wherein the method comprises: The loss function in S7 is: ； ； ； ； ； where, is the binary total loss, is the binary cross-entropy, which is the class loss function for binary classification; is the multi-class total loss, is the cross-entropy, which is the class loss function for multi-class classification; is the Dice loss, which represents the similarity between the segmentation result and the ground truth; N represents the total number of samples, and C represents the total number of classes; and represent the ground truth and the predicted label, respectively, is an identifier, which is equal to 1 if the sample belongs to class c, and is equal to 0 otherwise; is the probability that the model predicts the sample belongs to class c; and represent the ground truth and the predicted result, respectively; , control the relative importance of different loss terms in the formula, and the default value is 1.

Images