Multi-source remote sensing crop classification method and system based on multi-modal deep learning

Abstract

The application is suitable for the field of agricultural remote sensing technology, and provides a multi-source remote sensing crop classification method and system based on multi-modal deep learning, which comprises the following steps: calling historical multi-source remote sensing data, which is labeled with crop categories and growth stages; extracting historical feature information of each mode in the historical multi-source remote sensing data; constructing a dynamic weight learning model, the input of which is a multi-modal splicing vector, and the output is a dynamic weight vector corresponding to each growth stage of each crop category; determining the historical fusion features based on the dynamic weight vector, and constructing a classification network; collecting real-time multi-source remote sensing data, and extracting real-time feature information of each mode; randomly calling the dynamic weight vector, determining real-time fusion features based on the dynamic weight vector and the real-time feature information, inputting the real-time fusion features into the classification network, and determining the crop category. Through the construction of the dynamic weight learning model, the application can more accurately capture the feature requirements of the crop growth stage, and improve the classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of agricultural remote sensing technology, and in particular to a multi-source remote sensing crop classification method and system based on multimodal deep learning. Background Art

[0002] With the rapid development of remote sensing technology, multi-source remote sensing data provides a wealth of information for crop classification. Optical data (such as multispectral / hyperspectral imagery) can reflect the spectral characteristics of vegetation, radar data (such as fully polarimetric SAR) is sensitive to vegetation structure and soil moisture, and thermal infrared data can capture surface temperature information. However, the degree to which crops rely on each modality varies significantly at different growth stages. For example, during the seedling stage, where crop canopy cover is low, radar data, due to its high penetration, can capture soil and crop structure information. During the heading stage, where the crop canopy is dense, the spectral characteristics of optical data become crucial for classification. During the maturity stage, where crop moisture content varies significantly, surface temperature differences from thermal infrared data can be used to distinguish crop types. Traditional multimodal fusion methods typically use fixed weights, which cannot adapt to the dynamic needs of crop growth stages, resulting in limited classification accuracy. Therefore, a multi-source remote sensing crop classification method and system based on multimodal deep learning is needed to address these issues. Summary of the Invention

[0003] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a multi-source remote sensing crop classification method and system based on multimodal deep learning to solve the problems existing in the above-mentioned background technology.

[0004] The present invention is implemented as follows: a multi-source remote sensing crop classification method based on multimodal deep learning, the method comprising the following steps:

[0005] Retrieving historical multi-source remote sensing data, wherein the historical multi-source remote sensing data includes optical data, radar data, and thermal infrared data, and each historical multi-source remote sensing data is annotated with crop category and growth stage;

[0006] Extract historical feature information of each modality in historical multi-source remote sensing data, including high-dimensional spectral feature vector F1, radar feature vector F2 and thermal feature vector F3;

[0007] Construct a dynamic weight learning model with the input being the multimodal concatenation vector F0=[F1, F2, F3, crop category, growth stage] and the output being the dynamic weight vector W=[w1, w2, w3] corresponding to each growth stage of each crop category, where w1+w2+w3=1.

[0008] Based on the dynamic weight vector, the historical fusion features are determined and a classification network is constructed. The input is the historical fusion features and the output is the probability distribution of crop categories.

[0009] Collect real-time multi-source remote sensing data, pre-process the real-time multi-source remote sensing data, and extract real-time feature information of each modality;

[0010] The dynamic weight vector is randomly retrieved, and the real-time fusion feature is determined based on the dynamic weight vector and real-time feature information. The real-time fusion feature is input into the classification network to determine the crop category. After inversion verification, the crop category and growth stage are obtained after the verification is passed.

[0011] As a further solution of the present invention: the step of extracting historical feature information of each modality in historical multi-source remote sensing data specifically includes:

[0012] A three-dimensional convolutional neural network is constructed, with the input being a time series optical image block, and a 512-dimensional high-dimensional spectral feature vector F1 is extracted;

[0013] Freeman-Durden decomposition is used to decompose the fully polarized radar data into surface scattering, secondary scattering, and volume scattering components to generate three-dimensional polarization features. A Gabor filter bank is used to extract texture features. Principal component analysis is then used to fuse the polarization and texture features and reduce them to 256 dimensions to form the radar feature vector F2.

[0014] The thermal infrared data is converted into surface temperature data, and a long short-term memory network is constructed. The input is the time series surface temperature data, and the long short-term memory network hidden state is extracted as a 128-dimensional thermal feature vector F3.

[0015] As a further solution of the present invention: the step of constructing a dynamic weight learning model specifically includes:

[0016] The multimodal feature vectors F1, F2, F3 are concatenated with the crop category and growth stage to form the input vector F0;

[0017] Determine the feature encoding layer: Use the Transformer encoder to process F0, including 4 multi-head self-attention layers to capture cross-modal interactions;

[0018] Determine the weight generation layer: The fully connected layer maps the encoded features to a 3D space and generates a dynamic weight vector W=[w1,w2,w3] through the Softmax activation function, where w1+w2+w3=1;

[0019] Determine the loss function: the main loss term is the Kullback-Leibler divergence loss, and the regularization term is the L2 norm to constrain the weight amplitude to prevent overfitting.

[0020] As a further solution of the present invention: the step of determining the historical fusion features based on the dynamic weight vector and constructing the classification network specifically includes:

[0021] Perform weighted summation on the multimodal features of historical data, Ff=w1×F1⊕w2×F2⊕w3×F3, where ⊕ represents the feature concatenation operation, and the final fusion feature dimension = 512+256+128=896;

[0022] Determine the attention enhancement layer: embed the Squeeze-and-Excitation module, obtain channel attention weights through global average pooling, and strengthen key feature channels;

[0023] Determine the backbone network, based on the improvement of ResNet-50, replace the first layer 7×7 convolution with 1×1 convolution to adapt to multimodal fusion features, and maintain the residual structure in subsequent layers;

[0024] Determine the classification head: The fully connected layer maps the backbone network output to N dimensions, where N represents the number of crop categories, and outputs the probability distribution through Softmax.

[0025] As a further solution of the present invention: when randomly calling the dynamic weight vector, the potential growth stage of each crop category will be determined according to the acquisition time of the real-time multi-source remote sensing data, and some of the dynamic weight vectors that can be called will be determined. Every time a dynamic weight vector is called, the corresponding crop category and growth stage will be called at the same time; when performing inversion verification, it will be determined whether the crop category output by the classification network is the same as the crop category called before. If they are the same, the inversion verification is passed.

[0026] As a further solution of the present invention: when preprocessing multi-source remote sensing data, it is necessary to convert the original digital values ​​of the optical image into apparent reflectivity to eliminate sensor gain bias; and use a method based on the radiation transfer model to remove the influence of atmospheric molecules and aerosol scattering to obtain surface reflectivity; for radar data, the radar backscatter coefficient is converted into a normalized value to eliminate system noise, and the geometric distortion caused by terrain undulation is compensated based on the DEM.

[0027] Another object of the present invention is to provide a multi-source remote sensing crop classification system based on multimodal deep learning, the system comprising:

[0028] A historical remote sensing data module is used to retrieve historical multi-source remote sensing data, including optical data, radar data, and thermal infrared data. Each historical multi-source remote sensing data is annotated with crop category and growth stage;

[0029] The historical feature information module is used to extract the historical feature information of each mode in the historical multi-source remote sensing data. The historical feature information includes the high-dimensional spectral feature vector F1, the radar feature vector F2 and the thermal feature vector F3;

[0030] The dynamic weight learning module is used to build a dynamic weight learning model. The input is the multimodal concatenation vector F0=[F1, F2, F3, crop category, growth stage], and the output is the dynamic weight vector W=[w1, w2, w3] corresponding to each growth stage of each crop category, where w1+w2+w3=1.

[0031] The classification network construction module is used to determine the historical fusion features based on the dynamic weight vector and construct the classification network. The input is the historical fusion features and the output is the probability distribution of crop categories.

[0032] Real-time data processing module, used to collect real-time multi-source remote sensing data, pre-process the real-time multi-source remote sensing data, and extract real-time feature information of each modality;

[0033] The crop classification module is used to randomly retrieve dynamic weight vectors, determine real-time fusion features based on the dynamic weight vectors and real-time feature information, input the real-time fusion features into the classification network, determine the crop category, perform inversion verification, and after verification, obtain the crop category and growth stage.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] This invention achieves adaptive modal fusion by constructing a dynamic weight learning model that automatically adjusts the weights of each modality based on crop type and growth stage. Compared to traditional fixed weights, dynamic weights can more accurately capture the characteristic requirements of crop growth stages, improving classification accuracy. By randomly retrieving dynamic weight vectors and combining them with real-time features for classification, and using inversion verification to ensure consistency between the results and the training phase, this inversion verification mechanism effectively avoids model drift and ensures the reliability and interpretability of classification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Flowchart of the multi-source remote sensing crop classification method based on multimodal deep learning.

[0037] Figure 2 Flowchart for extracting feature information from multi-source remote sensing crop classification method based on multimodal deep learning.

[0038] Figure 3 Flowchart for constructing a dynamic weight learning model for multi-source remote sensing crop classification based on multimodal deep learning.

[0039] Figure 4 Flowchart for constructing a classification network for a multi-source remote sensing crop classification method based on multimodal deep learning.

[0040] Figure 5 Schematic diagram of the structure of the multi-source remote sensing crop classification system based on multimodal deep learning. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0043] like Figure 1 As shown, an embodiment of the present invention provides a multi-source remote sensing crop classification method based on multimodal deep learning, the method comprising the following steps:

[0044] S100, retrieving historical multi-source remote sensing data, wherein the historical multi-source remote sensing data includes optical data, radar data, and thermal infrared data, and each historical multi-source remote sensing data is annotated with crop category and growth stage;

[0045] S200, extracting historical feature information of each modality in historical multi-source remote sensing data, the historical feature information including a high-dimensional spectral feature vector F1, a radar feature vector F2, and a thermal feature vector F3;

[0046] S300: Build a dynamic weight learning model, with the input being the multimodal concatenation vector F0 = [F1, F2, F3, crop category, growth stage], and the output being the dynamic weight vector W = [w1, w2, w3] corresponding to each growth stage of each crop category, where w1 + w2 + w3 = 1;

[0047] S400, based on the dynamic weight vector, determines the historical fusion features and constructs a classification network, with the historical fusion features as input and the crop category probability distribution as output;

[0048] S500, collecting real-time multi-source remote sensing data, pre-processing the real-time multi-source remote sensing data, and extracting real-time feature information of each modality;

[0049] S600 randomly retrieves a dynamic weight vector, determines a real-time fusion feature based on the dynamic weight vector and real-time feature information, inputs the real-time fusion feature into the classification network, determines the crop category, performs inversion verification, and after verification, obtains the crop category and growth stage.

[0050] It should be noted that existing methods (such as feature concatenation and weighted averaging) assign fixed weights to each modal data type and are unable to adapt to the dynamic changes in crop growth stages. For example, corn seedlings rely more on the penetration of radar data, while the heading stage relies more on the spectral characteristics of optical data. Fixed weights cannot reflect this difference. Traditional methods typically process single-phase data independently and fail to fully utilize the temporal information of crop growth. For example, the spectral, radar, and thermal infrared characteristics of crops at different growth stages exhibit significant temporal variations. Ignoring this characteristic can lead to unstable classification results. The embodiments of the present invention are intended to address the above-mentioned problems.

[0051] In this embodiment of the present invention, a large amount of historical multi-source remote sensing data is stored. This data includes optical data, radar data, and thermal infrared data. Each historical multi-source remote sensing data set is annotated with crop type, growth stage, and acquisition time. Optical data includes visible light bands (blue, green, and red), which reflect vegetation pigment content (such as chlorophyll a / b and carotenoids) and are used to monitor vegetation health; near-infrared bands (NIR), which are sensitive to vegetation canopy structure; high reflectivity is a typical characteristic of healthy vegetation; and short-wave infrared bands (SWIR), which reflect vegetation moisture content; reflectivity increases significantly under drought stress. The optical data can be used to derive the Normalized Difference Vegetation Index (NDVI) and the Enhanced Vegetation Index (EDVI). Radar data can be used to derive the backscatter coefficient, which is affected by vegetation moisture content, canopy structure, and soil roughness. Thermal infrared data can be used to derive the Land Surface Temperature (LST). Next, historical feature information for each modality in the historical multi-source remote sensing data needs to be extracted. This historical feature information includes the high-dimensional spectral feature vector F1, the radar feature vector F2, and the thermal feature vector F3. This allows the construction of a dynamic weight learning model. The input is the multimodal concatenation vector F0 = [F1, F2, F3, crop category, growth stage]. The crop category is one-hot encoded, and the growth stage is ordinal encoded. The output is a dynamic weight vector W = [w1, w2, w3] corresponding to each growth stage of each crop category. This dynamic weight learning model automatically adjusts the weights of each modality based on the crop category and growth stage (for example, increasing radar weight at the seedling stage and optical weight at the heading stage), achieving adaptive modal fusion. The dynamic weight vector is then used to determine the historical fusion features. A classification network is then constructed, which takes the historical fusion features as input and outputs a probability distribution of crop categories, thereby determining the most likely crop category. When acquiring real-time multi-source remote sensing data, preprocessing is performed to extract real-time feature information (i.e., feature vectors) for each modality. Dynamic weight vectors are then randomly retrieved. The potential growth stage of each crop category is determined based on the acquisition time of the real-time multi-source remote sensing data. Potential growth stages refer to all possible growth stages at the time of acquisition. This allows the determination of a subset of available dynamic weight vectors. Each time a dynamic weight vector is retrieved, the corresponding crop category and growth stage are retrieved. A real-time fused feature is then determined based on the dynamic weight vector and real-time feature information. This fused feature is then fed into the classification network to determine the crop category. Inversion verification is then performed. The crop category output by the classification network is then checked to see if it matches the previously retrieved category. If so, inversion verification passes, resulting in the crop category and growth stage. In this manner, by randomly retrieving dynamic weight vectors and combining them with real-time features for classification, inversion verification ensures consistency between the results and those obtained during training. This inversion verification mechanism effectively prevents model drift and ensures the reliability and interpretability of classification results.

[0052] like Figure 2 As shown, as a preferred embodiment of the present invention, the step of extracting historical feature information of each modality in historical multi-source remote sensing data specifically includes:

[0053] S201, constructing a three-dimensional convolutional neural network, taking the time series optical image blocks as input, and extracting a 512-dimensional high-dimensional spectral feature vector F1;

[0054] S202: Freeman-Durden decomposition is used to decompose the fully polarized radar data into surface scattering, secondary scattering, and volume scattering components to generate three-dimensional polarization features. A Gabor filter bank is used to extract texture features. The polarization and texture features are fused through principal component analysis and reduced to 256 dimensions to form the radar feature vector F2.

[0055] S203, converting the thermal infrared data into surface temperature data, constructing a long short-term memory network, inputting the time series surface temperature data, and extracting the long short-term memory network hidden state as a 128-dimensional thermal feature vector F3.

[0056] In this embodiment of the present invention, multi-source remote sensing data must be preprocessed before extracting feature vectors. First, the raw digital values ​​of the optical image are converted into apparent reflectance to eliminate sensor gain bias. A radiative transfer model-based approach (such as FLAASH or the 6S model) is then used to remove the effects of atmospheric molecules and aerosol scattering to obtain surface reflectance. Orthorectification is also performed using a digital elevation model (DEM) and rational polynomial coefficients (RPC) to eliminate geometric distortion caused by terrain and sensor viewing angle. For radar data, the radar backscatter coefficients are converted to normalized values ​​to eliminate system noise, and geometric distortion caused by terrain undulation is compensated for using the DEM. For thermal infrared data, a split-window algorithm or a single-channel algorithm is used to convert the thermal infrared data into land surface temperature (LST). Cloud contaminated areas are identified using a threshold method, and missing values ​​are filled using spatiotemporal interpolation.

[0057] After preprocessing, a 3D convolutional neural network is constructed, taking as input a time series of optical image patches (size: 7×7×12, time step × spatial window × spectral band). The network consists of four convolutional layers, each followed by batch normalization and Reluctant Unit (ReLU) activation. The convolution kernel size is gradually reduced (e.g., 3×3×3 for the first layer and 1×1×1 for the last layer). Ultimately, a 512-dimensional high-dimensional spectral feature vector F1 is extracted. Furthermore, Freeman-Durden decomposition is used to decompose the fully polarimetric radar data into surface scattering, secondary scattering, and volume scattering components to generate 3D polarimetric features. A Gabor filter bank is used to extract texture features to capture periodic variations in crop structure. Principal component analysis is used to fuse the polarimetric and texture features and reduce the dimension to 256, forming the radar feature vector F2. Finally, a long short-term memory (LSTM) network is constructed, taking as input a 7-day time series of surface temperature data to capture the diurnal temperature variations caused by crop transpiration. The LSTM hidden state is extracted as a 128-dimensional thermal feature vector F3. By introducing time series factors into feature extraction, we can use multi-temporal data to capture the dynamic changes in crop growth. Through time series correlation analysis, we can reduce the noise interference of single-temporal data and improve the stability and robustness of classification results.

[0058] like Figure 3 As shown, as a preferred embodiment of the present invention, the step of constructing a dynamic weight learning model specifically includes:

[0059] S301, concatenating the multimodal feature vectors F1, F2, F3 with the crop category and growth stage into an input vector F0;

[0060] S302, determine the feature encoding layer: use the Transformer encoder to process F0, including 4 multi-head self-attention layers to capture cross-modal interaction relationships;

[0061] S303, determine the weight generation layer: the fully connected layer maps the encoded features to a 3D space and generates a dynamic weight vector W=[w1,w2,w3] through the Softmax activation function;

[0062] S304, determine the loss function: the main loss term is Kullback-Leibler divergence loss, and the regularization term is L2 norm constraint weight amplitude to prevent overfitting.

[0063] In the embodiment of the present invention, after the multimodal splicing vector F0 is determined, the Transformer encoder is used to process F0. The feature encoding layer contains 4 multi-head self-attention layers (each head has a dimension of 64) to capture cross-modal interaction relationships. Then, the fully connected layer maps the encoded features to a 3D space and generates a dynamic weight vector W=[w1,w2,w3] through the Softmax activation function, satisfying w1+w2+w3=1. The main loss term of the loss function is the Kullback-Leibler (KL) divergence loss L KL , the regularization term is the L2 norm constraint weight amplitude L reg , to prevent overfitting, total loss = L KL +L reg The training optimizer is AdamW, which gradually introduces training data in the order of growth stages (germination stage → harvest stage) to improve model stability.

[0064] like Figure 4 As shown, as a preferred embodiment of the present invention, the step of determining the historical fusion features based on the dynamic weight vector and constructing the classification network specifically includes:

[0065] S401: Perform weighted summation on the multimodal features of the historical data, Ff = w1×F1⊕w2×F2⊕w3×F3, where ⊕ represents the feature concatenation operation. The final fusion feature dimension = 512+256+128=896;

[0066] S402, determine the attention enhancement layer: embed the Squeeze-and-Excitation module, obtain the channel attention weight through global average pooling, and strengthen the key feature channels;

[0067] S403: Determine the backbone network. Based on the improvement of ResNet-50, replace the first layer 7×7 convolution with 1×1 convolution to adapt to the multimodal fusion features, and maintain the residual structure in the subsequent layers.

[0068] S404, determine the classification head: the fully connected layer maps the backbone network output to N dimensions, where N represents the number of crop categories, and outputs the probability distribution through Softmax.

[0069] In this embodiment of the present invention, the fusion feature Ff = w1×F1⊕w2×F2⊕w3×F3 is determined. The classification network architecture includes an attention enhancement layer, a backbone network, and a classification head. Evaluation metrics for the classification network include: 1. Macro-average F1 score: This mitigates class imbalance. 2. Growth stage transfer accuracy: This verifies the model's robustness to phenological changes.

[0070] like Figure 5 As shown, an embodiment of the present invention further provides a multi-source remote sensing crop classification system based on multimodal deep learning, the system comprising:

[0071] A historical remote sensing data module 100 is used to retrieve historical multi-source remote sensing data, including optical data, radar data, and thermal infrared data, each of which is annotated with crop category and growth stage;

[0072] A historical feature information module 200 is used to extract historical feature information of each modality in historical multi-source remote sensing data, wherein the historical feature information includes a high-dimensional spectral feature vector F1, a radar feature vector F2, and a thermal feature vector F3;

[0073] The dynamic weight learning module 300 is used to build a dynamic weight learning model. The input is the multimodal concatenation vector F0 = [F1, F2, F3, crop category, growth stage], and the output is the dynamic weight vector W = [w1, w2, w3] corresponding to each growth stage of each crop category, where w1 + w2 + w3 = 1;

[0074] A classification network construction module 400 is used to determine historical fusion features based on a dynamic weight vector and construct a classification network, with the historical fusion features as input and the crop category probability distribution as output;

[0075] The real-time data processing module 500 is used to collect real-time multi-source remote sensing data, pre-process the real-time multi-source remote sensing data, and extract real-time feature information of each modality;

[0076] The crop classification module 600 is used to randomly retrieve dynamic weight vectors, determine real-time fusion features based on the dynamic weight vectors and real-time feature information, input the real-time fusion features into the classification network, determine the crop category, perform inversion verification, and after verification, obtain the crop category and growth stage.

[0077] As a preferred embodiment of the present invention, the historical feature information module 200 includes:

[0078] The spectral feature vector unit is used to construct a three-dimensional convolutional neural network. The input is a time series optical image block, and a 512-dimensional high-dimensional spectral feature vector F1 is extracted;

[0079] The radar feature vector unit is used to decompose the fully polarized radar data into surface scattering, secondary scattering, and volume scattering components using Freeman-Durden decomposition to generate three-dimensional polarization features. The texture features are extracted using a Gabor filter bank. The polarization and texture features are fused and reduced to 256 dimensions through principal component analysis to form the radar feature vector F2.

[0080] The thermal feature vector unit is used to convert thermal infrared data into surface temperature data, build a long short-term memory network, input the time series surface temperature data, and extract the long short-term memory network hidden state as a 128-dimensional thermal feature vector F3.

[0081] As a preferred embodiment of the present invention, the dynamic weight learning module 300 includes:

[0082] An input vector determination unit, configured to concatenate the multimodal feature vectors F1, F2, and F3 with the crop category and growth stage into an input vector F0;

[0083] Feature encoding layer unit, used to determine the feature encoding layer: using Transformer encoder to process F0, including 4 multi-head self-attention layers to capture cross-modal interaction relationships;

[0084] The weight generation layer unit is used to determine the weight generation layer: the fully connected layer maps the encoded features to a 3D space and generates a dynamic weight vector W=[w1,w2,w3] through the Softmax activation function, where w1+w2+w3=1;

[0085] The loss function unit is used to determine the loss function: the main loss term is the Kullback-Leibler divergence loss, and the regularization term is the L2 norm to constrain the weight amplitude to prevent overfitting.

[0086] As a preferred embodiment of the present invention, the classification network construction module 400 includes:

[0087] The feature concatenation unit is used to perform weighted summation of the multimodal features of historical data. Ff = w1×F1⊕w2×F2⊕w3×F3, where ⊕ represents the feature concatenation operation. The final fusion feature dimension = 512+256+128=896.

[0088] The attention enhancement layer unit is used to determine the attention enhancement layer: embedding the Squeeze-and-Excitation module, obtaining channel attention weights through global average pooling, and strengthening key feature channels;

[0089] The backbone network unit is used to determine the backbone network. It is based on the improvement of ResNet-50. The first layer 7×7 convolution is replaced by 1×1 convolution to adapt to the multimodal fusion features. The subsequent layers maintain the residual structure.

[0090] The classification head unit is used to determine the classification head: the fully connected layer maps the backbone network output to N dimensions, where N represents the number of crop categories, and outputs the probability distribution through Softmax.

[0091] The above is only a detailed description of the preferred embodiments of the present invention, which is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0092] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0093] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When executed, the program can include the processes of the above-described method embodiments. Any reference to memory, storage, database, or other media used in the various embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0094] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A multi-source remote sensing crop classification method based on multimodal deep learning, characterized by: The method comprises the following steps: Retrieving historical multi-source remote sensing data, wherein the historical multi-source remote sensing data includes optical data, radar data, and thermal infrared data, and each historical multi-source remote sensing data is annotated with crop category and growth stage; Extract historical feature information of each modality in historical multi-source remote sensing data, including high-dimensional spectral feature vector F1, radar feature vector F2 and thermal feature vector F3; Construct a dynamic weight learning model with the input being the multimodal concatenation vector F0=[F1, F2, F3, crop category, growth stage] and the output being the dynamic weight vector W=[w1, w2, w3] corresponding to each growth stage of each crop category, where w1+w2+w3=1. Based on the dynamic weight vector, the historical fusion features are determined and a classification network is constructed. The input is the historical fusion features and the output is the probability distribution of crop categories. Collect real-time multi-source remote sensing data, pre-process the real-time multi-source remote sensing data, and extract real-time feature information of each modality; Randomly retrieve dynamic weight vectors, determine real-time fusion features based on the dynamic weight vectors and real-time feature information, input the real-time fusion features into the classification network, determine the crop category, perform inversion verification, and after verification, obtain the crop category and growth stage; Among them, when randomly calling the dynamic weight vector, the potential growth stage of each crop category will be determined according to the acquisition time of real-time multi-source remote sensing data, and some of the dynamic weight vectors that can be called will be determined. Every time a dynamic weight vector is called, the corresponding crop category and growth stage will be called at the same time; when performing inversion verification, it will be determined whether the crop category output by the classification network is the same as the crop category called before. If they are the same, the inversion verification passes.

2. The multi-source remote sensing crop classification method based on multimodal deep learning according to claim 1 is characterized in that: The step of extracting historical feature information of each modality in the historical multi-source remote sensing data specifically includes: A three-dimensional convolutional neural network is constructed, with the input being a time series optical image block, and a 512-dimensional high-dimensional spectral feature vector F1 is extracted; Freeman-Durden decomposition is used to decompose the fully polarized radar data into surface scattering, secondary scattering, and volume scattering components to generate three-dimensional polarization features. A Gabor filter bank is used to extract texture features. Principal component analysis is then used to fuse the polarization and texture features and reduce them to 256 dimensions to form the radar feature vector F2. The thermal infrared data is converted into surface temperature data, and a long short-term memory network is constructed. The input is the time series surface temperature data, and the long short-term memory network hidden state is extracted as a 128-dimensional thermal feature vector F3.

3. The multi-source remote sensing crop classification method based on multimodal deep learning according to claim 1 is characterized in that: The steps of constructing a dynamic weight learning model specifically include: The multimodal feature vectors F1, F2, F3 are concatenated with the crop category and growth stage to form the input vector F0; Determine the feature encoding layer: Use the Transformer encoder to process F0, including 4 multi-head self-attention layers to capture cross-modal interactions; Determine the weight generation layer: The fully connected layer maps the encoded features to a 3D space and generates a dynamic weight vector W=[w1,w2,w3] through the Softmax activation function, where w1+w2+w3=1; Determine the loss function: the main loss term is the Kullback-Leibler divergence loss, and the regularization term is the L2 norm to constrain the weight amplitude to prevent overfitting.

4. The multi-source remote sensing crop classification method based on multimodal deep learning according to claim 2 is characterized in that: The step of determining the historical fusion features based on the dynamic weight vector and constructing the classification network specifically includes: Perform weighted summation on the multimodal features of historical data, Ff=w1×F1⊕w2×F2⊕w3×F3, where ⊕ represents the feature concatenation operation, and the final fusion feature dimension = 512+256+128=896; Determine the attention enhancement layer: embed the Squeeze-and-Excitation module, obtain channel attention weights through global average pooling, and strengthen key feature channels; Determine the backbone network, based on the improvement of ResNet-50, replace the first layer 7×7 convolution with 1×1 convolution to adapt to multimodal fusion features, and maintain the residual structure in subsequent layers; Determine the classification head: The fully connected layer maps the backbone network output to N dimensions, where N represents the number of crop categories, and outputs the probability distribution through Softmax.

5. The multi-source remote sensing crop classification method based on multimodal deep learning according to claim 1, characterized in that: When preprocessing multi-source remote sensing data, it is necessary to convert the original digital values ​​of the optical image into apparent reflectance to eliminate the sensor gain bias; and use a method based on the radiation transfer model to remove the influence of atmospheric molecules and aerosol scattering to obtain the surface reflectance; For radar data, the radar backscatter coefficient is converted into a normalized value, the systematic noise is eliminated, and the geometric distortion caused by the terrain undulation is compensated based on the DEM.

6. A multi-source remote sensing crop classification system based on multimodal deep learning, characterized by: The system comprises: A historical remote sensing data module is used to retrieve historical multi-source remote sensing data, including optical data, radar data, and thermal infrared data. Each historical multi-source remote sensing data is annotated with crop category and growth stage; The historical feature information module is used to extract the historical feature information of each mode in the historical multi-source remote sensing data. The historical feature information includes the high-dimensional spectral feature vector F1, the radar feature vector F2 and the thermal feature vector F3; The dynamic weight learning module is used to build a dynamic weight learning model. The input is the multimodal concatenation vector F0=[F1, F2, F3, crop category, growth stage], and the output is the dynamic weight vector W=[w1, w2, w3] corresponding to each growth stage of each crop category, where w1+w2+w3=1. The classification network construction module is used to determine the historical fusion features based on the dynamic weight vector and construct the classification network. The input is the historical fusion features and the output is the probability distribution of crop categories. Real-time data processing module, used to collect real-time multi-source remote sensing data, pre-process the real-time multi-source remote sensing data, and extract real-time feature information of each modality; The crop classification module is used to randomly retrieve dynamic weight vectors, determine real-time fusion features based on the dynamic weight vectors and real-time feature information, input the real-time fusion features into the classification network, determine the crop category, perform inversion verification, and after verification, obtain the crop category and growth stage; Among them, when randomly calling the dynamic weight vector, the potential growth stage of each crop category will be determined according to the acquisition time of real-time multi-source remote sensing data, and some of the dynamic weight vectors that can be called will be determined. Every time a dynamic weight vector is called, the corresponding crop category and growth stage will be called at the same time; when performing inversion verification, it will be determined whether the crop category output by the classification network is the same as the crop category called before. If they are the same, the inversion verification passes.

7. The multi-source remote sensing crop classification system based on multimodal deep learning according to claim 6, characterized in that: The historical feature information module includes: The spectral feature vector unit is used to construct a three-dimensional convolutional neural network. The input is a time series optical image block, and a 512-dimensional high-dimensional spectral feature vector F1 is extracted; The radar feature vector unit is used to decompose the fully polarized radar data into surface scattering, secondary scattering, and volume scattering components using Freeman-Durden decomposition to generate three-dimensional polarization features. The texture features are extracted using a Gabor filter bank. The polarization and texture features are fused and reduced to 256 dimensions through principal component analysis to form the radar feature vector F2. The thermal feature vector unit is used to convert thermal infrared data into surface temperature data, build a long short-term memory network, input the time series surface temperature data, and extract the long short-term memory network hidden state as a 128-dimensional thermal feature vector F3.

8. The multi-source remote sensing crop classification system based on multimodal deep learning according to claim 6, characterized in that: The dynamic weight learning module includes: An input vector determination unit, configured to concatenate the multimodal feature vectors F1, F2, and F3 with the crop category and growth stage into an input vector F0; Feature encoding layer unit, used to determine the feature encoding layer: using Transformer encoder to process F0, including 4 multi-head self-attention layers to capture cross-modal interaction relationships; The weight generation layer unit is used to determine the weight generation layer: the fully connected layer maps the encoded features to a 3D space and generates a dynamic weight vector W=[w1,w2,w3] through the Softmax activation function, where w1+w2+w3=1; The loss function unit is used to determine the loss function: the main loss term is the Kullback-Leibler divergence loss, and the regularization term is the L2 norm to constrain the weight amplitude to prevent overfitting.

9. The multi-source remote sensing crop classification system based on multimodal deep learning according to claim 7, characterized in that: The classification network building module includes: The feature concatenation unit is used to perform weighted summation of the multimodal features of historical data. Ff = w1×F1⊕w2×F2⊕w3×F3, where ⊕ represents the feature concatenation operation. The final fusion feature dimension = 512+256+128=896. The attention enhancement layer unit is used to determine the attention enhancement layer: embedding the Squeeze-and-Excitation module, obtaining channel attention weights through global average pooling, and strengthening key feature channels; The backbone network unit is used to determine the backbone network. It is based on the improvement of ResNet-50. The first layer 7×7 convolution is replaced by 1×1 convolution to adapt to the multimodal fusion features. The subsequent layers maintain the residual structure. The classification head unit is used to determine the classification head: the fully connected layer maps the backbone network output to N dimensions, where N represents the number of crop categories, and outputs the probability distribution through Softmax.

Images