Multi-modal remote sensing tea garden automatic identification method and system integrating phenolic indexes and phenological characteristics

Abstract

The invention provides a multi-mode remote sensing tea garden automatic identification method and system integrating phenolic indexes and phenological characteristics. According to the method, the growth amplitude GA reflecting tea phenological characteristics is calculated through medium-resolution time sequence radar data, and the normalized vegetation index NDVI, the phenolic compound index PCI and the phenol high-value dominant index PHD are calculated from optical remote sensing images so as to represent the growth condition of the tea tree and the change rule of the phenol content. A double-branch lightweight deep learning model is constructed, a first branch processes a high-resolution remote sensing image, and a second branch processes multi-modal tea garden remote sensing features; and then multi-modal geoscience knowledge of the tea garden is fully extracted and fused through a double-branch fusion module, refined and automatic extraction of large-area tea garden distribution is realized, and key technical support is provided for tea garden resource monitoring, management and the like.

Description

TECHNICAL FIELD

[0001] The application provides a multi-modal remote sensing tea garden automatic recognition method and system integrating phenol index and phenological characteristics, and relates to the technical field of fine tea garden mapping. BACKGROUND

[0002] Tea garden is an important agricultural resource, and its spatial distribution information is of great significance for resource survey, precision agriculture and ecological environment monitoring. With the development of deep learning technology, remote sensing image automatic recognition methods based on convolutional neural networks, attention mechanisms and other technologies have gradually become a research hotspot. Through semantic segmentation technology, precise recognition and segmentation of large-area crops can be achieved. In recent years, the development of remote sensing technology has also provided a new means for large-area tea garden recognition. Optical remote sensing images can reflect the spectral characteristics of ground objects and are widely used in thematic information extraction. Radar images can obtain ground object scattering information under cloud and fog blocking conditions and have the advantages of all-weather and all-day. However, a single remote sensing data source often cannot fully reflect the spectral and phenological characteristics of tea gardens during the growth process. Therefore, there is an urgent need for a tea garden automatic recognition method that can effectively fuse multi-source remote sensing data features and balance recognition accuracy and model lightweight, to achieve efficient, accurate and automated extraction of large-scale tea gardens and provide reliable technical support for tea garden resource investigation and agricultural management. SUMMARY

[0003] The application aims to solve the problem of fine identification of tea gardens in complex hilly areas and provides a multi-modal remote sensing tea garden automatic recognition method and system integrating phenol index and phenological characteristics. The application calculates the growth amplitude GA reflecting the phenological characteristics of tea leaves from medium-resolution time-series radar data, and calculates the normalized difference vegetation index NDVI, phenolic compound index PCI and phenolic high-value dominance index PHD from optical remote sensing images to represent the growth status of tea trees and the variation of phenolic content. A dual-branch lightweight deep learning model is constructed, the first branch processes high-resolution remote sensing images, and the second branch processes multi-modal tea garden remote sensing features. Then, through a dual-branch fusion module, the multi-modal geosciences knowledge of tea gardens is fully extracted and fused to realize fine and automated extraction of large-area tea garden distribution and provide key technical support for tea garden resource monitoring and management.

[0004] Therefore, in order to make up for the shortcomings of the prior art, the application provides a multi-modal remote sensing tea garden automatic recognition method and system integrating phenol index and phenological characteristics, which includes the following contents.

[0005] The application provides a multi-modal remote sensing tea garden automatic recognition method integrating phenol index and phenological characteristics, which is characterized by realizing automatic recognition of tea gardens by analyzing phenol index and phenological characteristics. The multi-modal remote sensing tea garden automatic recognition method integrating phenol index and phenological characteristics includes the following contents:

[0006] Step S1: Obtain medium-resolution intra-annual time-series radar images in the study area, pre-process the images, and calculate the growth amplitude reflecting the phenological characteristics of tea;

[0007] Step S2: Obtain medium-resolution intra-annual time-series optical images in the study area, pre-process the images, and calculate features related to tea growth conditions and substance content, including normalized vegetation index, phenolic compound index, and phenolic high-value dominance index;

[0008] Step S3: Obtain high-resolution optical remote sensing images of the same year in the study area, pre-process the images, and combine the multi-modal features calculated in steps S1 and S2 as sample image data. Use a sliding window to crop the sample images and manually labeled labels to obtain a sample dataset of HxW pixels. Divide the training sample set, validation sample set, and test sample set according to a certain proportion;

[0009] Step S4: Construct a multi-scale lightweight model with a fusion attention mechanism as the first branch, input the high-resolution remote sensing image into the model, and deeply mine the semantic information in the high-resolution remote sensing image;

[0010] Step S5: Construct a semantic segmentation model with multi-modal information aggregation as the second branch, input the features extracted in steps S1 and S2 into the model, and extract multi-modal tea garden knowledge features;

[0011] Step S6: Construct a dual-branch feature fusion module to fuse the high-resolution image features and multi-modal tea garden knowledge features;

[0012] Step S7: Train the model using the training sample set obtained in step S3. Calculate the error of the model prediction using a custom composite loss function, adjust the model parameters based on the optimizer, use the cosine annealing learning rate as the learning rate decay strategy, and train the model. Adjust the hyperparameters using the validation sample set to monitor whether overfitting occurs, and output the model weights;

[0013] Step S8: Based on the model weights obtained by training and adjusting in step S7, use the weights for the test sample set constructed in step S3 to evaluate the generalization ability of the model, and use the weights to realize automatic identification of large-area tea garden information.

[0014] Further, step S1 includes the following content:

[0015] Step S11: Obtain ground range detection data of intra-annual time-series radar images in interferometric wide swath mode, where the ground range detection data has been processed by thermal noise removal, orbit file correction, radiation scaling, and other processes. Then, perform filtering correction and terrain correction through a custom data processing function to finally obtain backscatter information;

[0016] Step S12: In order to reflect the growth rule of the tea garden in a year, the time sequence curve of the image value of the tea garden in the time sequence radar image in a year is analyzed, and the growth amplitude reflecting the phenological characteristics of tea is constructed. The growth amplitude is the difference between the image value of the peak value of the tea growth period and the low value of the frost period, and the calculation formula is as follows:

[0017] GA = VH 生长期峰值 -VH 霜冻期低值

[0018] Wherein, GA represents the growth amplitude, and VH represents the image value.

[0019] Further, step S2 includes the following contents:

[0020] Step S21: Obtain the time sequence optical image product in a year, wherein the time sequence optical image product in a year has been processed by radiation calibration and atmospheric correction; wherein the optical image product is clouded by using quality evaluation QA60 band and cloud probability product;

[0021] Step S22: Calculate the normalized vegetation index of the study area, and obtain the image with the best vegetation condition in a year by using the normalized vegetation index maximum synthesis method. The calculation formula of the normalized vegetation index is as follows:

[0022]

[0023] Wherein, NDVI represents the normalized vegetation index, NIR represents the reflectivity of the near-infrared band, and RED represents the reflectivity of the red light band.

[0024] Step S23: Calculate the phenolic compound index of the study area, and obtain the image with the best phenolic condition in a year by using the phenolic compound index maximum synthesis method. The calculation formula of the phenolic compound index is as follows:

[0025]

[0026] Wherein, PCI represents the phenolic compound index, GREEN represents the reflectivity of the green light band, and SWIR2 represents the reflectivity of the short wave infrared band.

[0027] Step S24: Calculate the enhanced vegetation index EVI2 of the study area, and the calculation formula of the enhanced vegetation index is as follows:

[0028]

[0029] Wherein, EVI2 represents the enhanced vegetation index, NIR represents the reflectivity of the near-infrared band, and RED represents the reflectivity of the red light band.

[0030] Step S25: To reflect the change rule of phenolic content in tea garden within a year, a high-value dominant index of phenolic compounds is constructed using the phenolic compound index and the enhanced vegetation index, 50% of the enhanced vegetation index is used to identify the high-value interval T, the high-value dominant index of phenolic compounds is calculated through the time series of the phenolic compound index in the interval, the high-value dominant index of phenolic compounds represents the average value of the time curve of the phenolic compound index in the high-value interval, and the calculation formula of the high-value dominant index of phenolic compounds is as follows:

[0031]

[0032] Wherein PHD represents the high-value dominant index of phenolic compounds, p k represents the phenolic compound index value at the corresponding time point k of a high value, T represents the set of time points of the high value interval, and |T| represents the length of the high value interval.

[0033] Further, step S3 includes the following contents:

[0034] Step S31: Resample the multi-modal tea garden knowledge features to the same spatial resolution as the high-resolution image, and perform band synthesis with the RGB band of the high-resolution remote sensing image to obtain the initial sample image dataset of the study area; wherein the multi-modal tea garden knowledge features include normalized vegetation index, phenolic compound index, phenolic compound index and growth amplitude;

[0035] Step S32: Use the face vector construction method built-in the software to make tea garden labels, assign 1 to tea garden and 0 to non-tea garden, and convert the tea garden vector label to a grid to obtain the initial sample label dataset in the study area;

[0036] Step S33: Based on the initial tea garden sample dataset, the image and label sample data are divided into HxW pixel size sample datasets through sliding window clipping, and data enhancement operations are adopted to increase the diversity of samples and improve the generalization ability of the model; wherein the data enhancement operations include horizontal flip, vertical flip, diagonal mirror image and chroma enhancement;

[0037] Step S34: Divide the training sample set, the verification sample set and the test sample set according to a certain proportion.

[0038] Further, step S4 includes the following contents:

[0039] Step S41: Based on the encoder-decoder architecture, the encoder is composed of convolutional blocks and tokenization multi-layer perception, the features of the input high-resolution remote sensing image are extracted, and maximum pooling and block embedding are used for down-sampling of the feature map;

[0040] Step S42: The decoder is composed of a tokenization multi-layer perception and a double convolution block, and adopts a transpose convolution to up-sample the feature map, and the encoder and the decoder are combined through a skip connection;

[0041] Step S43: A double attention guide module is embedded in the skip connection, which is composed of a channel attention mechanism module and a spatial attention mechanism module, to simulate the channel and spatial area dependency in the feature map, improve the loss of important features in the continuous down-sampling process of convolution and pooling operations, and effectively focus on important features of the tea garden in the high-resolution image;

[0042] Step S44: To improve the distinguishability of semantic representation, a multi-scale feature extraction module is designed to aggregate decoding features of different depths and scales on the decoder subnetwork, process the decoder features of each layer to a unified feature size, and finally fuse them in a connection layer to obtain high-resolution image features with multi-scale feature expression. Through this branch, information in high-resolution remote sensing images can be deeply mined, and visual expression of key features in the tea garden can be captured.

[0043] Further, step S5 includes the following contents:

[0044] Step S51: The encoder is composed of multiple double convolution modules and a series of max pooling modules, and the main features are pooled through a dilated spatial pyramid pooling module, which effectively captures multi-modal tea garden knowledge features at different scales and fully utilizes the diversified information of multi-modal remote sensing features;

[0045] Step S52: The decoder has multiple double convolution modules and a series of bilinear interpolation up-sampling modules. Each up-sampling module in the decoder cascades the features from the encoder and the previous double convolution module and then up-samples them, and integrates a double attention guide module in the skip connection, so as to more fully describe the image scene and capture more key information;

[0046] Step S53: In the decoder sub-module layer, an improved multi-scale feature extraction module is constructed, fully considering the importance of multi-scale features, adjusting the channel number through double convolution modules and transpose convolution, and merging the outputs of each layer through channel splicing to obtain multi-modal tea garden knowledge features that can express multi-scale features. The design of this branch enables the model to fully utilize the rich information of multi-modal remote sensing features, further improving the accuracy and integrity of feature representation.

[0047] Further, step S6 includes the following contents:

[0048] Step S61: A double-branch feature fusion module is constructed by combining attention mechanism, and the features extracted by the two branches are fused;

[0049] Step S62: Perform a concatenation operation on the features from the two branches to initially fuse the feature maps extracted from the multimodal data;

[0050] Step S63: Utilize the compression and activation operations of the SE attention mechanism to enhance the semantic association between features of different modal data, capture key information of feature maps between different modalities, and dynamically perform multimodal feature fusion;

[0051] Step S64: The dual-branch feature fusion module fully utilizes the information from high-resolution remote sensing images and multimodal tea garden knowledge features, enabling the model to acquire more comprehensive and richer feature information, thereby capturing the diversity and complexity of the tea garden scene.

[0052] Further, step S7 includes the following:

[0053] Step S71: Combine Focal loss focal and Dice loss dice A custom composite loss function is used; Focal loss, compared to traditional cross-entropy loss, has the advantage of addressing class imbalance and increasing the weight of hard-to-classify samples; Dice loss measures the overlap between predicted results and true labels, and is combined with l when calculating the intersection-union ratio. focal and l dice It addresses the issues of imbalance between positive and negative samples, imbalance between easy and difficult samples, and unstable training.

[0054] Step S72: Given a set of N pixels Real mask image I M ;set up It is a pixel The tag value, These are the predicted mask labels after Sigmoid function activation. α controls the imbalance between positive and negative samples, and γ controls the imbalance between easy and difficult samples. Therefore, the focus loss l focal Defined as:

[0055]

[0056] Let P and T represent the predicted mask images, respectively. and real mask image I M The label vectors are given by |P| and |T|, which represent the norms of P and T, respectively, and |P∩T| is their intersection norm. Therefore, the Dice loss is... dice Defined as:

[0057]

[0058] In the formula, ∈ = 10-5, which can avoid the denominator from having a zero value;

[0059] Step S73: based on l focal and l dice The final custom composite loss function is set as different weights of loss:

[0060] l focaldice = xl focal +yl dice

[0061] Wherein l focaldice represents the custom composite loss function;

[0062] Step S74: the model is trained by using the training sample set, the error of the model prediction is calculated by the custom composite loss function, the optimizer based on the initial learning rate is used as the optimizer of the model training, the training batch size of the model is set as N, the iteration number of the model is set as R, and the cosine annealing learning rate is used as the learning rate decay strategy;

[0063] Step S75: the intersection over union is used to measure the overlap between the model prediction result and the true value in each batch, the hyperparameters are adjusted by using the verification sample set, whether overfitting occurs is monitored, and the optimal weight of the model is output when the intersection over union of the verification set is the highest.

[0064] Further, step S8 comprises the following contents:

[0065] Step S81: based on the model weight obtained by training adjustment, the weight is used for the test sample set, and the generalization ability of the model is evaluated;

[0066] Step S82: by using the weight, the sliding window prediction method is adopted, that is, the image is cropped with overlap and the edge is ignored during splicing to perform model prediction, and automatic identification of tea garden information is realized.

[0067] According to the second aspect of the application, an automatic tea garden identification system integrating phenol index and phenological characteristics of multi-modal remote sensing comprises an electronic device and a computer readable storage medium, wherein the electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer readable storage medium stores a computer program; characterized in that the processor executes the computer program to realize an automatic tea garden identification method integrating phenol index and phenological characteristics of multi-modal remote sensing according to any one of the application; and the computer program is executed by the processor to realize an automatic tea garden identification method integrating phenol index and phenological characteristics of multi-modal remote sensing according to any one of the application.

[0068] The application has the following advantages:

[0069] 1. Multi-modal remote sensing features are constructed by medium resolution time series data to represent tea tree growth conditions and phenolic content, which can provide more abundant information for tea garden recognition compared to traditional tea garden recognition based on single modal image.

[0070] 2. A dual-branch lightweight deep learning model is constructed to extract visual texture of high-resolution image and relevant tea garden biochemical phenology knowledge of multi-modal remote sensing features through different branches, which can more targetedly process different modal information compared to traditional single-branch model.

[0071] The purpose of the present application is to face the fine recognition problem of complex hilly tea garden, and provide a lightweight tea garden automatic recognition method integrating multi-modal remote sensing features, which constructs multi-modal tea garden remote sensing features through medium resolution time series Sentinel-1 SAR and Sentinel-2 MSI optical remote sensing images, and processes high-resolution remote sensing images and multi-modal tea garden remote sensing features through a dual-branch lightweight deep learning model, and finally fuses different modal features at the feature level to enhance the connection between different modal data, thereby improving the overall recognition effect of tea garden information. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 The step flow chart of the present application.

[0073] Figure 2 The flowchart of the present application.

[0074] Figure 3 The structure diagram of the dual-branch lightweight deep learning model of the present application fusing multi-modal remote sensing features.

[0075] Figure 4 The dual-branch feature fusion module structure diagram of the present application.

[0076] Figure 5 The tea garden extraction result diagram of the present application. DETAILED DESCRIPTION

[0077] The technical solutions of the present application will be specifically described below in combination with the drawings.

[0078] It should be pointed out that the following detailed description is exemplary and is intended to provide further description of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0079] It is to be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application; as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0080] As shown in Figures 1 to 5 The present application proposes an automatic tea garden recognition method and system integrating phenolic index and phenological characteristics of multi-modal remote sensing, including the following contents:

[0081] As shown in Figure 1 and Figure 2 The present application proposes an automatic tea garden recognition method integrating phenolic index and phenological characteristics of multi-modal remote sensing, characterized by including the following contents:

[0082] Step S1: Obtain medium-resolution intra-annual time-series Sentinel-1 SAR radar images in the study area, pre-process the images, and calculate the growth amplitude GA reflecting the phenological characteristics of tea;

[0083] Step S2: Obtain medium-resolution intra-annual time-series Sentinel-2 MSI optical images in the study area, pre-process the images, and calculate features related to tea growth status and substance content, including normalized vegetation index NDVI, phenolic compound index PCI, and phenolic high-value dominance index PHD;

[0084] Step S3: Obtain high-resolution optical remote sensing images of the same year in the study area, pre-process the images, and combine them with the multi-modal features calculated in steps S1 and S2 as sample image data. Use a sliding window to crop the sample images and manually labeled labels to obtain a sample dataset of HxW pixels, and divide the training sample set, validation sample set, and test sample set according to a certain proportion;

[0085] Step S4: Construct a multi-scale lightweight model with attention mechanism as the first branch, input the high-resolution remote sensing image into the model, and deeply mine the semantic information in the high-resolution remote sensing image;

[0086] Step S5: Construct a semantic segmentation model with multi-modal information aggregation as the second branch, input the features extracted in steps S1 and S2 into the model, and extract multi-modal tea garden knowledge features;

[0087] Step S6: Construct a dual-branch feature fusion module to fuse the high-resolution image features and multi-modal tea garden knowledge features extracted;

[0088] Step S7: training of the model is performed in combination with the training sample set obtained in step S3, error of model prediction is calculated through a self-defined composite loss function FocalDiceLoss, model parameters are adjusted based on an Adam optimizer, a cosine annealing learning rate is used as a learning rate decay strategy, the model is trained; a validation sample set is used to adjust hyperparameters, overfitting is monitored, and model weights are outputted;

[0089] Step S8: based on the model weights obtained by training and adjustment in step S7, the weights are used for the test sample set constructed in step S3, generalization ability of the model is evaluated, and the weights are used to realize automatic identification of large-area tea garden information.

[0090] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S1 includes the following content:

[0091] Step S11: ground range detection GRD data of year-in-time Sentinel-1 SAR radar image in interferometric wide swath IW mode is obtained, wherein the ground range detection GRD data has been processed by thermal noise removal, orbit file correction, radiation calibration, etc., and then filtered and corrected by a self-defined data processing function, terrain correction, and finally backscattering information is obtained;

[0092] Wherein, the GRD data is ground range detection data;

[0093] Step S12: in order to reflect the year-in-time growth rule of tea garden, the time series curve of VH value of tea garden in year-in-time Sentinel-1 SAR radar image is analyzed, and the growth amplitude GA reflecting the phenological characteristics of tea is constructed, GA is the difference between the peak value of tea growth period and the low value of frost period, and the calculation formula is as follows:

[0094] GA = VH 生长期峰值 -VH 霜冻期低值

[0095] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S2 includes the following content:

[0096] Step S21: year-in-time Sentinel-2 MSI optical image Level-1C product is obtained, which has been processed by radiation calibration and atmospheric correction. Since the optical image will be disturbed by clouds, the cloud coverage percentage of the image is controlled to be less than 10%, and the image is de-clouded by using quality assessment QA60 band and cloud probability product;

[0097] Step S22: calculating the normalized difference vegetation index (NDVI) of the study area, and using the maximum value composition method of NDVI to obtain the image of the best vegetation condition in a year, and the calculation formula of NDVI is as follows:

[0098]

[0099] In the formula, NIR represents the reflectivity of the near-infrared band, and RED represents the reflectivity of the red light band.

[0100] Step S23: calculating the phenolic compound index (PCI) of the study area, and using the maximum value composition method of PCI to obtain the image of the best phenolic condition in a year, and the calculation formula of PCI is as follows:

[0101]

[0102] In the formula, GREEN represents the reflectivity of the green light band, and SWIR2 represents the reflectivity of the short-wave infrared band.

[0103] Step S24: calculating the enhanced vegetation index (EVI2) of the study area, and the calculation formula is as follows:

[0104]

[0105] In the formula, NIR represents the reflectivity of the near-infrared band, and RED represents the reflectivity of the red light band.

[0106] Step S25: in order to reflect the change rule of the phenolic content in a year, the phenolic high-value dominant index (PHD) is constructed by using PCI and EVI2, the high-value interval T is identified by using 50% of EVI2, PHD is calculated by using the time sequence of PCI in the interval, and the calculation formula of PHD is as follows:

[0107]

[0108] In the formula, p k represents the PCI value of the corresponding time point k of a high value, T represents the set of time points of the high value interval, and |T| represents the length of the high value interval.

[0109] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S3 includes the following contents:

[0110] Step S31: resampling the multi-modal tea garden knowledge features (NDVI, PCI, PHD and GA) to the same spatial resolution as the high-resolution image, and performing band composition with the RGB bands of the high-resolution remote sensing image to obtain the initial sample image data set of the study area;

[0111] Step S32: Using the face vector construction method built in the ArcGIS software, tea garden labels are made, tea gardens are assigned a value of 1, non-tea gardens are assigned a value of 0, and tea garden vector labels are converted into a grid to obtain an initial sample label data set in the study area;

[0112] Step S33: Based on the initial tea garden sample data set, the image and label sample data are divided into a sample data set with a size of HxW pixels through sliding window cropping, and data enhancement operations such as horizontal flipping, vertical flipping, diagonal mirroring, and chroma enhancement are adopted to increase the diversity of the samples and improve the generalization ability of the model.

[0113] Step S34: The training sample set, the verification sample set and the test sample set are divided according to a certain proportion.

[0114] Further, in an embodiment of the present application, the image and label sample data are divided into a sample data set with a size of 256x256 pixels through sliding window cropping, and data enhancement operations such as horizontal flipping, vertical flipping, diagonal mirroring, and chroma enhancement are adopted to increase the diversity of the samples and improve the generalization ability of the model.

[0115] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S4 includes the following content:

[0116] Step S41: Based on the encoder-decoder architecture, the encoder is composed of 4 convolutional blocks and 2 tokenization multi-layer perception machines, feature extraction is performed on the input high-resolution remote sensing image, and Max Pooling and Patch Embedding are adopted to down-sample the feature map;

[0117] Step S42: The decoder is composed of 2 tokenization multi-layer perception machines and 3 double convolutional blocks, and adopts transposed convolution to up-sample the feature map, and the encoder and the decoder are combined through a jump connection;

[0118] Step S43: A double attention guide module scSE is embedded in the jump connection, the scSE is composed of a channel attention mechanism module cSE and a spatial attention mechanism module sSE, to simulate the channel and spatial region dependency relationship in the feature map, which can improve the loss of important features in the continuous down-sampling process of convolution and pooling operations, and effectively focus on the important features of the tea garden in the high-resolution image;

[0119] Step S44: To improve the distinguishability of the semantic representation, a multi-scale feature extraction module is designed to aggregate decoding features of different depths and scales on the decoder subnetwork. By processing the decoder features of each layer to a unified feature size and finally fusing them in a connection layer, high-resolution image features with multi-scale feature expression are obtained. Through this branch, information in high-resolution remote sensing images can be deeply mined, and the visual expression of key features in tea gardens can be captured.

[0120] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S5 includes the following content:

[0121] Step S51: The encoder is composed of multiple double convolution modules and a series of max-pooling modules, and the backbone features are pooled by an ASPP module, effectively capturing multi-modal tea garden knowledge features (NDVI, PCI, PHD and GA) at different scales, and fully utilizing the diversified information of multi-modal remote sensing features;

[0122] Step S52: The decoder has multiple double convolution modules and a series of bilinear interpolation up-sampling modules. Each up-sampling module in the decoder cascades the features from the encoder and the previous layer double convolution module and then up-samples them, and integrates a double attention guide module in the jump connection, so as to more fully describe the image scene and capture more key information;

[0123] Step S53: In the decoder sub-module layer, an improved multi-scale feature extraction module is constructed, fully considering the importance of multi-scale features. The channel number is adjusted through double convolution modules and transpose convolution, and the outputs of each layer are merged through channel splicing to obtain multi-modal tea garden knowledge features that can express multi-scale features. The design of this branch enables the model to fully utilize the rich information of multi-modal remote sensing features, further improving the accuracy and integrity of feature representation.

[0124] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S6 includes the following content:

[0125] Step S61: A double-branch feature fusion module combining attention mechanism is constructed to fuse the features extracted by the two branches;

[0126] Step S62: Cascade operation is performed on the features from the two branches to preliminarily fuse the feature maps extracted from the multi-modal data;

[0127] Step S63: the compression and excitation operation of the SE attention mechanism is used to strengthen the semantic association between the features of different modal data, capture the key information of the feature maps between different modalities, and dynamically perform multi-modal feature fusion;

[0128] Step S64: the dual-branch feature fusion module makes full use of the information of the high-resolution remote sensing image and the multi-modal tea garden knowledge features, so that the model can obtain more comprehensive and rich feature information, thereby capturing the diversity and complexity characteristics of the tea garden scene.

[0129] Further, as shown in Figure 3 and Figure 4 , in an embodiment of the present application, step S7 includes the following contents:

[0130] Step S71: combine Focal loss l focal and Dice loss l dice to define a composite loss function; wherein the advantage of the Focal loss over the traditional cross-entropy loss is that it can solve the class imbalance problem and improve the weight of difficult classification samples; the Dice loss can measure the overlap between the predicted result and the true label, and ignores a large number of background pixels when calculating the intersection over union. The background proportion of the tea garden sample is high, and the l focal and l dice can handle the imbalance between positive and negative samples, the imbalance between difficult and easy samples, and the instability of training;

[0131] Step S72: given a real mask image I with N pixels M ; let be the label value of the pixel , and be the predicted mask label after Sigmoid function activation, α is used to control the imbalance between positive and negative samples, and γ is used to control the imbalance between difficult and easy samples, therefore, the focal loss l focal is defined as:

[0132]

[0133] Let P and T represent the label vectors of the predicted mask image and the real mask image I M respectively, |P| and |T| represent the norms of P and T, and |P∩T| is the intersection norm, therefore, the Dice loss l dice is defined as:

[0134]

[0135] In the formula, ∈=10 -5 , which can avoid zero values in the denominator.

[0136] Step S73: Based on l focal and l dice Different weights are assigned to the loss, resulting in a final custom composite loss. focaldice for:

[0137] l focaldice =xl focal +yl dice

[0138] Step S74: Train the model using the training sample set, through a custom composite loss function.

[0139] FocalDiceLoss is used to calculate the error of model prediction. The Adam optimizer based on the initial learning rate Lr is used as the optimizer for model training. The batch size of the model training is set to N, the number of model iterations is set to R, and the learning rate decay strategy uses cosine annealing learning rate.

[0140] Step S75: Use the Intersection over Union (IoU) ratio to measure the overlap between the model's predictions and the true values ​​in each batch, adjust the hyperparameters using the validation set, monitor for overfitting, and output the optimal model weights when the IoU ratio on the validation set is the highest.

[0141] Furthermore, in one embodiment of the present invention, step S74 further includes: training the model using the training sample set, calculating the model prediction error using a custom composite loss function FocalDiceLoss, using the Adam optimizer with an initial learning rate of 0.0001 as the optimizer for model training, setting the training batch size of the model to 8, setting the number of iterations of the model to 50, and using a cosine annealing learning rate as the learning rate decay strategy.

[0142] Furthermore, such as Figure 3 and Figure 4 As shown, in one embodiment of the present invention, step S8 includes the following:

[0143] Step S81: Based on the model weights obtained from training, use these weights on the test sample set to evaluate the model's generalization ability.

[0144] Step S82: Using this weight, a sliding window prediction method is adopted, that is, overlapping silhouette images are cropped and the edges are ignored when splicing to perform model prediction, so as to realize the automatic identification of tea garden information in a large area.

[0145] According to a second aspect of the present application, the present application provides a multi-modal remote sensing tea garden automatic identification system integrating phenol index and phenological characteristics, comprising an electronic device and a computer readable storage medium, wherein the electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer readable storage medium stores a computer program; characterized in that the processor executes the computer program to realize a multi-modal remote sensing tea garden automatic identification method integrating phenol index and phenological characteristics as described in the present application; and the computer program is executed by the processor to realize a multi-modal remote sensing tea garden automatic identification method integrating phenol index and phenological characteristics as described in the present application.

[0146] In addition to the above, the present application also has related embodiments, including the following:

[0147] In an embodiment of the present application, the study area is located in the central region of a certain region, and the data sources include GF-2 images (2021, 0.8m), Sentinel-1 SAR data (2021, 10m), and Sentinel-2 optical images (2021, 10m). Since the data comes from different sensors, the spatial and temporal resolution difference is large, and the geographical position of the same ground object in different images may have a large deviation. The same pixel points are manually collected as the reference for GF-2 images to calibrate the remaining grid data, and the nearest neighbor method is used for resampling, with a sampling spatial resolution of 0.8m, to eliminate the geographical position difference between images. In this embodiment, 3600 remote sensing images with a pixel size of 256x256 and corresponding label images obtained after data enhancement are used for model training.

[0148] As shown in Figure 3 , it is a flowchart of the preferred embodiment of the present application.

[0149] As shown in Figure 4 , it is a structure diagram of the fusion multi-modal remote sensing feature dual-branch lightweight deep learning model of the present embodiment.

[0150] As shown in Figure 2 , it is a structure diagram of the dual-branch feature fusion module of the present embodiment.

[0151] As shown in Figure 3 Figure 4 Figure 5 , it is a high-resolution remote sensing image used in the present embodiment and a tea garden distribution result map extracted based on the dual-branch lightweight deep learning model. As can be seen from the figure, after using the RGB+NDVI+PCI+HD+GA data set, the false positives and missed positives are significantly reduced; the experimental results show the effectiveness of the proposed method for tea garden identification.

[0152] The application discloses an integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method and system, aiming at the demand of fine extraction of tea gardens in complex hilly areas, a double-branch lightweight deep learning model fusing high-resolution optical remote sensing images and multi-modal remote sensing features is provided.The first branch of the model is used for extracting spatial texture and structure information in the high-resolution remote sensing image, and the second branch is used for learning multi-modal tea garden knowledge features (including growth amplitude GA, normalized vegetation index NDVI, phenolic compound index PCI and phenolic high-value dominant index PHD) constructed based on Sentinel-1 SAR and Sentinel-2 MSI, so as to depict the features of tea tree growth conditions and material content.In the decoding stage, different modal features are fully fused at the feature level through a double-branch fusion module, and the complementarity and relevance between the modes are enhanced.Compared with the traditional method which depends on single image features, the application can still maintain high fine segmentation accuracy under the condition of limited samples, realize the automatic identification of large-area tea garden distribution, and the method of the application takes into account the advantages of high-resolution optical images and multi-modal geosciences knowledge, and provides key technical support for dynamic monitoring and fine management of tea garden resources.

[0153] The above is the preferred embodiment of the application, any changes made according to the technical solutions of the application, as long as the generated function does not exceed the scope of the technical solutions of the application, belongs to the protection scope of the application.

Claims

1. An integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method, characterized in that, Automatic identification of tea gardens is realized by analyzing phenol index and phenological characteristics; the automatic identification method of the tea garden of the multi-modal remote sensing integrated with phenol index and phenological characteristics comprises the following contents: Step S1: acquiring medium-resolution intra-annual time-series radar images in a study area, pre-processing the images, and calculating growth amplitude reflecting the phenological characteristics of tea leaves; Step S2: acquiring medium-resolution intra-annual time-series optical images in the study area, pre-processing the images, and calculating features related to the growth status and material content of tea trees, including normalized vegetation index, phenolic compound index and phenolic high-value dominant index; Step S3: acquiring high-resolution optical remote sensing images of the same year in the study area, pre-processing the images, and combining the multi-modal features calculated in steps S1 and S2 as sample image data, using a sliding window to crop the sample images and manually labeled labels to obtain sample data sets with HxW pixel size, and dividing the training sample set, the verification sample set and the test sample set according to a certain proportion; Step S4: constructing a multi-scale lightweight model with a fusion attention mechanism as the first branch, inputting high-resolution remote sensing images into the model, and deeply mining semantic information in the high-resolution remote sensing images; Step S5: constructing a semantic segmentation model with multi-modal information aggregation as the second branch, inputting the features extracted in steps S1 and S2 into the model, and extracting multi-modal tea garden knowledge features; Step S6: constructing a double-branch feature fusion module to fuse the high-resolution image features and multi-modal tea garden knowledge features extracted; Step S7: training the model by combining the training sample set obtained in step S3, calculating the error of the model prediction by using a self-defined compound loss function, adjusting the model parameters based on an optimizer, using a cosine annealing learning rate as a learning rate decay strategy, and training the model; Adjust the hyperparameters using the verification sample set to monitor whether overfitting occurs, and output the model weights; Step S8: based on the model weights obtained by training and adjusting in step S7, use the weights for the test sample set constructed in step S3 to evaluate the generalization ability of the model, and use the weights to realize the automatic identification of large-area tea garden information.

2. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S1 comprises the following contents: Step S11: acquiring ground range detection data of the intra-annual time-series radar image in the interferometric wide swath mode, wherein the ground range detection data has been processed by thermal noise removal, orbit file correction, radiation calibration, and then filtered and corrected by a self-defined data processing function, and finally the backscattering information is obtained; Step S12: to reflect the intra-annual growth rule of tea gardens, analyze the time series curve of the image value of the tea garden in the intra-annual time-series radar image, and construct the growth amplitude reflecting the phenological characteristics of tea leaves, the growth amplitude being the difference between the image value of the peak value of the tea growth period and the low value of the frost period, and the calculation formula being as follows: GA = VH 生长期峰值 -VH 霜冻期低值 Wherein, GA represents the growth amplitude, and VH represents the image value.

3. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S2 comprises the following contents: Step S21: acquiring intra-annual time-series optical image products, wherein the intra-annual time-series optical image products have been processed by radiation calibration and atmospheric correction; wherein the optical image products are clouded by using quality evaluation QA60 band and cloud probability products; Step S22: Calculate the normalized vegetation index of the study area, and use the normalized vegetation index maximum synthesis method to obtain the image with the best vegetation condition in the year. The calculation formula of the normalized vegetation index is as follows: Wherein, NDVI represents the normalized vegetation index, NIR represents the reflectivity of the near-infrared band, and RED represents the reflectivity of the red light band; Step S23: Calculate the phenolic compound index of the study area, and use the phenolic compound index maximum synthesis method to obtain the image with the best phenolic condition in the year. The calculation formula of the phenolic compound index is as follows: Wherein, PCI represents the phenolic compound index, GREEN represents the reflectivity of the green light band, and SWIR2 represents the reflectivity of the short-wave infrared band; Step S24: Calculate the enhanced vegetation index EVI2 of the study area. The calculation formula of the enhanced vegetation index is as follows: Wherein, EVI2 represents the enhanced vegetation index, NIR represents the reflectivity of the near-infrared band, and RED represents the reflectivity of the red light band; Step S25: In order to reflect the change rule of the phenolic content in the tea garden in the year, the phenolic compound index and the enhanced vegetation index are used to construct the phenolic high-value dominance index. The 50% of the enhanced vegetation index is used to identify the high-value interval T. The phenolic high-value dominance index is calculated through the time series of the phenolic compound index in the interval. The phenolic high-value dominance index represents the average value of the phenolic compound index time curve in the high-value interval. The calculation formula of the phenolic high-value dominance index is as follows: where PHD represents a phenolic high-value dominance index, p k represents the phenolic compound index value at a certain high-value corresponding time point k, T represents a set of high-value interval time points, and |T| represents a high-value interval length.

4. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S3 includes the following contents: Step S31: Resample the multi-modal tea garden knowledge features to the same spatial resolution as the high-resolution image, and perform band synthesis with the RGB band of the high-resolution remote sensing image to obtain the initial sample image dataset of the study area. The multi-modal tea garden knowledge features include normalized vegetation index, phenolic compound index, phenolic compound index and growth amplitude; Step S32: Use the face vector construction method built-in the software to make tea garden labels. Assign 1 to tea gardens and 0 to non-tea gardens. Convert the tea garden vector label to a grid to obtain the initial sample label dataset in the study area; Step S33: Based on the initial tea garden sample dataset, the image and label sample data are divided into HxW pixel size sample datasets through sliding window cropping, and data enhancement operations are used to increase the diversity of samples and improve the generalization ability of the model; Wherein, the data enhancement operations include horizontal flip, vertical flip, diagonal mirror image and chroma enhancement; Step S34: Divide the training sample set, verification sample set and test sample set according to a certain proportion.

5. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S4 includes the following contents: Step S41: Based on the encoder-decoder architecture, the encoder is composed of convolutional blocks and tokenized multilayer perceptron. The features of the input high-resolution remote sensing image are extracted, and maximum pooling and block embedding are used for down-sampling of the feature map; Step S42: The decoder is composed of tokenized multilayer perceptron and double convolutional blocks, and uses transpose convolution for up-sampling of the feature map. The encoder and the decoder are connected through a jump connection to combine the features. Step S43: embedding a double attention guide module in the skip connection, the double attention guide module being composed of a channel attention mechanism module and a spatial attention mechanism module, to simulate the channel and spatial area dependency in the feature map, improve the important feature loss in the continuous down-sampling process of convolution and pooling operations, and effectively focus on the important features of the tea garden in the high-resolution image; Step S44: to improve the distinguishability of the semantic representation, a multi-scale feature extraction module is designed to aggregate the decoding features of different depths and scales on the decoder subnetwork, the decoder features of each layer are processed to a unified feature size, and finally fused in a connection layer to obtain high-resolution image features with multi-scale feature expression; Through this branch, the information in the high-resolution remote sensing image can be deeply mined, and the visual expression of the key features in the tea garden can be captured.

6. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S5 includes the following contents: Step S51: the encoder is composed of multiple double convolution modules and a series of max pooling modules, and the main features are pooled through the empty spatial pyramid pooling module, which effectively captures multi-modal tea garden knowledge features at different scales and fully utilizes the diversified information of multi-modal remote sensing features; Step S52: the decoder has multiple double convolution modules and a series of bilinear interpolation up-sampling modules, each up-sampling module in the decoder cascades the features from the encoder and the previous layer double convolution module and then up-samples, and integrates a double attention guide module in the skip connection, so as to more fully describe the image scene and capture more key information; Step S53: in the decoder sub-module layer, an improved multi-scale feature extraction module is constructed, fully considering the importance of multi-scale features, adjusting the channel number through double convolution module and transpose convolution, and merging the outputs of each layer through channel splicing to obtain multi-modal tea garden knowledge features that can express multi-scale features; the design of this branch enables the model to fully utilize the rich information of multi-modal remote sensing features, and further improves the accuracy and integrity of the feature representation.

7. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S6 includes the following contents: Step S61: a double-branch feature fusion module is constructed by combining the attention mechanism, and the features extracted from the two branches are fused; Step S62: the features from the two branches are cascaded to preliminarily fuse the feature maps extracted from the multi-modal data; Step S63: the compression and excitation operation of the SE attention mechanism is used to strengthen the semantic association between the features of different modal data, capture the key information of the feature maps between different modalities, and dynamically fuse the multi-modal features; Step S64: the double-branch feature fusion module fully utilizes the information of high-resolution remote sensing images and multi-modal tea garden knowledge features, so that the model can obtain more comprehensive and rich feature information, thereby capturing the diversity and complexity of the tea garden scene.

8. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S7 includes the following contents: Step S71: combine Focal loss l focal and Dice loss l dice Custom composite loss function; wherein the advantage of Focal loss compared with traditional cross-entropy loss is that it can solve the class imbalance problem and improve the weight of difficult classification samples; Dice loss can measure the overlap between the predicted results and the true labels, and combine l focal and l dice Handle positive and negative sample imbalance, difficult and easy sample imbalance, and training instability Step S72: Given a set of N pixels Real mask image I M ;set up It is a pixel The tag value, These are the predicted mask labels after Sigmoid function activation. α controls the imbalance between positive and negative samples, and γ controls the imbalance between easy and difficult samples. Therefore, the focus loss l focal Defined as: Let P and T represent the label vectors of the predicted mask image and the real mask image I M respectively, |P| and |T| represent the norms of P and T, |P∩T| is the intersection norm of them, therefore, the Dice loss l dice is defined as: In the formula, ∈=10-5, which can avoid zero in the denominator; Step S73: Based on l focal and l dice The final custom composite loss function is set as: l focaldice = x1 focal + y1 dice where l focaldice represents a custom composite loss function; Step S74: training the model using the training sample set, calculating the error of the model prediction by a self-defined composite loss function, using an optimizer based on an initial learning rate as the optimizer for model training, setting the training batch size of the model to N, setting the number of iterations of the model to R, and using cosine annealing learning rate for learning rate decay strategy; Step S75: using the intersection over union to measure the overlap between the model prediction result and the true value in each batch, adjusting the hyperparameters using the validation sample set, monitoring whether overfitting occurs, and outputting the optimal weight of the model when the intersection over union of the validation set is the highest.

9. The integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification method according to claim 1, characterized in that, Step S8 includes the following contents: Step S81, based on the model weight obtained by training adjustment, using the weight for the test sample set to evaluate the generalization ability of the model; Step S82, using the weight, using the sliding window prediction method, that is, overlappingly cutting the image and ignoring the edge when splicing to predict the model, to realize the automatic identification of tea garden information.

10. An integrated phenol index and phenological feature multi-modal remote sensing tea garden automatic identification system, comprising an electronic device and a computer readable storage medium, wherein the electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer readable storage medium stores a computer program; characterized in that, The processor executes the computer program to realize the multi-modal remote sensing tea garden automatic identification method integrating phenol index and phenological characteristics according to any one of claims 1-9; the computer program is executed by the processor to realize the multi-modal remote sensing tea garden automatic identification method integrating phenol index and phenological characteristics according to any one of claims 1-9.

Images