A fine-grained phenology parameter extraction method based on a feature fusion network

Abstract

The application provides a kind of fine-grained phenology parameter extraction method based on feature fusion network, comprising the following steps: (1) the photos taken in phenology camera are collected and pretreated, and a data set suitable for training is constructed;(2) phenology parameter extraction;First, introduce ResNet18, extract global features through 3x3 convolution kernel;Second, extract local features for lightweight part;(3) after two branches extract global and local features respectively, the two features are fused using an adaptive feature fusion module, the features after splicing are processed using a GELU activation function, and comprehensive features are generated;(4) the classification layer uses a Softmax function, and the fused features are mapped to different phenological stage class spaces using Dropout technology, achieving accurate classification of multiple phenological stages.The application realizes a high-precision phenological monitoring framework, is simple to operate, has high fine-grained phenological extraction, and is suitable for ecological research of plant growth process.

Description

TECHNICAL FIELD

[0001] The application provides a fine-grained phenological parameter extraction method based on a feature fusion network, and belongs to the technical field of image processing. BACKGROUND

[0002] Vegetation phenology refers to the seasonal physiological changes and phenomena that occur in the growth cycle of plants, including important stages such as germination, flowering, fruiting, and leaf fall. These changes are influenced by climate, environment, and ecological factors, and reflect the response of plants to climate change. By monitoring vegetation phenology, we can understand the health of the ecosystem, predict agricultural production, assess the impact of climate change, and provide scientific basis for ecological restoration and resource management. With the development of remote sensing technology and data analysis methods, the monitoring and research of vegetation phenology has become more accurate and efficient.

[0003] The vegetation phenology parameter extraction method is mainly for time series curves. The process of vegetation phenology parameter calculation mainly includes two steps: time series reconstruction and phenological parameter extraction. The time series reconstruction process mainly includes time series harmonic analysis, polynomial fitting, SG filtering, and local spline function fitting, etc. The phenological parameter extraction method mainly includes absolute threshold method, dynamic threshold method, and derivative function extreme value method, etc.

[0004] Most of the phenological parameter extraction methods can only extract the time series composed of vegetation indices from remote sensing images through smoothing fitting and other methods, and then extract the phenological parameters through threshold operation. The operation is complex. And most of the phenological product extraction methods can only extract the beginning and end of the growing season, and the fine-grained extraction of the phenological period is low. This limitation limits the in-depth understanding of plant response to environmental changes, especially in the face of climate change and ecological management needs. SUMMARY

[0005] The application provides a fine-grained phenological parameter extraction method based on a feature fusion network to achieve the following objectives:

[0006] (1) To solve the problem of complex operation of existing phenological parameter extraction methods, the application provides a fine-grained phenological parameter extraction framework (RBPhenology) suitable for real-time monitoring of multiple vegetation by a phenological camera. Based on the time series images taken by the phenological camera, the spatial and temporal information is used to determine the phenological parameters of each time series image from several pictures, construct a training data set, and use deep learning method to train a vegetation phenological parameter extraction model based on deep learning. The model can extract the phenological parameters of the test area based on deep learning, solving the problem of complex operation of high-quality phenology.

[0007] (2) The existing method can usually only extract a small number of specific phenological stages, which limits the comprehensive understanding of the plant growth process. The present application proposes a ResFormer double-branch feature fusion network structure, which can effectively identify up to seven different phenological stages, suitable for fine-grained phenology monitoring tasks, and significantly improve the accuracy and richness of phenological parameter extraction.

[0008] The specific technical solutions are:

[0009] A fine-grained phenological parameter extraction method based on a feature fusion network, comprising the following steps:

[0010] (1) Collect and preprocess the photos taken by the phenological camera to construct a data set suitable for training;

[0011] Perform image quality and image content detection, manually identify difficult-to-classify or unclear pictures, mark these pictures as "need to check", and then delete them by manual or other means. And label the data set, identify and delete or re-label pictures with incorrect labels through manual review, then perform data augmentation, increase the robustness of the model by rotating, cropping and randomly covering part of the data, and construct a data set suitable for training.

[0012] (2) Phenological parameter extraction;

[0013] First, introduce ResNet18 to extract global features through 3x3 convolution kernels;

[0014] The pooling layer down-samples by maximum pooling operation, gradually reducing the spatial resolution of the feature map, and directly transmits the input to the output layer through skip connections. The feature map processed by a series of convolution layers and pooling layers is extracted through a fully connected layer to extract the global semantic information of the phenological stage.

[0015] Use the focal loss function to alleviate the sensitivity of the residual network to gradient changes, the formula is as follows:

[0016] FL(pt)=-α t (1-pt) γ log(pt)

[0017] Where pt is the prediction probability of the model for different phenological stages, alpha t is the weight coefficient of different categories, and gamma is the focal parameter.

[0018] Second, for the lightweight part of extracting local features;

[0019] A double-layer routing module is adopted, which converts the input data into a shape of The query, key and value tensors are obtained by linear projection. The calculation of linear projection is obtained by multiplying the rearranged input data with the projection weight respectively to obtain the corresponding query, key and value. The routing between regions is realized by a directed graph.

[0020] In this stage, the query and key matrices at the region level are generated by averaging the query and key of each region. Only the connection between each region and the top k most relevant regions is retained;

[0021] A local context enhancement term is introduced to further improve the representation ability of local features. The specific formula is as follows:

[0022] O = Attention(X r W q ,K g ,V g + LCE(X r W v )

[0023] K g = gather(K, topIndex(Q r (X r W k ) T ))

[0024] V g = gather(V, topIndex(Q r (X r W v ) T )))

[0025] Where W q , W k , W v are the projection weights of Q query, K key and V value respectively, O is the attention sum, Q is the linear projection, K g is the gathered key, V g is the value tensor, LCE(X r W v ) is the value tensor, and each attention channel number is set to 32 and the kernel size is 5.

[0026] The lightweight local feature extraction part adopts GELU activation function and cross entropy loss function, and adopts AdamW algorithm for training and RandAugment automatic enhancement strategy to optimize the model parameters. The GELU activation function is as follows:

[0027]

[0028] where 0.5 is a coefficient to scale a portion of the input, x is the input value, and tanh represents the hyperbolic tangent function, is a constant factor approximately equal to 0.79788 to scale the input. 0.044715 is a constant used to adjust the cubic term of x to improve the approximation of the GELU function.

[0029] The cross-entropy loss function is as follows:

[0030]

[0031] C is the total number of categories of classification, which is set to 7 in the present application, y i is an indicator variable of the true label, p i is the probability value of the i-th category predicted by the model, log(p i ) represents the logarithmic value of the model prediction probability.

[0032] (3) After the global and local features are extracted in the two branches respectively, an adaptive feature fusion module is used to fuse the two parts of features, and a GELU activation function is used to process the spliced features to generate more comprehensive feature representation:

[0033] F fuse = Softmax (alpha x feature vector1 + beta x feature vector2)

[0034] Alpha and beta are learnable weight parameters that control the weight of feature vector1 and feature vector2 in feature fusion, respectively. Feature vector1 and feature vector2 are global features and local features, respectively.

[0035] (4) The classification layer uses a Softmax function and adds a Dropout technique to map the fused features to the category space of different phenological phases, achieving accurate classification of multiple phenological phases. When the model starts training, the model parameters will be fine-tuned every round of training.

[0036] Compared with the prior art, the present application has the following technical effects:

[0037] 1) The present application realizes a high-precision phenological monitoring framework, which improves the previous complex phenological parameter extraction method, and therefore has certain commercial value.

[0038] 2) Compared with the previous phenological parameter extraction method, the present application is simple to operate, has high granularity of phenological period extraction, and is suitable for ecological research of plant growth process.

[0039] 3) Without long time recording time series, this method can be well applied to different tree species, various different phenological camera stations, etc. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 is a flowchart of the present application;

[0041] Figure 2 is a fine-grained phenological data set of the embodiment;

[0042] Figure 3 is a fine-grained phenological monitoring framework of the embodiment;

[0043] Figure 4 is a site 1 fine-grained phenological identification confusion matrix of the embodiment;

[0044] Figure 5 is a site 2 fine-grained phenological identification confusion matrix of the embodiment;

[0045] Figure 6 is a site 3 fine-grained phenological identification confusion matrix of the embodiment;

[0046] Figure 7 is a site 4 fine-grained phenological identification confusion matrix of the embodiment. DETAILED DESCRIPTION

[0047] The embodiment is implemented by using the existing deep learning framework Pytorch and the corresponding programming library, mainly including Numpy, PIL, Scipy, etc. Among them, Pytorch mainly uses a pre-trained deep learning model, including linear modules, convolution modules, etc.

[0048] As shown in Figure 1 , a fine-grained phenological parameter extraction method based on a feature fusion network includes the following steps:

[0049] (1) Collect and preprocess the photos taken by the phenological camera to ensure the accuracy and consistency of the data, thereby improving the performance of the subsequent model. Perform image quality and image content detection, manually identify difficult-to-classify or unclear pictures, mark these pictures as "to be checked", and then delete them by manual or other means. And label the data set, identify and delete or re-label pictures with label errors through manual review, then perform data enhancement, increase the robustness of the model through rotation, cropping, random covering part of the data, etc., and construct a data set suitable for training, as shown in Figure 2 .

[0050] (2) Phenological parameter extraction;

[0051] Firstly, ResNet18 is introduced to extract features through a 3*3 convolution kernel. The pooling layer down-samples through the maximum pooling operation, gradually reducing the spatial resolution of the feature map, and the input is directly transmitted to the output layer through the skip connection, solving the problem of gradient disappearance in the deep network. After a series of convolution and pooling layers, the feature map is processed through the fully connected layer to extract the global semantic information of the phenological period. Since the residual network is sensitive to gradient changes, the invention proposes to use the focal loss function to alleviate the sensitivity of the residual network to gradient changes, and the formula is as follows:

[0052] FL(pt)=-α t (1-pt) γ log(pt)

[0053] Where pt is the prediction probability of the model for different phenological periods, α t is the weight coefficient of different categories, and γ is the focal parameter.

[0054] Secondly, for the lightweight part of extracting local features, the embodiment adopts a double-layer routing module, which converts the input data into the form of by rearranging the input data. Then, the query, key and value tensors are obtained by linear projection. The calculation of linear projection is obtained by multiplying the rearranged input data with the projection weight to obtain the corresponding query, key and value. The routing between regions is realized through a directed graph. In this stage, the query and key matrices at the region level are generated by averaging the query and key of each region. In order to improve the calculation efficiency, the invention only retains the connection between each region and the top k most relevant regions, which can significantly reduce the computational complexity. Finally, a local context enhancement term is introduced to further improve the representation ability of local features. The specific formula is as follows:

[0055] O=Attention(X r W q ,K g ,V g +LCE(X r W v )

[0056] K g =gather(K,topIndex(Q r (X r W k ) T ))

[0057] V g =gather(V,topIndex(Q r (X r W v )T )))

[0058] where W q ,W k ,W v are the projection weights of Q query, K key and V value respectively, O is the attention sum, Q is the linear projection, K g is the aggregated key, V g is the value tensor, LCE(X r W v ) is the value tensor, and is the local context enhancement term, the number of channels of each attention is set to 32, and the kernel size is 5;

[0059] The light-weight local feature extraction part adopts GELU activation function and cross-entropy loss function, and adopts AdamW algorithm for training and RandAugment automatic enhancement strategy to optimize the model parameters. The GELU activation function is as follows:

[0060]

[0061] where 0.5 is a coefficient used to scale a portion of the input, x is the input value, tanh represents the hyperbolic tangent function, is a constant factor, approximately equal to 0.79788, used to scale the input. 0.044715 is a constant used to adjust the cubic term of x to improve the approximation of the GELU function. The shape of the GELU function is smoother than ReLU, which is suitable for complex nonlinear tasks in neural networks. The cross-entropy loss function is as follows:

[0062]

[0063] C is the total number of categories of classification, which is set to 7 in this experiment, y i is an indicator variable of the true label, p i is the probability value of the i-th class predicted by the model, and log(p i ) represents the logarithmic value of the model prediction probability.

[0064] (3) After extracting global and local features in the two branches respectively, the model uses an adaptive feature fusion module to fuse the two parts of features, and uses a GELU activation function to further process the spliced features in order to generate more comprehensive feature representations.

[0065] F fuse = Softmax (alpha x feature vector1 + beta x feature vector2)

[0066] Alpha and beta are learnable weight parameters that control the weight of feature vector 1 and feature vector 2 in feature fusion, respectively.

[0067] (4) The classification layer uses a Softmax function to map the fused features to the class space of different phenological stages by adding Dropout technology, so as to realize accurate classification of multiple phenological stages. When the model starts training, the model parameters are fine-tuned every round of training.

[0068] The application provides a kind of real-time monitoring framework (RBPhenology) suitable for phenological camera monitoring multiple vegetation fine-grained phenological parameters and a kind of ResFormer double-branch feature fusion network structure.

[0069] As shown in Figure 3 The kind of real-time fine-grained phenological monitoring framework of the application has the ability of real-time monitoring and evaluation, supports the classification of up to seven different phenological stages, including but not limited to germination stage, flowering stage, fruiting stage, leaf fall stage, etc. This refined classification method can better understand the response of plants to environmental changes and improve the efficiency of ecological monitoring.

[0070] The application modifies the ResNet network structure, adds a double-layer routing attention mechanism, and introduces a focal loss function to improve the feature extraction capability of the network, constructs a double-branch network model structure, realizes the mutual optimization of local features and global features, and improves the expression capability of the network. Using the fine-tuned network model, the MLP classification module is improved again, and the generalization ability of the model is enhanced by adding Dropout. Not only the learning ability of the model is enhanced, but also the performance of the model in different scenarios is optimized, so that it has stronger adaptability in the actual application of phenological monitoring.

[0071] Finally, the embodiment is tested on the phenological cameras of 4 public sites. The experiment proves that the ResFormer proposed in the application is superior to the most advanced method. Compared with Swin-TransFormer and other algorithms, the experiment proves that the ResFormer proposed in the application is superior to the most advanced method. The accuracy is improved by 0.31% on site 1, by 0.25% on site 2, by 0.18% on site 3, and by 2.25% on site 4. The confusion matrix of each data set is shown in Figures 4 to 7 .

Claims

1. A fine-grained phenological parameter extraction method based on feature fusion network, characterized in that: The following steps are involved: (1) Collect and preprocess photos taken by phenological cameras to construct a dataset suitable for training; (2) Extraction of phenological parameters; First, ResNet18 is introduced to extract global features through a 3×3 convolution kernel; Secondly, extract local features for lightweight parts; The specific method of local feature extraction is: A two-layer routing module is used to rearrange the input data and transform it into a shape of Then, linear projection is performed to obtain tensors of query, key, and value. The linear projection is calculated by multiplying the rearranged input data with the projection weight to obtain the corresponding query, key, and value. Routing between regions is achieved through a directed graph. In this phase, the query and key matrices at the region level are generated by averaging the queries and keys of each region; only the connections of each region with the top k most relevant regions are retained; The local context enhancement term is introduced to further improve the representation ability of local features; the specific formula is as follows: O=Attention(X r W q ,K g ,V g +LCE(X r W v ) K g =gather(K,topIndex(Q r (X r W k ) T )) V g =gather(V,topIndex(Q r (X r W v ) T ))) Among them, W q ,W k ,W v are the projection weights of Q query, K key and V value respectively, O is the sum of attention, Q is the linear projection, K g is the clustering key, V g is a value tensor, LCE(X r W v ) is the value tensor, which is the local context enhancement item. The number of channels for each attention is set to 32 and the kernel size is 5; The lightweight local feature extraction part uses the GELU activation function and cross entropy loss function, and uses the AdamW algorithm for training and the RandAugment automatic enhancement strategy to optimize the model parameters; the GELU activation function is as follows: Among them, 0.5 is a coefficient used to scale part of the input, x is the input value, and tanh represents the hyperbolic tangent function. Is a constant factor used to scale the input; 0.044715 is a constant used to adjust the cubic term of x to improve the approximation of the GELU function; The cross entropy loss function is as follows: C is the total number of categories, y i is the indicator variable of the true label, p i is the probability value of the i-th category predicted by the model, log(p i ) represents the logarithm of the model’s predicted probability; (3) After the two branches extract global and local features respectively, the adaptive feature fusion module is used to fuse the two parts of features, and the GELU activation function is used to process the spliced ​​features to generate a comprehensive feature representation: F fuse =Softmax(α×feature vector1+β×feature vector2) α and β are learnable weight parameters that control the weights of feature vector1 and feature vector2 in feature fusion respectively; feature vector1 and feature vector2 are global features and local features respectively; (4) The classification layer uses the Softmax function and adds the Dropout technology to map the fused features to the category space of different phenological periods, so as to achieve accurate classification of multiple phenological periods; when the model starts training, the model parameters will be fine-tuned in each round of training.

2. The fine-grained phenological parameter extraction method based on feature fusion network according to claim 1 is characterized in that: The specific method of step (1) is as follows: perform image quality and image content detection, manually identify difficult-to-classify or unclear images, mark these images as "need to be checked", and then delete them manually or by other means; and annotate the dataset, identify incorrectly labeled images through manual review and delete or re-label them, and then perform data enhancement. By rotating, cropping, and randomly masking part of the data, the robustness of the model is increased to construct a dataset suitable for training.

3. The fine-grained phenological parameter extraction method based on feature fusion network according to claim 1 is characterized in that: The specific method of global feature extraction in step (2) is: The pooling layer downsamples through the maximum pooling operation, gradually reducing the spatial resolution of the feature map, and passes the input directly to the output layer through skip connections. The feature map processed by a series of convolutional layers and pooling layers passes through the fully connected layer to extract the global semantic information of the phenotype. The focal loss function is used to alleviate the sensitivity of the residual network to gradient changes. The formula is as follows: FL(pt)=-α t (1-pt) γ log(pt) Among them, pt is the model's predicted probability for different phenological periods, α t is the weight coefficient for adjusting different categories, and γ is the focus parameter.

Images