Fine-grained phenological parameter extraction method based on feature fusion network

Abstract

The invention provides a fine-grained phenological parameter extraction method based on a feature fusion network, and the method comprises the following steps: (1) collecting and preprocessing photos shot in a phenological camera, and constructing a data set suitable for training; (2) phenological parameter extraction; the method comprises the following steps: firstly, introducing ResNet18, and carrying out global feature extraction through a 3 * 3 convolution kernel; secondly, extracting a local feature part for the lightweight; (3) respectively extracting global and local features from the two branches, fusing the two features by using an adaptive feature fusion module, and processing the spliced features by using a GELU activation function to generate comprehensive features; and (4) a Softmax function is used in the classification layer, a Dropout technology is added to map the fused features to category spaces of different phenological periods, and accurate classification of multiple phenological periods is realized. According to the invention, a high-precision phenological period monitoring framework is realized, the operation is simple, the fine granularity of phenological period extraction is high, and the method is suitable for ecological research in the plant growth process.

Description

Technical Field

[0001] The invention provides a fine-grained phenological parameter extraction method based on a feature fusion network, belonging to the technical field of image processing. Background Art

[0002] Vegetation phenology refers to the seasonal physiological changes and phenomena that occur in plants during their growth cycle, mainly including important stages such as germination, flowering, fruiting and leaf fall. These changes are affected by climate, environmental and ecological factors, and reflect the response of plants to climate change. By monitoring vegetation phenology, we can understand the health of ecosystems, predict agricultural production, assess the impact of climate change, and provide a scientific basis for ecological restoration and resource management. With the development of remote sensing technology and data analysis methods, the monitoring and research of vegetation phenology has become more and more accurate and efficient.

[0003] The vegetation phenological parameter extraction method is mainly for time series curves, and the process of vegetation phenological parameter calculation is mainly divided into two steps: time series reconstruction and phenological parameter extraction. The time series reconstruction process mainly includes time series harmonic analysis method, polynomial fitting method, SG filtering and local spline function fitting method; the phenological parameter extraction methods mainly include absolute threshold method, dynamic threshold method and derivative function extreme value method.

[0004] Most phenological parameter extraction methods can only extract phenological parameters from the time series composed of vegetation indices extracted from remote sensing images through smooth fitting and other methods, and then perform thresholding and other operations on the curve, which is very complicated. In addition, most phenological products can only extract the beginning and end of the growing season, and the granularity of phenological period extraction is low. This limitation restricts the in-depth understanding of plant responses to environmental changes, especially in response to climate change and ecological management needs. Summary of the invention

[0005] The present invention provides a fine-grained phenological parameter extraction method based on a feature fusion network to achieve the following objectives:

[0006] (1) In view of the defect of complex operation of existing phenological parameter extraction methods, the present invention proposes a fine-grained phenological parameter extraction framework (RBPhenology) suitable for real-time monitoring of multiple vegetations by phenological cameras. Based on the time series images taken by the phenological camera, the spatial and temporal information is used to determine the phenological parameters of each time series image from a number of pictures, and a training data set is constructed. The deep learning method is used to train a vegetation phenological parameter extraction model based on deep learning, and the phenological parameters of the test area are extracted based on deep learning, which solves the problem of complex operation of high-quality phenology.

[0007] (2) Existing methods can usually only extract a small number of specific phenological periods, which limits the comprehensive understanding of the plant growth process. The present invention proposes a dual-branch feature fusion network structure based on ResFormer, which can effectively identify up to seven different phenological periods, is suitable for fine-grained phenological monitoring tasks, and significantly improves the accuracy and richness of phenological parameter extraction.

[0008] The specific technical solutions are:

[0009] A fine-grained phenological parameter extraction method based on a feature fusion network comprises the following steps:

[0010] (1) Collect and preprocess photos taken by phenological cameras to construct a dataset suitable for training;

[0011] Image quality and image content detection is performed, and images that are difficult to classify or unclear are manually identified. These images are marked as "need to be checked" and then deleted manually or by other means. The dataset is annotated with labels, and images with incorrect labels are manually reviewed and deleted or re-labeled. Data enhancement is then performed to increase the robustness of the model by rotating, cropping, and randomly covering part of the data, so as to construct a dataset suitable for training.

[0012] (2) Extraction of phenological parameters;

[0013] First, ResNet18 is introduced to extract global features through a 3×3 convolution kernel;

[0014] The pooling layer downsamples through the maximum pooling operation, gradually reducing the spatial resolution of the feature map, and directly passes the input to the output layer through skip connections. After a series of convolutional layers and pooling layers, the feature map passes through the fully connected layer to extract the global semantic information of the phenotype.

[0015] The focal loss function is used to alleviate the sensitivity of the residual network to gradient changes. The formula is as follows:

[0016] FL(pt)=-α t (1-pt) γ log(pt)

[0017] Among them, pt is the model's predicted probability for different phenological periods, α t is the weight coefficient for adjusting different categories, and γ is the focus parameter.

[0018] Secondly, extract the local feature part for lightweight;

[0019] A two-layer routing module is used to rearrange the input data and transform it into a shape of Then, the tensors of query, key and value are obtained through linear projection. The linear projection is calculated by multiplying the rearranged input data with the projection weights to obtain the corresponding query, key and value. Routing between regions is achieved through a directed graph.

[0020] In this phase, the query and key matrices at the region level are generated by averaging the queries and keys of each region. Only the connections of each region with the top k most relevant regions are retained;

[0021] The local context enhancement term is introduced to further improve the representation ability of local features. The specific formula is as follows:

[0022] O = Attention(X r W q ,K g ,V g +LCE(X r W v )

[0023] K g =gather(K,topIndex(Q r (X r W k ) T ))

[0024] V g =gather(V,topIndex(Q r (X r W v ) T )))

[0025] Among them, W q ,W k ,W v are the projection weights of Q query, K key and V value respectively, O is the sum of attention, Q is the linear projection, K is the g is the clustering key, V g is a value tensor, LCE(X r W v ) is the value tensor, which is the local context enhancement item. The number of channels for each attention is set to 32 and the kernel size is 5;

[0026] The lightweight local feature extraction part uses the GELU activation function and cross entropy loss function, and uses the AdamW algorithm for training and the RandAugment automatic enhancement strategy to optimize the model parameters. The GELU activation function is as follows:

[0027]

[0028] Among them, 0.5 is a coefficient used to scale part of the input, x is the input value, and tanh represents the hyperbolic tangent function. is a constant factor, approximately equal to 0.79788, used to scale the input. 0.044715 is a constant used to adjust the cubic term of x to improve the approximation of the GELU function.

[0029] The cross entropy loss function is as follows:

[0030]

[0031] C is the total number of categories, which is set to 7 in the present invention. i is the indicator variable of the true label, p i is the probability value of the i-th category predicted by the model, log(p i ) represents the logarithm of the model's predicted probability.

[0032] (3) After the two branches extract global and local features respectively, the adaptive feature fusion module is used to fuse the two parts of features, and the GELU activation function is used to process the spliced ​​features to generate a more comprehensive feature representation:

[0033] F fuse =Softmax(α×feature vector1+β×feature vector2)

[0034] α and β are learnable weight parameters, which control the weights of feature vector1 and feature vector2 in feature fusion respectively. Feature vector1 and feature vector2 are global features and local features respectively.

[0035] (4) The classification layer uses the Softmax function and adds the Dropout technology to map the fused features to the category space of different phenological periods, thus achieving accurate classification of multiple phenological periods. After the model starts training, the model parameters will be fine-tuned in each round of training.

[0036] Compared with the prior art, the present invention has the following technical effects:

[0037] 1) The present invention realizes a high-precision phenological period monitoring framework and improves the previous complex phenological parameter extraction method, so it has certain commercial value.

[0038] 2) Compared with the previous phenological parameter extraction methods, the present invention is simple to operate, has high granularity in phenological period extraction, and is suitable for ecological research on plant growth processes.

[0039] 3) There is no need to record time series for a long time, and this method can be well applied to different tree species, various phenological camera sites, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a flow chart of the present invention;

[0041] Figure 2 is a fine-grained phenological period dataset of an embodiment;

[0042] Figure 3 is a fine-grained phenological period monitoring framework of an embodiment;

[0043] Figure 4 is the confusion matrix for fine-grained phenological period identification at site 1 of the embodiment;

[0044] Figure 5 is the confusion matrix for fine-grained phenological period identification at site 2 of the embodiment;

[0045] Figure 6 is the confusion matrix for fine-grained phenological period identification at site 3 of the embodiment;

[0046] Figure 7 is the confusion matrix of fine-grained phenological period identification for site 4 of the example. DETAILED DESCRIPTION

[0047] This embodiment is implemented through the existing deep learning framework Pytorch and the corresponding programming library, mainly including Numpy, PIL, Scipy, etc. Among them, Pytorch mainly uses pre-trained deep learning models, including linear modules, convolution modules, etc.

[0048] like Figure 1 As shown, a fine-grained phenological parameter extraction method based on a feature fusion network includes the following steps:

[0049] (1) Collect and preprocess the photos taken by the phenological camera to ensure the accuracy and consistency of the data, thereby improving the performance of subsequent models. Perform image quality and image content detection, manually identify difficult-to-classify or unclear images, mark these images as "need to be checked", and then delete them manually or by other means. In addition, label the dataset, manually review and identify images with incorrect labels and delete or re-label them, and then perform data enhancement to increase the robustness of the model by rotating, cropping, randomly covering part of the data, etc., to construct a dataset suitable for training, such as Figure 2 shown.

[0050] (2) Extraction of phenological parameters;

[0051] First, ResNet18 is introduced to extract features through a 3×3 convolution kernel. The pooling layer downsamples through the maximum pooling operation, gradually reducing the spatial resolution of the feature map, and directly passes the input to the output layer through skip connections, solving the problem of gradient vanishing in deep networks. The feature map processed by a series of convolutional layers and pooling layers will pass through the fully connected layer to extract the global semantic information of the phenotype. Since the residual network is sensitive to gradient changes, the present invention proposes to use the focal loss function to alleviate the sensitivity of the residual network to gradient changes. The formula is as follows:

[0052] FL(pt)=-α t (1-pt) γ log(pt)

[0053] Among them, pt is the model's predicted probability for different phenological periods, α t is the weight coefficient for adjusting different categories, and γ is the focus parameter.

[0054] Secondly, for the lightweight extraction of local features, this embodiment adopts a two-layer routing module to rearrange the input data and convert it into a shape of In the form of. Then, tensors of query, key and value are obtained by linear projection. The linear projection is calculated by multiplying the rearranged input data with the projection weights respectively to obtain the corresponding query, key and value. Routing between regions is implemented through a directed graph. At this stage, the query and key matrices at the regional level are generated by averaging the queries and keys of each region. In order to improve the computational efficiency, the present invention only retains the connection between each region and the top k most relevant regions, which can significantly reduce the computational complexity. Finally, the local context enhancement term is introduced to further enhance the representation ability of local features. The specific formula is as follows:

[0055] O = Attention(X r W q ,K g ,V g +LCE(X r W v )

[0056] K g =gather(K,topIndex(Q r (X r W k ) T ))

[0057] V g =gather(V,topIndex(Q r (X r W v )T )))

[0058] Among them, W q ,W k ,W v are the projection weights of Q query, K key and V value respectively, O is the sum of attention, Q is the linear projection, K is the g is the clustering key, V g is a value tensor, LCE(X r W v ) is the value tensor, which is the local context enhancement item. The number of channels for each attention is set to 32 and the kernel size is 5;

[0059] The lightweight local feature extraction part uses the GELU activation function and cross entropy loss function, and uses the AdamW algorithm for training and the RandAugment automatic enhancement strategy to optimize the model parameters. The GELU activation function is as follows:

[0060]

[0061] Among them, 0.5 is a coefficient used to scale part of the input, x is the input value, and tanh represents the hyperbolic tangent function. is a constant factor, approximately equal to 0.79788, used to scale the input. 0.044715 is a constant used to adjust the cubic term of x to improve the approximation of the GELU function. The shape of the GELU function is smoother than that of the ReLU, and is suitable for complex nonlinear tasks in neural networks. The cross entropy loss function is as follows:

[0062]

[0063] C is the total number of categories, which is set to 7 in this experiment. i is the indicator variable of the true label, p i is the probability value of the i-th category predicted by the model, log(p i ) represents the logarithm of the model's predicted probability.

[0064] (3) After the two branches extract global and local features respectively, the model uses an adaptive feature fusion module to fuse the two parts of features and further processes the concatenated features using the GELU activation function to generate a more comprehensive feature representation.

[0065] F fuse =Softmax(α×feature vector1+β×feature vector2)

[0066] α and β are learnable weight parameters, which control the weights of feature vector1 and feature vector2 in feature fusion respectively. Feature vector1 and feature vector2 are global features and local features respectively.

[0067] (4) The classification layer uses the Softmax function and adds the Dropout technology to map the fused features to the category space of different phenological periods, thereby achieving accurate classification of multiple phenological periods. When the model starts training, the model parameters will be fine-tuned in each round of training.

[0068] The present invention proposes a fine-grained phenological parameter extraction framework (RBPhenology) suitable for real-time monitoring of multiple vegetation by phenological cameras and a dual-branch feature fusion network structure based on ResFormer.

[0069] like Figure 3 As shown, the real-time fine-grained phenological period monitoring framework of the present invention has the ability of real-time monitoring and evaluation, and supports the classification of up to seven different phenological periods, including but not limited to budding period, flowering period, fruiting period, leaf fall period, etc. This refined classification method can better understand the response of plants to environmental changes and improve the efficiency of ecological monitoring.

[0070] The present invention modifies the ResNet network structure, adds a double-layer routing attention mechanism, and introduces a focal loss function to improve the network's feature extraction capability, build a dual-branch network model structure, achieve mutual optimization of local features and global features, and improve the network's expression capability. Using the fine-tuned network model, the MLP classification module is improved again, and the generalization ability of the model is enhanced by adding Dropout, which not only enhances the model's learning ability, but also optimizes the model's performance in different scenarios, making it more adaptable in practical applications of phenological monitoring.

[0071] Finally, this example was experimented on phenological cameras at four public sites. The experiments show that the proposed ResFormer is superior to the most advanced methods. Compared with algorithms such as Swin-TransFormer, the proposed ResFormer is superior to the most advanced methods at four public sites, with an accuracy improvement of 0.31% at site 1, 0.25% at site 2, 0.18% at site 3, and 2.25% at site 4. The confusion matrices of the datasets are shown in Figure 2. Figures 4 to 7 shown.

Claims

1. A fine-grained phenological parameter extraction method based on feature fusion network, characterized in that: The following steps are involved: (1) Collect and preprocess photos taken by phenological cameras to construct a dataset suitable for training; (2) Extraction of phenological parameters; First, ResNet18 is introduced to extract global features through a 3×3 convolution kernel; Secondly, extract the local feature part for lightweight; (3) After the two branches extract global and local features respectively, the adaptive feature fusion module is used to fuse the two parts of features, and the GELU activation function is used to process the spliced ​​features to generate a comprehensive feature representation: F fuse =Softmax(α×feature vector1+β×feature vector2) α and β are learnable weight parameters, which control the weights of feature vector1 and feature vector2 in feature fusion respectively; feature vector1 and feature vector2 are global features and local features respectively; (4) The classification layer uses the Softmax function and adds the Dropout technology to map the fused features to the category space of different phenological periods, thereby achieving accurate classification of multiple phenological periods. After the model starts training, the model parameters will be fine-tuned in each round of training.

2. According to the method for extracting fine-grained phenological parameters based on feature fusion network in claim 1, it is characterized in that: The specific method of step (1) is as follows: image quality and image content detection is performed, and images that are difficult to classify or unclear are manually identified, and these images are marked as "need to be checked", and then deleted manually or by other means; and the data set is labeled and annotated, and images with incorrect labels are identified through manual review and deleted or re-labeled, and then data enhancement is performed to increase the robustness of the model by rotating, cropping, and randomly masking part of the data, so as to construct a data set suitable for training.

3. According to the method for extracting fine-grained phenological parameters based on feature fusion network in claim 1, it is characterized in that: The specific method of global feature extraction in step (2) is: The pooling layer downsamples through the maximum pooling operation, gradually reducing the spatial resolution of the feature map, and directly passes the input to the output layer through skip connections; the feature map processed by a series of convolutional layers and pooling layers passes through the fully connected layer to extract the global semantic information of the phenological period; The focal loss function is used to alleviate the sensitivity of the residual network to gradient changes. The formula is as follows: FL(pt)=-α t (1-pt) γ log(pt) Among them, pt is the model's predicted probability for different phenological periods, α t is the weight coefficient for adjusting different categories, and γ is the focus parameter.

4. According to the method for extracting fine-grained phenological parameters based on feature fusion network in claim 1, it is characterized in that: The specific method of local feature extraction in step (2) is: A two-layer routing module is used to rearrange the input data and transform it into a shape of In the form of; then, tensors of query, key and value are obtained through linear projection; the linear projection is calculated by multiplying the rearranged input data with the projection weight respectively to obtain the corresponding query, key and value; the routing between regions is realized through a directed graph; In this phase, the query and key matrices at the region level are generated by averaging the queries and keys of each region; only the connections of each region to the top k most relevant regions are retained; The local context enhancement term is introduced to further improve the representation ability of local features; the specific formula is as follows: O=Attention(X r W q ,K g ,V g +LCE(X r W v ) K g =gather(K,topIndex(Q r (X r W k ) T )) V g =gather(V,topIndex(Q r (X r W v ) T ))) Among them, W q ,W k ,W v are the projection weights of Q query, K key and V value respectively, O is the sum of attention, Q is the linear projection, K is the g is the clustering key, V g is a value tensor, LCE(X r W v ) is the value tensor, which is the local context enhancement item. The number of channels for each attention is set to 32 and the kernel size is 5; The lightweight local feature extraction part uses the GELU activation function and cross entropy loss function, and uses the AdamW algorithm for training and the RandAugment automatic enhancement strategy to optimize the model parameters; the GELU activation function is as follows: Among them, 0.5 is a coefficient used to scale part of the input, x is the input value, and tanh represents the hyperbolic tangent function. is a constant factor used to scale the input; 0.044715 is a constant used to adjust the cubic term of x to improve the approximation of the GELU function; The cross entropy loss function is as follows: C is the total number of categories, y i is the indicator variable of the true label, p i is the probability value of the i-th category predicted by the model, log(p i ) represents the logarithm of the model's predicted probability.

Images