A crop phenology fine-grained recognition method, device, equipment and medium

Abstract

The application discloses a crop phenology fine-grained identification method, device, equipment and medium. The method combines a double attention mechanism, a global description operator and a ResNet50 model to obtain an AResNet50 model and training is performed until a preset standard or an iteration number is reached, and a trained AResNet50 model is output; a to-be-identified image is input into the trained AResNet50 model to output a first classification result, and high-order features of the to-be-identified image are extracted; the to-be-identified image is input into a phenotype similarity model to obtain a phenotype similarity of the crop in combination with the high-order features, and a second classification result is determined based on a KNN algorithm and the phenotype similarity of the crop; and the first classification result and the second classification result are fused by using a late fusion method to obtain a final classification result. The application can effectively improve fine-grained identification of water-cultivated crops in an entire growth period in an indoor scene.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a method, apparatus, equipment and medium for fine-grained identification of crop phenology. Background Technology

[0002] In order to increase crop yield, reduce waste of resources (energy, fertilizer, fuel, etc.) and harvest crops in a timely manner, monitoring crop growth is a very important part of the agricultural system. In the monitoring process, accurately identifying crop phenological stages is an important reference indicator for guiding spraying and an important factor in determining the quantity and type of crop nutrients. However, the basic method of obtaining crop phenology depends on human observation of plants, and manual methods have problems such as large workload, low accuracy and inability to monitor in real time. In order to better distinguish the growth stages of crops, in 1991, Lancashire et al. proposed the BBCH phenological scale, which divides the plant growth and development process into 80 main stages, covering the various phenological stages from seed germination, sprouting, leaf growth, flowering and senescence[4]. The BBCH scale can define different crop growth stages, providing a guide for phenological stage information for later scholars. However, the BBCH scale classifies crop phenology by phenotypic traits such as the number and shape of plant leaves. For similar phenological stages of crops, the differences in phenotypic traits are small, which increases the difficulty of manual observation and also poses a challenge to automated methods.

[0003] Manual identification of crop phenology relies on visual observation of changes in the external morphology and structure of crops, enabling the use of new technologies to distinguish phenological stages. With the development of computer vision technology, it has been widely applied in agriculture and successfully used for phenological stage identification in crops such as rice, cotton, and corn. Therefore, several computer vision-based methods for crop phenological stage identification have been proposed, which can be categorized into traditional machine learning methods and deep learning-based methods. Traditional machine learning methods typically use vegetation indices or color features, followed by machine learning algorithms such as k-nearest neighbors (KNN) and support vector machine (SVM) for phenological stage classification. For example, in 2018, Seyednasrollah proposed a green chromaticity coordinate (GCC) method for estimating phenology based on color space features, using the intensity and hue of the green component to determine the crop's growth stage. In 2019, Hufkens et al. used the GCC phenological index to quantify important phenological stages of crops, accurately monitoring the four main phenological stages of wheat by quantifying near-surface remote sensing images. In the same year, Zainuddin et al. used histogram methods to extract leaf color image features and used an SVM classifier to classify phenological stages. In 2021, Ge et al. extracted three features—RGB-VIs, color space, and textures—from UAV RGB images, integrating five algorithms—KNN, SVM, and RF—as an ensemble model to identify the main phenological stages of rice. In 2022, Zhang et al. used an improved shape-model method to extract multi-peak features of three vegetation indices for winter rapeseed and used the SMM transformation equation to identify six phenological stages of winter rapeseed. The methods mentioned above typically use four phenological indicators based on color indices (NDVI, GCC, EXG, and EVI), which can only identify the main growth stages of crops at the plot and regional scale. However, the color characteristics of adjacent growth stages of crops are very similar, so it is impossible to achieve fine-grained phenological period identification based on the BBCH scale. To determine crop phenological periods more accurately, it is necessary to combine other information such as crop structure, morphology, and leaf size.

[0004] In recent years, with the excellent results achieved by deep learning in image classification, object detection, and image segmentation, deep learning has been widely applied in plant counting and pest and disease identification, and has also begun to be applied in plant growth stage identification. In 2018, Zhang Yunde et al. proposed a method for corn growth stage identification based on feature extraction from deep convolutional neural networks. This method extracts features of different growth stages of corn through deep convolutional neural networks and combines particle swarm optimization to optimize SVM parameters, constructing a multi-level support vector machine model to classify corn. Yang et al. designed an efficient convolutional neural network (CNN) structure and identified nine main growth stages of rice through RGB images, demonstrating the excellent performance of the proposed deep learning method in near real-time phenological identification. Han et al. proposed an efficient deep convolutional neural network (DCNN) structure, using photos taken by a handheld camera to detect rice development stages. DCNN achieved more accurate growth stage classification. Rasti used CNN to classify images of 12 wheat growth stages and 11 barley growth stages taken before canopy closure. The aforementioned studies have demonstrated that deep learning offers higher accuracy and robustness than traditional machine learning methods in crop phenological stage identification. While the image features of major crop phenological stages differ significantly, deep learning methods can accurately identify specific phenological stages. However, due to the minimal feature differences between adjacent crop phenological stages, deep learning methods also cannot identify all BBCH categories with fine-grained precision.

[0005] The aforementioned studies, whether employing machine learning or deep learning methods, are based on crop population characteristics and can only achieve phenological identification of group crops. However, with the application of technologies such as plant factories, facility agriculture, and robotics in agriculture, the phenological identification of individual crops is also of great significance for precision agriculture, as their growth status is a crucial basis for precise harvesting and pesticide spraying. Individual crop growth information mainly includes phenotypic features such as shape, color, and plant height, which can be used to more effectively identify crop phenology. Traditional machine learning methods primarily identify the phenology of individual crops through phenotypic information such as leaf shape, leaf texture, and leaf quantity. For example, in 2013, Yu extracted image features from a single corn plant to identify the emergence and three-leaf stages; in 2017, Minervini et al. analyzed the relationship between the number of plant leaves and growth stages; and in 2020, Concepcion extracted morphological features such as lettuce head circumference, area, and dominant leaf length to classify lettuce into three growth stages. Deep learning, however, has greater advantages in monitoring the growth of individual crops, and deep learning-based image classification, object detection, and image segmentation methods are widely used in monitoring the growth of individual crops. Furthermore, Ngorima et al., in an indoor farm environment, combined a deep learning feature extractor with a traditional machine learning classifier to classify the four growth stages of two broadleaf hydroponic crops, overcoming the problem that the similarity of morphology at different growth stages of hydroponic crops complicates phenological identification. In addition, plant individual similarity measurements have been applied in plant classification and variety identification, but research on phenological identification is limited. Traditional plant similarity studies have mostly focused on plant geometry, color, and texture. In 2012, Zhou and Dong studied the similarity between individual plant leaves based on color and texture features, respectively; in 2015, Ding et al. proposed a method for calculating plant body similarity by integrating multiple morphological features such as plant topology, external outline, and internal details; in 2019, Ding et al. proposed a method for calculating plant morphological similarity based on three aspects of image features: the looseness of plant branches, the density of leaves, and the color of leaves. These methods for calculating plant image similarity are all designed for plant variety classification. However, the differences between images of plants of the same variety at similar growth stages are very small, making it very difficult to directly use image similarity for crop phenological identification. In 2021, Gao et al. integrated HOG features, SILTP features, and color features, and used the cross-view quadratic discriminant analysis (XQDA) algorithm to measure similarity for classifying and identifying the growth stages of individual maize plants. However, they only achieved classification for four growth stages with relatively obvious feature differences.

[0006] In summary, many fruitful studies have been conducted on crop growth stage monitoring using machine learning and deep learning methods. However, the visual characteristics of crops with similar growth stages classified according to the BBCH scale are relatively small, making it very difficult to achieve fine-grained identification of individual crop phenological stages using either traditional machine learning or deep learning methods. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a method, apparatus, device, and medium for fine-grained identification of crop phenology. It also proposes a fine-grained identification model (PhenologyNet) that integrates convolutional neural networks and phenotypic similarity, which can effectively improve the fine-grained identification of hydroponic crops throughout their entire growth period in indoor settings.

[0008] To achieve the above objectives, the technical solution of the present invention is as follows:

[0009] A method for fine-grained identification of crop phenology includes the following steps:

[0010] The AResNet50 model is obtained by combining the dual attention mechanism, the global description operator and the ResNet50 model and training it until the number of iterations or the preset standard is reached, and the trained AResNet50 model is output. The image to be recognized is input into the trained AResNet50 model and the first classification result is output, and the high-order features of the image to be recognized are extracted.

[0011] The image to be identified is input into the phenotypic similarity model and combined with higher-order features to obtain the phenotypic similarity of the crop. The second classification result is determined based on the KNN algorithm and the phenotypic similarity of the crop.

[0012] The late fusion method is used to fuse the first and second classification results to obtain the final classification result.

[0013] Preferably, the AResNet50 model uses the ReLU activation function and cross-entropy loss function in the fully connected layers and convolutional layers of the model, wherein the fully connected layers of the ResNet50 model are adjusted to BN layers and fully connected layers.

[0014] Preferably, the AResNet50 model is trained using a mini-batch stochastic gradient descent algorithm to optimize the model parameters.

[0015] Preferably, the dual attention mechanism includes a channel attention mechanism and a spatial attention mechanism.

[0016] Preferably, the step of inputting the image to be identified into a phenotypic similarity model and combining it with higher-order features to obtain the phenotypic similarity of crops specifically includes the following steps:

[0017] The ORB algorithm is used to extract low-order features from the image to be identified, and these low-order features are concatenated with high-order features. Dimensionality reduction is then performed using PCA to obtain a fused feature vector X. Image similarity is then calculated. As shown in the following formula:

[0018]

[0019] Where X is the fused feature vector after PCA dimensionality reduction, and F concat For feature fusion operators, This represents the feature vector of the i-th image. Dot( represents the feature vector of the j-th image) ) represents a vector and dot product, and | Representing vectors respectively and norm, For image and The image similarity after normalization ranges from 0 to 1;

[0020] The U-NET model with an attention mechanism is used to separate crops from the background in the image to be identified. Phenotypic features of the crops are extracted using relevant methods provided by Python. Normalized cosine similarity is used to calculate the phenotypic similarity between the image to be identified and the training set images. As shown in the following formula,

[0021]

[0022] In the formula, Let represent the phenotypic parameter vectors of the i-th and j-th images, respectively. The phenotypic similarity is the normalized phenotypic parameter similarity, with values ​​ranging from 0 to 1;

[0023] The phenotypic similarity is obtained by fusing image similarity with phenotypic parameter similarity, as shown in the following formula:

[0024]

[0025] Among them, S i,j Let γ be the phenotypic similarity between image i and image j, γ be the image similarity weight, and β be the phenotypic parameter similarity weight. The phenotypic similarity value ranges from 0 to 1.

[0026] Preferably, the determination of the second classification result based on the KNN algorithm and crop phenotypic similarity specifically includes the following steps:

[0027] The K training samples that are most similar to the image to be classified are determined based on the phenotypic similarity of the crops.

[0028] The image to be classified is assigned to the category that appears most frequently in the K training samples to obtain the second classification result.

[0029] Preferably, the step of fusing the first classification result and the second classification result using the late fusion method to obtain the final classification result specifically includes the following steps:

[0030] The random forest method is used to fuse the first and second classification results, and the output is a probability vector p = (p1, p2, ..., p...). k Perform soft voting fusion of multiple classifiers according to the following steps:

[0031] Calculate the entropy of the i-th classifier, as shown in Formula 7:

[0032]

[0033] Accuracy of the comprehensive classifier i and entropy e i Calculate the entropy-weighted weights of the i-th classifier, as shown in Formula 8:

[0034]

[0035] Where α is the balance factor and β is the entropy weighting coefficient.

[0036] Calculate the normalized weights of the i-th classifier, as shown in Formula 9:

[0037]

[0038] Where M is the number of classifiers,

[0039] The fusion probability vector is obtained by weighted averaging the predicted probabilities of the M classifiers, as shown in Formula 10:

[0040]

[0041] The category with the highest fusion probability is selected as the final prediction result, as shown in Formula 11:

[0042]

[0043] Among them, y pred This represents the final category label, and k represents the number of categories.

[0044] Based on the above, the present invention also discloses a crop phenology fine-grained identification device, comprising: a first classification unit, a second classification unit, and a fusion unit, wherein,

[0045] The first classification unit is used to combine the dual attention mechanism, the global descriptor operator and the ResNet50 model to obtain the AResNet50 model and train it until the number of iterations or a preset standard is reached, and output the trained AResNet50 model; input the image to be recognized into the trained AResNet50 model to output the first classification result, and extract the high-order features of the image to be recognized.

[0046] The second classification unit is used to input the image to be identified into the phenotypic similarity model and combine it with higher-order features to obtain the phenotypic similarity of the crop, and to determine the second classification result based on the KNN algorithm and the phenotypic similarity of the crop.

[0047] The fusion unit is used to fuse the first classification result and the second classification result using the late fusion method to obtain the final classification result.

[0048] Based on the foregoing, the present invention also discloses a computer device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement any of the methods described above.

[0049] Based on the foregoing, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the methods described above.

[0050] Based on the above technical solution, the beneficial effects of the present invention are as follows: The above-mentioned crop phenology fine-grained recognition method, device, equipment, and medium combine a dual attention mechanism, a global descriptor operator, and a ResNet50 model to obtain an AResNet50 model and train it until the required number of iterations or a preset standard is reached, and output the trained AResNet50 model; the image to be recognized is input into the trained AResNet50 model to output a first classification result, and high-order features of the image to be recognized are extracted; the image to be recognized is input into a phenotypic similarity model and combined with high-order features to obtain the phenotypic similarity of the crop, and a second classification result is determined based on the KNN algorithm and the phenotypic similarity of the crop; the first classification result and the second classification result are fused using the late fusion method to obtain the final classification result. This provides a crop phenology fine-grained recognition model (PhenologyNet) that integrates convolutional neural networks and phenotypic similarity, which can effectively improve the fine-grained recognition of hydroponic crops throughout their entire growth period in indoor settings. Attached Figure Description

[0051] Figure 1 This is an application environment diagram of a crop phenology fine-grained identification method in one embodiment;

[0052] Figure 2 This is a flowchart illustrating a fine-grained crop phenology identification method in one embodiment;

[0053] Figure 3 This is a schematic diagram of the overall structure of PhenologyNet in a crop phenology fine-grained identification method in one embodiment;

[0054] Figure 4 This is a schematic diagram of image data in a crop phenology fine-grained identification method in one embodiment;

[0055] Figure 5 This is a schematic diagram of the structure of a CNN Classifier in a crop phenology fine-grained identification method in one embodiment;

[0056] Figure 6 This is a schematic diagram illustrating the introduction of an attention mechanism in a fine-grained crop phenology identification method in one embodiment;

[0057] Figure 7 This is a schematic diagram of the image similarity structure in a crop phenology fine-grained identification method in one embodiment;

[0058] Figure 8 This is a schematic diagram of the phenotypic parameter similarity structure in a crop phenology fine-grained identification method in one embodiment;

[0059] Figure 9 This is a schematic diagram of the segmentation result in one embodiment;

[0060] Figure 10 This is a schematic diagram of phenotypic parameter results in one embodiment;

[0061] Figure 11 This is a schematic diagram of a reference image in one embodiment;

[0062] Figure 12 This is a schematic diagram of the phenotypic similarity module in one embodiment;

[0063] Figure 13 This is an example diagram of the classification results of a KNN model based on phenotypic similarity in one embodiment;

[0064] Figure 14 This is a schematic diagram of the fitting curve of the prediction result of the Phenology model in one embodiment;

[0065] Figure 15 This is a schematic diagram of the structure of a crop phenology fine-grained identification device in one embodiment. Detailed Implementation

[0066] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0067] This application provides a method for fine-grained identification of crop phenology, which can be applied to, for example... Figure 1 The application environment shown. For example... Figure 1 As shown, the application environment includes computer device 110. Computer device 110 can combine an attention mechanism, a global descriptor operator, and a ResNet50 model to obtain an AResNet50 model and train it until the required number of iterations or a preset standard is reached, outputting the trained AResNet50 model. The image to be recognized is input into the trained AResNet50 model, which outputs a first classification result and extracts high-order features from the image. Computer device 110 can then input the image to be recognized into a phenotypic similarity model, combining the high-order features to obtain the phenotypic similarity of the crop, and determine a second classification result based on the KNN algorithm and the phenotypic similarity. Finally, computer device 110 can use a late fusion method to fuse the first and second classification results to obtain the final classification result. Computer device 110 can be, but is not limited to, various personal computers, laptops, robots, tablets, and other devices.

[0068] like Figure 2 As shown, this embodiment provides a fine-grained crop phenology identification method, proposing a fine-grained crop phenology identification model that integrates convolutional neural networks and phenotypic similarity, called PhenologyNet (Fine-grained Growth Stage Identification Network). The overall structure of this model is as follows: Figure 3 As shown, PhenologyNet mainly consists of two parts: a CNN Classifier and a Phenotype Similarity Classifier. The CNN Classifier primarily comprises an improved ResNet50 model, referred to as AResNet50. The Phenotype Similarity Classifier module mainly consists of image similarity, phenotypic parameter similarity, and the KNN algorithm, as shown in the diagram. Figure 4 The image data provided in this embodiment specifically includes the following steps:

[0069] Step 210: Combine the dual attention mechanism, global description operator and ResNet50 model to obtain AResNet50 model and train it until the number of iterations or preset standard is reached, and output the trained AResNet50 model; input the image to be recognized into the trained AResNet50 model to output the first classification result, and extract the high-order features of the image to be recognized.

[0070] The computer device uses a CNN Classifier to perform classification and output the classification results. The structure of this module is as follows: Figure 5 As shown, firstly, the CNN Classifier uses the ResNet50 model as its backbone network, combining a dual attention mechanism (channel attention and spatial attention) and a global descriptor operator with the ResNet50 model. Specifically, the channel attention and spatial attention modules are combined sequentially and placed after the residual unit to help the model extract key image features, improving its robustness, generalization ability, and accuracy. The global descriptor operator is integrated, using pooling or aggregation operations on the feature maps to integrate local information into a global feature vector, which represents the features of the entire image and thus better represents the image content. Secondly, the last fully connected layer of the ResNet50 model is removed, and BatchNormalization and a new fully connected layer are added based on the number of categories in the crop phenology dataset. This processes the global descriptor operator features to assist classification, and the ReLU activation function and cross-entropy loss function are applied to the fully connected and convolutional layers of the model. Thirdly, mini-batch stochastic gradient descent is used for optimization to improve the model's convergence. Normalization solves the problems of vanishing and exploding gradients, further improving model performance; fourth, the classification probabilities of each phenological stage are output through a fully connected layer; fifth, the AResNet50 model is trained using the collected image data and the weight file is saved; finally, the trained AResNet50 model is used to identify the category and classification probability of the image to be identified.

[0071] The attention mechanism combines the channel attention mechanism and the spatial attention mechanism modules in a sequential manner, placing them after the residual unit. The specific method is as follows: Figure 6 As shown, the image passes through the ResNet50 backbone network to obtain feature X, then through the channel attention mechanism module to obtain feature X1, and finally through the spatial attention mechanism module to obtain X2. X2 replaces the original feature X for residual calculation, as shown in Formulas 1 and 2. This cascaded structure helps the model better understand the relationship between different channels and spatial locations in the image, thereby improving the feature representation ability and model performance.

[0072]

[0073] in and These represent the weight coefficients of the channel attention module and the spatial attention module, respectively. This indicates that the corresponding elements in the matrix are multiplied one by one.

[0074] Step 220: Input the image to be identified into the phenotypic similarity model and combine it with higher-order features to obtain the phenotypic similarity of the crop, and determine the second classification result based on the KNN algorithm and the phenotypic similarity of the crop.

[0075] The computer equipment uses the AResNet50 model and ORB algorithm to extract high-order and low-order features of the image, respectively, and then concatenates them to obtain new image features. Next, the features are dimensionality-reduced using PCA, and image similarity is calculated using normalized cosine similarity. Third, the U-NET model with an attention mechanism is used to distinguish the background from the crops in the image, and phenotypic features of the crops are extracted using relevant methods provided by Python. Phenotypic similarity is then calculated using normalized cosine similarity. Fourth, image similarity and phenotypic parameter similarity are fused using different weights to obtain phenotypic similarity, and the KNN algorithm is used to output the classification results of the phenotypic similarity algorithm.

[0076] Step 230: Use the late fusion method to fuse the first classification result and the second classification result to obtain the final classification result.

[0077] The computer equipment uses the random forest algorithm in the late fusion method to fuse the image classification results output by the AResNet50 model with the classification results output by the KNN algorithm to obtain the final classification result.

[0078] In one embodiment of a crop phenology fine-grained identification method, a specific process is also provided for inputting the image to be identified into a phenotypic similarity model to obtain phenotypic similarity, and determining the second classification result based on the KNN algorithm and the phenotypic similarity of the crops, including:

[0079] Step 221 calculates image similarity using normalized cosine distance. The AResNet50 model trained in Step 210 extracts high-order features from the input image, then uses the ORB algorithm to extract low-order features. These two features are then concatenated and fused, and PCA is used to reduce the dimensionality of the fused features. The concatenation method is shown in Equation 3. Furthermore, normalized cosine distance is used to calculate the image similarity between the image to be recognized and the training set images, as shown in Equation 4. This feature fusion method can compensate for the deficiencies of both high-order and low-order features, fully leveraging the advantages of both to more comprehensively represent image features. This module is as follows: Figure 7 As shown.

[0080]

[0081] Where X is the fused feature vector after PCA dimensionality reduction, and F concatFor feature fusion operators, .

[0082]

[0083] in, This represents the feature vector of the i-th image. Dot( represents the feature vector of the j-th image) ) represents a vector and dot product, and | Representing vectors respectively and norm, For image and The image similarity after normalization ranges from 0 to 1.

[0084] Step 222: Calculate the phenotypic parameter similarity. The normalized cosine distance of Formula 5 is used to calculate the phenotypic parameter similarity between images. First, the U-Net model proposed by Olaf Ronneberger in 2015 is used to separate crops from the background, enabling more accurate extraction of crop phenotypic features and ultimately improving classification accuracy. Second, relevant API methods provided by OpenCV are used to extract crop phenotypic features, which mainly include projected area (PA), projected perimeter (PP), convex hull area (CA), convex hull perimeter (CP), circumcircle diameter of the projected area (PCD), width of the minimum area rectangle of the projected area (ARW), and height of the minimum area rectangle of the projected area (ARW). Finally, the phenotypic similarity between the image to be identified and the training set images is calculated using normalized cosine distance. Its structure is as follows: Figure 8 As shown, the segmentation effect is as follows Figure 9 As shown, the phenotypic feature extraction results are as follows: Figure 10 As shown.

[0085]

[0086] In the formula, Let represent the phenotypic parameter vectors of the i-th and j-th images, respectively. This represents the phenotypic similarity after normalization, with values ​​ranging from 0 to 1.

[0087] Calculating phenotypic features such as projected area requires the use of a reference object method. Taking the calculation of projected area as an example, firstly, images of the reference object are acquired using cameras of the same resolution at the same height. The reference object images are then segmented, and the projected area of ​​the reference object is obtained by calculating the number of pixels. Since the actual area of ​​the reference object is known, the actual projected area of ​​the crop can then be calculated. In this embodiment, a 5-cent coin is used as the reference object. Figure 11 As shown.

[0088] Step 223: Calculate phenotypic similarity. Different individuals of the same crop variety exhibit high similarity in phenotypic characteristics within the same phenological stage. Although the main phenotypic characteristics of crops are expressed through visual features such as morphology, size, and color, image features can also differ due to growth variations. Therefore, neither image similarity nor phenotypic parameter similarity can be used as the primary basis for phenological stage classification. To improve the accuracy of phenotypic similarity measurement, this invention fuses image similarity and phenotypic parameter similarity to obtain phenotypic similarity. This module is as follows: Figure 12 As shown. Step 2 is used to obtain the image similarity between the sample to be classified and each training sample, and Step 3 is used to obtain the phenotypic parameter similarity between the sample to be classified and each training sample. Then, Formula 6 is used to fuse the image similarity and phenotypic parameter similarity to calculate the phenotypic similarity between the sample to be classified and each training sample.

[0089]

[0090] Among them, S i,j Let γ be the phenotypic similarity between image i and image j, γ be the image similarity weight, and β be the phenotypic parameter similarity weight. The phenotypic similarity value ranges from 0 to 1.

[0091] Step 224: The KNN algorithm is used to classify the phenotypic similarity between the target image and each training image, outputting the probability of all phenological period categories. The KNN algorithm uses the calculated phenotypic similarity to find the K training samples most similar to the image to be classified, and assigns the image to be classified to the category that appears most frequently among the K training samples, outputting the classification results for all phenological periods. The classification results are as follows: Figure 13 As shown.

[0092] One embodiment of a crop phenology fine-grained identification method also provides a specific process for fusing the first classification result and the second classification result using a late fusion method to obtain the final classification result, including:

[0093] The computer equipment employs soft voting, which considers classifier confidence, as a late fusion method, supplemented by entropy or other information entropy-related metrics. Entropy is used as an auxiliary indicator, and adjusting this indicator corrects the classifier weights to balance the two important metrics of accuracy and entropy, thus more comprehensively evaluating the performance of each classifier, reducing differences between models, and improving the final classification accuracy. This invention uses a random forest method to fuse two classifiers: the AResNet50 model and a phenotypic similarity model. Both predict 28 categories, and the output is a probability vector p = (p1, p2, ..., p...). k Perform soft voting fusion of multiple classifiers according to the following steps:

[0094] (1) Calculate the entropy of the i-th classifier, as shown in Formula 7:

[0095]

[0096] (2) Accuracy of the comprehensive classifier i and entropy e i Calculate the entropy-weighted weights of the i-th classifier, as shown in Formula 8:

[0097]

[0098] Here, α is the balance factor and β is the entropy weighting coefficient. These two parameters can be adjusted according to specific circumstances.

[0099] (3) Calculate the normalized weights of the i-th classifier, as shown in Formula 9:

[0100]

[0101] Where M is the number of classifiers,

[0102] (4) The fusion probability vector is obtained by weighted averaging the predicted probabilities of the M classifiers, as shown in Formula 10:

[0103]

[0104] (5) Select the category with the highest fusion probability as the final prediction result, as shown in Formula 11:

[0105]

[0106] Among them, y pred This represents the final category label, and k represents the number of categories.

[0107] It should be understood that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart above may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0108] Experimental Analysis

[0109] Table 1

[0110]

[0111] As shown in Table 1, among the five models compared, the PhenologyNet model has the highest accuracy and the highest mPrecision. The experimental results demonstrate the effectiveness of the PhenologyNet model in fine-grained crop phenological period classification.

[0112] Table 2

[0113]

[0114] The experimental results in Table 2 show that fusing ORB features or phenotypic features into the AResNet50 model significantly improves the accuracy of fine-grained crop phenological classification. Compared to the AResNet50 model, the accuracy of the other three models improved by 4.48%, 5.35%, and 11.6%, respectively, while the mPrecision improved by 4.04%, 6.01%, and 10.04%, respectively. However, the PhenologyNet model of this invention exhibits the highest accuracy and mPrecision. The experimental results clearly demonstrate that fusing ORB features and phenotypic features is more effective in improving the accuracy of fine-grained phenological classification, and also prove the effectiveness of the PhenologyNet model in fine-grained crop phenological classification.

[0115] Table 3

[0116]

[0117] Using hydroponic lettuce as the research object, a new crop phenological period recognition model, PhenologyNet, was obtained by fusing phenotypic similarity classification with deep learning classification. Figure 14The fitted curve of the Phenology model prediction results shows that the model achieved a classification accuracy of 91.96% for 28 BBCH phenological categories, which is significantly higher than that of CNN models that use image features alone. Furthermore, the fusion of phenotypic features can significantly improve the fine-grained classification performance of crop phenological stages. In addition, as shown in Table 3, the classification accuracy of PhenologyNet gradually increases as the number of categories decreases. When the number of categories is 3, the accuracy reaches 98.34%, which can meet the requirements of precision agriculture production scenarios.

[0118] like Figure 15 As shown, this embodiment provides a crop phenology fine-grained identification device 300, including: a first classification unit 310, a second classification unit 320, and a fusion unit 330, wherein,

[0119] The first classification unit 310 is used to combine the dual attention mechanism, the global description operator and the ResNet50 model to obtain the AResNet50 model and train it until the number of iterations or the preset standard is reached, and output the trained AResNet50 model; input the image to be recognized into the trained AResNet50 model to output the first classification result, and extract the high-order features of the image to be recognized.

[0120] The second classification unit 320 is used to input the image to be identified into the phenotypic similarity model and combine it with higher-order features to obtain the phenotypic similarity of the crop, and to determine the second classification result based on the KNN algorithm and the phenotypic similarity of the crop.

[0121] The fusion unit 330 is used to fuse the first classification result and the second classification result using the late fusion method to obtain the final classification result.

[0122] In one embodiment, a computer device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the crop phenology fine-grained identification method as described above.

[0123] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the crop phenology fine-grained identification method described above.

[0124] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0125] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0126] The embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0127] The above are merely preferred embodiments of the present application and are not intended to limit the embodiments of the present application. For those skilled in the art, the embodiments of the present application can have various modifications and variations. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application should be included within the protection scope of the embodiments of the present application.

Claims

1. A method for fine-grained identification of crop phenology, characterized in that, Includes the following steps: The AResNet50 model is obtained by combining the dual attention mechanism, the global description operator and the ResNet50 model and training it until the number of iterations or the preset standard is reached, and the trained AResNet50 model is output. The image to be recognized is input into the trained AResNet50 model and the first classification result is output, and the high-order features of the image to be recognized are extracted. The image to be identified is input into the phenotypic similarity model and combined with higher-order features to obtain the phenotypic similarity of the crop. The second classification result is determined based on the KNN algorithm and the phenotypic similarity of the crop. The late fusion method is used to fuse the first and second classification results to obtain the final classification result; The step of inputting the image to be identified into a phenotypic similarity model and combining it with higher-order features to obtain the phenotypic similarity of crops specifically includes the following steps: The ORB algorithm is used to extract low-order features from the image to be identified, and these low-order features are concatenated with high-order features. Dimensionality reduction is then performed using PCA to obtain a fused feature vector X. Image similarity is then calculated. As shown in the following formula: Where X is the fused feature vector after PCA dimensionality reduction, and F concat For feature fusion operators, This represents the feature vector of the i-th image. Dot( represents the feature vector of the j-th image) ) represents a vector and dot product, and | Representing vectors respectively and norm, For image and The image similarity after normalization ranges from 0 to 1; The U-NET model with an attention mechanism is used to separate crops from the background in the image to be identified. Phenotypic features of the crops are extracted using relevant methods provided by Python. Normalized cosine similarity is used to calculate the phenotypic similarity between the image to be identified and the training set images. As shown in the following formula, In the formula, Let represent the phenotypic parameter vectors of the i-th and j-th images, respectively. The phenotypic similarity is the normalized phenotypic parameter similarity, with values ​​ranging from 0 to 1; The phenotypic similarity is obtained by fusing image similarity with phenotypic parameter similarity, as shown in the following formula: Among them, S i,j Let γ be the phenotypic similarity between image i and image j, γ be the image similarity weight, and β be the phenotypic parameter similarity weight. The phenotypic similarity value ranges from 0 to 1.

2. The method for fine-grained identification of crop phenology according to claim 1, characterized in that, The AResNet50 model uses the ReLU activation function and cross-entropy loss function in its fully connected and convolutional layers. The fully connected layers of the ResNet50 model are adjusted to BN layers and fully connected layers.

3. The method for fine-grained identification of crop phenology according to claim 2, characterized in that, The AResNet50 model is trained using a mini-batch stochastic gradient descent algorithm to optimize model parameters.

4. The method for fine-grained identification of crop phenology according to claim 1, characterized in that, The dual attention mechanism includes channel attention and spatial attention.

5. The method for fine-grained identification of crop phenology according to claim 1, characterized in that, The determination of the second classification result based on the KNN algorithm and crop phenotypic similarity specifically includes the following steps: The K training samples that are most similar to the image to be classified are determined based on the phenotypic similarity of the crops. The image to be classified is assigned to the category that appears most frequently in the K training samples to obtain the second classification result.

6. The method for fine-grained identification of crop phenology according to claim 1, characterized in that, The method of using latefusion to fuse the first classification result and the second classification result to obtain the final classification result includes the following steps: The random forest method is used to fuse the first and second classification results, and the output is a probability vector p = (p1, p2, ..., p...). k Perform soft voting fusion of multiple classifiers according to the following steps: Calculate the entropy of the i-th classifier, as shown in Formula 7: Accuracy of the comprehensive classifier i and entropy e i Calculate the entropy-weighted weights of the i-th classifier, as shown in Formula 8: Where α is the balance factor and β is the entropy weighting coefficient. Calculate the normalized weights of the i-th classifier, as shown in Formula 9: Where M is the number of classifiers, The fusion probability vector is obtained by weighted averaging the predicted probabilities of the M classifiers, as shown in Formula 10: The category with the highest fusion probability is selected as the final prediction result, as shown in Formula 11: Among them, y pred This represents the final category label, and k represents the number of categories.

7. A fine-grained identification device for crop phenology, characterized in that, include: The classification units are: the first classification unit, the second classification unit, and the fusion unit, among which... The first classification unit is used to combine the dual attention mechanism, the global descriptor operator and the ResNet50 model to obtain the AResNet50 model and train it until the number of iterations or a preset standard is reached, and output the trained AResNet50 model; input the image to be recognized into the trained AResNet50 model to output the first classification result, and extract the high-order features of the image to be recognized. The second classification unit is used to input the image to be identified into the phenotypic similarity model and combine it with higher-order features to obtain the phenotypic similarity of the crop, and to determine the second classification result based on the KNN algorithm and the phenotypic similarity of the crop. The fusion unit is used to fuse the first classification result and the second classification result using the late fusion method to obtain the final classification result; The step of inputting the image to be identified into a phenotypic similarity model and combining it with higher-order features to obtain the phenotypic similarity of crops specifically includes the following steps: The ORB algorithm is used to extract low-order features from the image to be identified, and these low-order features are concatenated with high-order features. Dimensionality reduction is then performed using PCA to obtain a fused feature vector X. Image similarity is then calculated. As shown in the following formula: Where X is the fused feature vector after PCA dimensionality reduction, and F concat For feature fusion operators, This represents the feature vector of the i-th image. Dot( represents the feature vector of the j-th image) ) represents a vector and dot product, and | Representing vectors respectively and norm, For image and The image similarity after normalization ranges from 0 to 1; The U-NET model with an attention mechanism is used to separate crops from the background in the image to be identified. Phenotypic features of the crops are extracted using relevant methods provided by Python. Normalized cosine similarity is used to calculate the phenotypic similarity between the image to be identified and the training set images. As shown in the following formula, In the formula, Let represent the phenotypic parameter vectors of the i-th and j-th images, respectively. The phenotypic similarity is the normalized phenotypic parameter similarity, with values ​​ranging from 0 to 1; The phenotypic similarity is obtained by fusing image similarity with phenotypic parameter similarity, as shown in the following formula: Among them, S i,j Let γ be the phenotypic similarity between image i and image j, γ be the image similarity weight, and β be the phenotypic parameter similarity weight. The phenotypic similarity value ranges from 0 to 1.

8. A computer device, characterized in that, Includes: memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images