A crop growth detection and evaluation method based on multi-angle images

Abstract

The present invention provides a method for crop growth detection and evaluation based on multi-angle images, which belongs to the technical field of crop growth evaluation based on computer data processing. First, crop images are collected; multi-scale features are extracted based on a crop feature perception model, and overlapping boundary recognition is enhanced using residual coding blocks. The segmentation accuracy is optimized through a self-attention mechanism and multi-channel deconvolution to achieve crop detection and instance segmentation; a cross-frame matching strategy is further designed, firstly tracking plants by coarse matching based on the similarity of detection frame features, and then fine-grained alignment of instances by fusing shape gradients and local texture features, integrating growth indicators frame by frame; finally, the overall growth characteristics of crops are obtained using a k-means clustering algorithm, and the detailed growth characteristics of crops are obtained based on data analysis, and the crop growth evaluation results are calculated based on the two types of features. The present invention forms a precise monitoring system from data collection, feature perception to growth evaluation, and realizes the evaluation of crop growth conditions efficiently and accurately.

Description

Technical Field

[0001] The present invention belongs to the technical field of crop growth assessment based on computer data processing, and in particular relates to a crop growth detection and assessment method based on multi-angle images. Background Art

[0002] Agricultural planting is currently progressing towards large-scale production, and agricultural production is rapidly becoming automated and intelligent. In this transformation, accurate, real-time assessment of crop growth status has become a key technical support for the successful implementation of smart agriculture. Traditional methods that rely on manual assessment of crop growth status, while ensuring quality to a certain extent, face significant challenges in the large-scale, industrialized world of modern agricultural production. As planting scale expands, the number of crops requiring monitoring increases dramatically. This results in manual assessments that consume significant human resources and come with high time and financial costs, severely hindering the advancement of intelligent agriculture. Recent advances in computer vision technology have made it possible to efficiently assess crop growth on a large scale, providing a reliable basis for assessing crop growth status in large-scale, industrialized smart planting, and thus promoting the further development of smart agriculture.

[0003] The existing crop growth detection methods mainly include the following three types:

[0004] Practical judgment method based on expert experience: Inspectors directly observe the leaf color, leaf shape, flowering status, and signs of pests and diseases of crops, and use their accumulated experience and professional knowledge to judge the growth status of crops. This method has a certain degree of intuitiveness and flexibility, but this judgment method is highly subjective. Due to differences in knowledge reserves and practical experience among different inspectors, it is difficult to unify the judgment standards, resulting in inaccurate and unstable test results. In addition, the detection efficiency of this method is too low, and it is difficult to achieve rapid detection when facing large areas of farmland;

[0005] Image analysis method based on agricultural remote sensing images: Using agricultural remote sensing images to analyze crop growth conditions, first obtain remote sensing images of crop areas at different periods and in different bands through satellite and other remote sensing platforms. Then perform image preprocessing to correct the geometric distortion and radiation distortion of the image, remove noise interference, and improve image quality. Then combine the color, texture, shape and other characteristics of the crop to determine the type, health, growth stage, etc. of the crop. Although this method has a high judgment efficiency, the cost of obtaining high-resolution remote sensing images is high, which limits the use scenarios of this method; in addition, this method is greatly affected by weather, and cloudy and foggy weather will reduce the image quality or even make it impossible to obtain images; and this method is prone to misjudgment for complex planting patterns and mixed vegetation areas, making it difficult to ensure a high analysis accuracy;

[0006] Model simulation method based on actual planting parameters: A crop growth model is established based on the physiological and ecological process of crop growth. By inputting meteorological parameters, soil parameters, and crop variety parameters, the growth and development process of crops is simulated, thereby effectively evaluating the growth and development of crops. However, the crop growth model parameters required by the present invention require a large number of field experiments to obtain and calibrate. The entire process is complicated, time-consuming, labor-intensive, and costly. On the other hand, the natural environment is complex and changeable, and it is difficult for the model to accurately simulate sudden environmental changes, resulting in an inability to maintain a high level of evaluation accuracy under unstable environmental conditions.

[0007] Therefore, existing methods have limitations to varying degrees in crop growth detection, making it difficult to achieve efficient and accurate growth and development assessment of crops planted over large areas. Summary of the Invention

[0008] To address the above problems, the present invention proposes a crop growth detection and assessment method based on multi-angle images, which is characterized by comprising the following steps:

[0009] S1, real-time acquisition of Z-frame crop sequence images above the crop area;

[0010] S2, inputting the collected image into the trained crop feature perception model to obtain crop detection results and instance segmentation results, wherein the instance image segmentation result is a feature instance map of the crop during growth;

[0011] S3, based on the obtained crop detection results, use the crop coarse-grained matching method to match each crop in the video, and match and integrate to obtain the common crops; at the same time, based on the instance segmentation results, each instance of each crop is matched using the feature fine-grained matching method; by performing crop coarse-grained matching frame by frame and feature fine-grained matching of each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are counted, and the final result is Growth characteristics of individual crops;

[0012] S4, based on the growth characteristics of all crops in the obtained land, uses the k-means clustering algorithm to obtain the overall growth characteristics of the crops, obtains the detailed growth characteristics of the crops based on the data analysis method, and calculates the crop growth evaluation results based on the above two types of characteristics.

[0013] Preferably, the instance segmentation results include five categories: leaves, flowers, fruits, lesions, and withered.

[0014] Preferably, the crop feature perception model uses the first Q layers of the VGG19 network as well as multi-layer convolutional layers and maximum pooling layers to perform multi-level feature extraction, and then uses residual coding blocks and self-attention layers to complete adaptive fusion of features; and the obtained adaptive fusion features are respectively sent to the constructed crop detection head and feature instance segmentation network.

[0015] Preferably, the crop feature perception model is specifically:

[0016] First, the front Q layer network in VGG19 is used as the VGG19 feature extraction module to perform preliminary feature extraction on the crop image to obtain the initial feature information data. ; Then, a convolution layer with a convolution kernel size of 3*3 and a maximum pooling layer are used to extract multi-level features, and the output features of the first 3*3 convolution layer in the model are defined as shallow features , the output features of the second 3*3 convolutional layer are defined as the middle layer features , the output of the third 3*3 convolutional layer is defined as the deep feature ;

[0017] Then the deep features Input the third residual coding block to perform feature extraction processing to obtain features , and the features and mid-level features After addition, the second residual coding block is input to extract the fusion features to obtain the features , and then the features and shallow features After addition, the first residual coding block is input to perform fusion feature processing to obtain the feature ; Finally, the feature Input the self-attention layer to perform multi-level feature fusion processing to obtain features ;

[0018] The features Send it to the crop detection head to obtain the target detection result of the crop ;

[0019] The features The feature instance segmentation network is fed into three different channels, each of which performs upsampling operations and is processed in combination with the residual decoding block to obtain three sets of features. The three sets of features are fed into the channel attention layer for adaptive weighted processing, and the weighted results are fed into The final processing is performed in the activation function to obtain the instance segmentation image .

[0020] Preferably, the first residual coding block, the second residual coding block and the third residual coding block are specifically:

[0021] First, a convolution layer with a convolution kernel size of 3*3 is used for feature perception, and then the features are regularized using the BN normalization layer to enhance the generalization ability of the model and reduce the interference caused by instance overlap. The activation function is used to activate the features, and three 3*3 convolutional layers are used to extract features again. At this time, the residual connection is introduced to directly add the output of the first 3*3 convolutional layer in the residual coding block and the output of the last 3*3 convolutional layer in the module, and the added features are reactivated through the ReLU activation function.

[0022] Preferably, the feature instance segmentation network is specifically:

[0023] In the first pass, two deconvolution layers with a convolution kernel size of 3*3 are used to perform feature Implement upsampling processing, then perform feature dimension increase operation through the first residual decoding block, and obtain the feature ; In the second path, two deconvolution layers with a convolution kernel size of 5*5 are first used to perform feature Upsampling is performed to obtain feature information , then the features and features After addition, it is sent to the second residual decoding block for processing to obtain the feature ; In the third path, two deconvolution layers with a convolution kernel size of 7*7 are used to perform feature Upsampling is performed to obtain feature information , and then the feature information and After addition, it is sent to the third residual decoding block for processing to obtain the feature ; Then the feature All of them are sent to the channel attention layer for adaptive weighted processing, and the weighted processing results are sent to Instance segmentation image obtained in the activation function .

[0024] Preferably, the specific process of S3 is:

[0025] Crop coarse-grained matching: for the Frame and Crop detection results of frame images and ,in , and get the crop detection results according to the position of the rectangular frame Chinese Communist Party The image rectangle corresponding to each plant , and crop testing results Chinese Communist Party The image rectangle corresponding to each plant ;

[0026] Afterwards, the pre-trained Inceptionv3 model is used to extract features from all rectangular block images and obtain Characteristics of each plant ,as well as Characteristics of each plant ; Then, preset the crop similarity threshold , and calculate the cosine similarity between the two sets of features. When the similarity between two features in different image frames exceeds When the two crops are matched as the same crop, the first The crop in the frame image is the same as the Accurate matching between crops in frame images; In addition, for the Crops that are not matched in the frame image are considered as newly detected crops, and all crops are numbered accordingly;

[0027] Fine-grained feature matching: Directed gradient histogram is used to directly calculate the first Frame and Shape structure features of each instance in the frame image , ; And calculate the LBP of the joint pixel block of each instance and the adjacent instances to obtain the environmental features of each instance , ;

[0028] Afterwards, for the Shape structure features of each instance in the frame image , and Shape structure features of each instance in the frame image , calculate the shape similarity between two sets of data based on cosine similarity; similarly, for the Environmental features of each instance in the frame image , and Environmental features of each instance in the frame image , calculate the environmental similarity between two sets of data based on cosine similarity;

[0029] Preset shape similarity threshold and environment similarity threshold , when the shape similarity of two instances in different image frames exceeds , and the environmental similarity exceeds , it indicates that the two instances are the same matching instance in different video frames, thus completing the first The instance in the frame image is the same as the Accurate matching between instances in the frame image; in addition, for the Instances that are not matched in the frame image are considered as newly detected instances and all instances are numbered accordingly;

[0030] Acquisition of growth characteristics of all crops: based on the obtained crop detection results , using the crop coarse-grained matching method to match each crop in the video, and matching integration to obtain the common crops; at the same time, based on the instance segmentation results , use the feature fine-grained matching method to match each instance of each crop; by performing crop coarse-grained matching frame by frame and feature fine-grained matching of each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are counted to obtain the growth characteristics of the corresponding crop , and finally get Growth characteristics of crops .

[0031] Preferably, the specific process of obtaining the overall growth characteristics of crops by using the k-means clustering algorithm in S4 is as follows:

[0032] S41, based on the obtained growth characteristics of Total crops , set the number of initialized cluster centers to ] Indicates rounding operation and randomly selects middle features as cluster centers, and the obtained c cluster centers are defined as ;

[0033] S42, according to the obtained Different cluster centers , calculate in sequence Each crop characteristic in distance The Mahalanobis distance of each cluster center point and the characteristics of each crop Assign to the nearest cluster center point. When all crop features are assigned, a temporary cluster is formed. ;

[0034] S43, based on the c temporary clusters obtained in the above S42 process , calculate the geometric mean of each crop feature in each temporary cluster in turn, get the crop mean feature of c temporary clusters, and use it as the new cluster center point ;

[0035] S44, a new cluster center point is obtained according to the above S43 process , calculate the Euclidean distance between the center points of each cluster , and set the judgment threshold , when the Euclidean distance between the two cluster centers is When , the two clusters are merged into one cluster; the merged Cluster centers ;

[0036] S45, will Cluster centers Re-initialize the cluster center point and repeat the iterative process S42, S43, S44 After the iteration is completed, the number of clusters remaining after the last iteration is recorded as .

[0037] Preferably, the data analysis-based method for obtaining detailed growth characteristics of crops is specifically as follows:

[0038] According to the growth characteristics of each crop The number of leaves, fruits, flowers, spots and withered parts contained in the total number of leaves are used to analyze the growth of the crop itself; the average number of leaves of the total crop is calculated respectively. , average number of fruits , average number of crop flowers , average number of crop lesions and finally the average number of crop blights .

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] (1) A feature perception model for large-scale crops is proposed: a multi-level feature extraction module is designed to obtain multi-level features of crop images, and the designed residual coding block is combined to further enhance the detection accuracy of the model for overlapping instances. Then, a multi-level feature fusion mechanism, a target detection head, and an instance segmentation network are combined to quickly locate the crop position and obtain its instance segmentation map. The crop feature perception model proposed in this invention has an added residual coding block that can reduce the impact of instance overlap. Combined with the multi-level feature fusion design, the accuracy of crop target extraction is guaranteed. In addition, the residual decoding block and the multi-channel deconvolution layer are combined to ensure the processing accuracy of the instance segmentation image, and good detection accuracy can be maintained even in complex farmland environments.

[0041] (2) A crop matching and feature reconstruction strategy was constructed: based on the target detection results and instance segmentation maps corresponding to different video frames obtained by model prediction, the same crop was matched in different video frames based on cosine similarity, and comprehensive crop feature instance reconstruction was achieved based on the directional gradient histogram and local binary model feature histogram, obtaining comprehensive growth characteristics of all crops and eliminating the interference caused by object occlusion; thus, accurate matching of the same crop in different images was achieved;

[0042] (3) A crop growth assessment unit was designed: To effectively assess the growth and development of all crops in the land, the present invention obtains the growth differences of all crops in the land by designing an improved K-means clustering algorithm, and combines data analysis to obtain the specific growth and development characteristics of the crops. Combining the above two information analyses, the overall development of the crops is obtained. The constructed cluster center initialization strategy and dynamic merging mechanism can significantly enhance the ability to capture crop growth differences. The number of clusters can intuitively reflect the uniformity of crop growth. It also combines the specific growth characteristic parameters of the crops to further improve the rationality of the assessment. The use of the above two assessment methods can significantly improve the accuracy of growth assessment and have good practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of the overall technical route of the present invention.

[0044] Figure 2 This is the overall framework diagram of the crop feature perception model of the present invention.

[0045] Figure 3 Schematic diagram of crop target detection and instance segmentation results.

[0046] Figure 4 2 is a comparison chart of model performance in the embodiments. DETAILED DESCRIPTION

[0047] This paper proposes a crop growth detection and assessment method based on multi-angle imagery. This method uses drones to capture crop images along zigzag trajectories and construct a growth feature dataset. Rectangular box annotation and instance annotation tools are then combined to generate annotations with crop locations and growth characteristics. An improved deep learning model extracts multi-scale features, utilizes residual coding blocks to enhance overlapping boundary recognition, and optimizes segmentation accuracy through a self-attention mechanism and multi-channel deconvolution, achieving crop detection and instance segmentation. Furthermore, a cross-frame matching strategy is designed. Plants are tracked using coarse matching based on the similarity of detection box features. Shape gradients and local texture features are then integrated to fine-grainedly align instances, integrating growth indicators such as leaves, flowers, and fruits frame by frame. Finally, an improved clustering algorithm is used to dynamically initialize the centers of scattered clusters. Clusters are then optimized by combining Mahalanobis distance partitioning with Euclidean distance merging. Growth differences are assessed by the number of clusters. Health parameters such as lesions and wilt are also calculated. Farmland growth scores are calculated by integrating population distribution and individual status. This creates a comprehensive, precise monitoring system from data collection and feature perception to growth assessment, providing intelligent decision support for agricultural management.

[0048] The specific implementation process of the present invention is described in detail below with reference to specific embodiments.

[0049] The overall process of this embodiment is as follows Figure 1 As shown:

[0050] S1, Construction of crop growth feature dataset: First, deploy a drone above the crop planting land and mount an image acquisition device on its belly. The drone collects crop images in real time according to a zigzag flight trajectory, and then performs crop labeling and instance labeling on the collected images; specifically, use Labelimg to label the crops in the image with rectangular frames, including the position and size information of the crops in the image, so as to obtain crop labeling labels; use LabelMe to perform instance labeling on the leaves, flowers, fruits, lesions, withering and other features of each crop in the crop image, and obtain a crop growth feature instance map; finally, repeat the above image acquisition, crop labeling and instance labeling steps to obtain L groups of crop growth feature instance labeling images, completing the crop growth feature dataset. The construction of

[0051] S2, Crop Feature Perception Model Construction: The constructed crop feature perception model uses the first Q layers of the VGG19 network, as well as multiple convolutional layers and max pooling layers to achieve multi-level feature extraction. Residual coding blocks and self-attention layers are then used to adaptively fuse features. The resulting adaptively fused features are fed into the constructed crop detection head and feature instance segmentation network, respectively, to obtain crop detection results and instance segmentation images. The model is then trained using the dataset constructed in S1, resulting in the final trained crop feature perception model.

[0052] S3, Crop matching and feature reconstruction strategy construction: Data collection is carried out in actual crop planting fields, and the collected images are sent to the crop feature perception model described in S2 for processing. A coarse-grained crop matching strategy and a fine-grained feature matching strategy are constructed to re-match the crop detection results and instance segmentation results obtained in different video frames to obtain comprehensive growth characteristics of all crops in the field;

[0053] S4, construction of crop growth assessment unit: Based on the comprehensive growth characteristics of all crops in the land obtained in S3, the k-means clustering algorithm is used to obtain the overall growth characteristics of the crops, and the detailed growth characteristics of the crops are obtained based on the data analysis method. The crop growth assessment results are further calculated based on the above two types of characteristics.

[0054] 1. Construction of Crop Growth Characteristics Dataset

[0055] To achieve automated assessment of crop growth in farmland, this method requires automated detection of all crops in the field and automated extraction of crop growth and development characteristics. To this end, a crop growth feature dataset must be constructed, including crop growth image acquisition from different angles, crop position calibration, and crop growth feature instance annotation.

[0056] Crop growth image acquisition from different angles: In this embodiment, a drone is deployed above the crop planting land, and an image acquisition device is mounted on the belly of the drone to collect images containing crop growth information in real time. When using drones for image acquisition, the drones follow a zigzag flight path to acquire images of crop growth information at different angles. , avoiding the crop overlap problem that exists when collecting images from a single angle, and ensuring the robustness of crop growth information image collection;

[0057] Initial image crop annotation: Since the image In addition to crops, there are also irrigation facilities, weeds, unused soil blocks and other interferences. Accurate identification of crops in the land is the basis for crop growth analysis. Therefore, based on the collected crop images In this embodiment, the image annotation software Labelimg is used to annotate crop images. All crops in the image are marked with rectangular frames to obtain the position and size information of the crops in the image, and finally the crop rectangular frame label data is obtained. ;

[0058] Crop growth characteristic example annotation: Based on the collected crop images, in addition to the crop annotation to obtain crop location information as described above, the present invention aims to predict and evaluate the growth and development of crops. To this end, this embodiment further annotates the growth characteristics of each crop. The number of leaves, flowers, fruits, lesion distribution, and withering distribution of crops are all useful information for measuring crop growth and development. The tool carries out crop instance annotation. When performing instance annotation, a total of five categories are included: leaves, flowers, fruits, disease spots, and withering. Finally, the crop growth feature instance annotation image is obtained. ;

[0059] Construction of crop growth characteristic dataset: Repeat the above process to obtain Group of images of crop growth information , and the corresponding crop rectangle label data and crop growth feature instance annotation images , these data together constitute the crop growth characteristics dataset .

[0060] 2. Feature Perception Model Construction

[0061] The present invention develops a crop feature perception model to achieve automatic detection of crop plants and crop feature instance segmentation. The model is mainly composed of residual coding block, residual decoding block and attention mechanism. The overall architecture of the model is shown in the figure below. Figure 2 As shown in Figure 2, the model building process specifically includes the following steps:

[0062] S2-1, for crop growth information images First, the front Q layer network in VGG19 is used as the VGG19 feature extraction module of the present invention to perform preliminary feature extraction on the crop image to obtain initial feature information data. In this invention, since the model needs to be able to process crop images at different angles, and there will be light interference when the model is actually used, in order to ensure that the model can accurately identify crops and accurately segment instances under complex conditions, the initial feature information data , further use the convolution layer with a convolution kernel size of 3*3 and the maximum pooling layer to perform multi-level feature extraction, and define the output feature of the first 3*3 convolution layer in the model as the shallow feature , the output features of the second 3*3 convolutional layer are defined as the middle layer features , the output of the third 3*3 convolutional layer is defined as the deep feature ;

[0063] S2-2, when collecting images in actual crop-growing land, multiple instances of the same crop may overlap with each other, resulting in blurred boundaries between instances, making it difficult to accurately segment each instance. In order to ensure that the network can accurately obtain the feature representation of crops in various instances of overlap, the present invention designs a residual coding block to further adapt to the instance boundary blurring problem caused by instance overlap. In the residual coding block, a convolution layer with a convolution kernel size of 3*3 is first used to further perform feature perception, and then the BN normalization layer is used to further regularize the features to enhance the generalization ability of the model to further reduce the interference caused by instance overlap, and then the residual coding block is used. The activation function performs feature activation, and three 3*3 convolutional layers are used to achieve further feature extraction. At this time, the residual connection is introduced to directly add the output of the first 3*3 convolutional layer in the residual coding block to the output of the last 3*3 convolutional layer in the module, and the added features are further activated through the ReLU activation function to complete the construction of the residual coding block;

[0064] S2-3, residual coding block designed based on S2-2 and shallow features obtained from S2-1 , mid-level features , deep features , further feature fusion processing is performed to enhance the accuracy of crop detection in various scenarios. First, the deep features Input the third residual coding block to perform feature extraction processing to obtain features , and the features and mid-level features After addition, the second residual coding block is input to extract the fusion features to obtain the features , and then the features and shallow features After addition, the first residual coding block is input to perform fusion feature processing to obtain the feature Finally, the feature Input the self-attention layer to perform multi-level feature fusion processing to obtain features This multi-level feature fusion processing method can further enhance the robustness of the model and reduce the impact of rain and sunlight interference on crop feature perception;

[0065] S2-4, features obtained based on S2-3 ,Will Input into the crop detection head to get the target detection result , and Feed it into the feature instance segmentation network to obtain instance segmentation image ; The specific process includes:

[0066] (1) Target detection process

[0067] Get features Then, it is sent to the crop detection head to obtain the target detection result of the crop. In the crop detection head, an average pooling layer is first used to further compress the features to meet the needs of subsequent target detection. Then, two convolutional layers are randomly connected in sequence to further enhance the feature representation ability. After the BN normalization layer and the ReLU activation function are processed, the final crop target detection result is obtained. ;

[0068] (2) Instance segmentation process

[0069] Get features Then, the features The image is sent to three different channels for upsampling. Specifically, in the first channel, two deconvolution layers with a convolution kernel size of 3*3 are used to perform upsampling on the features. Implement upsampling processing, and then further feature dimension increase operation is achieved through residual decoding block 1, and the feature ; In the second path, two deconvolution layers with a convolution kernel size of 5*5 are first used to perform feature Upsampling is performed to obtain feature information , then the features and features After addition, it is sent to the residual decoding block 2 for processing to obtain the feature ; In the third path, two deconvolution layers with a convolution kernel size of 7*7 are used to perform feature Upsampling is performed to obtain feature information , and then the feature information and After addition, it is sent to the residual decoding block 3 for processing to obtain the feature Then the feature All of them are sent to the channel attention layer for adaptive weighted processing. And the weighted processing results are sent to The final processing is performed in the activation function to obtain the final predicted instance segmentation image ;

[0070] Among them, the network structure of the residual decoding block is as follows:

[0071] First, a 3*3 deconvolution layer is used to increase the dimensionality of the feature data; then a dropout layer is used to further improve the robustness of the model in generating instance segmentation maps; and then three 3*3 deconvolution layers are connected in sequence to further increase the dimensionality of the features. To ensure that instance segmentation maps with clear boundaries can still be generated when instances overlap, the output of the first 3*3 deconvolution layer in the designed residual decoding block is directly added to the output of the last 3*3 deconvolution layer in the module to introduce a residual connection, and the added features are further activated through the Silu activation function, thus completing the construction of the residual decoding block.

[0072] S2-5, using the crop growth characteristic dataset constructed in S1 , the above-mentioned crop feature perception model is trained, and during the training process, the Adam optimizer is used to optimize the various network parameters in the model, and finally the trained crop feature perception model is obtained.

[0073] 3. Crop Matching and Feature Reconstruction Strategies

[0074] To eliminate the crop occlusion problem in single-angle image acquisition and obtain comprehensive crop growth characteristic information, this paper constructs a crop matching and feature reconstruction strategy. The goal is to match the same crop in different video frames and further reconstruct the crop growth characteristics extracted in different video frames to obtain comprehensive crop growth characteristic data.

[0075] S3-1, image acquisition and corresponding plant detection and instance segmentation: A drone equipped with a camera is used to fly at a constant speed above the crops and uniformly collects images at 0.5s intervals to obtain the total image quality. Crop sequence images ; and input it into the crop feature perception model trained by S2, and finally obtain the corresponding crop detection results , and instance segmentation results ; Afterwards, the present invention performs crop coarse-grained matching and feature fine-grained matching on the crop detection and instance segmentation results frame by frame;

[0076] S3-2, coarse-grained crop matching: Coarse-grained matching is based on crop detection results. , aims to accurately match the same crop in different frames of images; specifically, for the Frame and Crop detection results of frame images and ,in , and get the crop detection results according to the position of the rectangular frame Chinese Communist Party The image rectangle corresponding to each plant , and crop testing results Chinese Communist Party The image rectangle corresponding to each plant ;

[0077] Afterwards, the pre-trained Inceptionv3 model is used to extract features from all rectangular block images and obtain Characteristics of each plant ,as well as Characteristics of each plant ; Then, preset the crop similarity threshold , and calculate the cosine similarity between the two sets of features. When the similarity between two features in different image frames exceeds When the two crops are matched as the same crop, the first The crop in the frame image is the same as the Accurate matching between crops in frame images; In addition, for the Crops that are not matched in the frame image are considered as newly detected crops, and all crops are numbered accordingly;

[0078] S3-3, fine-grained feature matching: Based on the coarse-grained crop matching process in S3-2, the first Frame and The same crop in the frame image is accurately matched; then, based on the successful matching of the same crop, the present invention is based on the Frame and Instance segmentation results of frame images and Matching and integrating crop growth characteristics;

[0079] First, since each instance in instance segmentation is an irregular pixel block, Contained in The pixel blocks corresponding to the instances are: ; Contained in The pixel blocks corresponding to the instances are: Moreover, the advantage of using the Histogram of Directed Gradients (HOG) and the Histogram of Local Binary Patterns (LBP) is that they can directly reflect the distribution of pixel block features. The present invention uses HOG to directly calculate the first Frame and Shape structure features of each instance in the frame image , ; And calculate the LBP of the joint pixel block of each instance and the adjacent instances to obtain the environmental features of each instance , ;

[0080] Afterwards, for the Shape structure features of each instance in the frame image , and Shape structure features of each instance in the frame image , calculate the shape similarity between two sets of data based on cosine similarity; similarly, for the Environmental features of each instance in the frame image , and Environmental features of each instance in the frame image , calculate the environmental similarity between two sets of data based on cosine similarity;

[0081] Preset shape similarity threshold and environment similarity threshold , when the shape similarity of two instances in different image frames exceeds , and the environmental similarity exceeds , it indicates that the two instances are the same matching instance in different video frames, thus completing the first The instance in the frame image is the same as the Accurate matching between instances in the frame image; in addition, for the Instances that are not matched in the frame image are considered as newly detected instances and all instances are numbered accordingly;

[0082] S3-4, frame-by-frame matching and integration of growth features: based on the crop detection results obtained in S3-1 , using the S3-2 crop coarse-grained matching method to match each crop in the video, and matching integration to obtain the common crops; at the same time, the instance segmentation results obtained based on S3-1 , using the fine-grained feature matching method of S3-3 to match each instance of each crop; by performing coarse-grained crop matching frame by frame and fine-grained feature matching of each crop, a complete instance of each crop can be obtained;

[0083] Finally, based on the complete crop example results, the number of leaves Lea, the number of flowers Flo, the number of fruits Fru, the number of lesions Sic, and the number of withered areas Wit of each crop are counted to obtain the growth characteristics of the corresponding crop. ,in Indicates the The growth characteristics of the crops, and ;

[0084] S3-5, acquisition of growth characteristics of all crops: Based on the frame-by-frame matching and integration method of growth characteristics in S3-4, we can obtain Growth characteristics of crops and used it in the crop growth assessment process of S4.

[0085] IV. Assessment of Crop Growth

[0086] Based on the final farmland Growth characteristics of crops , to evaluate the overall growth of crops planted in this farmland; specifically, first, the overall growth characteristics of the crops are obtained based on K-means, and then the detailed growth characteristics of the crops are obtained based on data analysis; finally, the overall growth characteristics of the crops and the detailed growth characteristics of the crops are used to evaluate the crop growth. The specific process includes:

[0087] S4-1, acquisition of overall crop growth characteristics: When the growth status of crops in a farmland varies greatly and the growth distribution is uneven, it often indicates that the plot has an overall growth imbalance problem, which is manifested as vigorous growth of crops in some areas, while other areas have slow growth, abnormal leaf color or delayed development. Therefore, in order to evaluate the uneven growth of crops as a whole, the present invention designs an improved K-means clustering algorithm to obtain the growth characteristics of all crops in the field. Automatic clustering is performed, and the number of cluster results reflects the uniformity of overall crop growth. Moreover, the more clusters there are, the less uniform the growth of crops in this piece of land is, and the fewer clusters there are, the more uniform the growth of crops in this piece of land is. The specific steps include:

[0088] (1) Based on the obtained Total crop growth characteristics , set the number of initialized cluster centers to ] Indicates rounding operation. And randomly select middle features as cluster centers, and the obtained c cluster centers are defined as ;

[0089] (2) According to the obtained Different cluster centers , calculate in sequence Each crop characteristic in distance The Mahalanobis distance of each cluster center point and the characteristics of each crop Assign to the nearest cluster center point. When all crop features are assigned, a temporary cluster is formed. ;

[0090] (3) Based on the c temporary clusters obtained in the above (2) process , calculate the geometric mean of each crop feature in each temporary cluster in turn, get the crop mean feature of c temporary clusters, and use it as the new cluster center point ;

[0091] (4) The new cluster center point obtained according to the above (3) process , calculate the Euclidean distance between the center points of each cluster , and set the judgment threshold , when the Euclidean distance between the two cluster centers is When , the two clusters are merged into one cluster; the merged Cluster centers ;

[0092] (5) Cluster centers Re-initialize the cluster center and repeat the iterative process (2), (3), (4) After the iteration is completed, the number of clusters remaining after the last iteration is recorded as ;

[0093] S4-2, acquisition of crop growth characteristics: In addition, the present invention obtains the growth characteristics of each crop according to the The number of leaves, fruits, flowers, spots, and withered parts contained in the data are used to further analyze the growth of the crop itself. Specifically, the average number of leaves per plant is calculated. , average number of fruits , average number of crop flowers , average number of crop lesions and finally the average number of crop blights ;

[0094] S4-3 Crop Growth Assessment: Calculate the number of clusters based on the S4-1 process , and the detailed crop growth characteristics obtained in the S4-2 process (including 、 、 、 、 ), the present invention performs an overall crop growth assessment;

[0095] First, due to Reflects the overall uniformity of crop growth, so first calculate the growth uniformity score :

[0096] ;

[0097] in, is the preset maximum reasonable number of clusters; ,therefore ,and Larger means more uniform growth;

[0098] Secondly, the detailed growth characteristics of the crop reflect the health of the crop itself (the more leaves, fruits and flowers, and the fewer disease spots and withered areas, the healthier the crop). Therefore, the growth uniformity score is calculated. :

[0099] ;

[0100] and, The larger the number, the better the health of the crop and the more ideal its overall growth status;

[0101] Finally, based on and An integrated assessment is conducted, and professionals select key segment values for scoring, and the final assessment results are obtained through Table 1.

[0102] Table 1 Crop growth assessment table

[0103]

[0104] 5. Experimental Results Verification

[0105] In order to verify the effect of the crop feature perception module proposed in the present invention in crop target detection and crop feature instance segmentation. This embodiment selects a group of actual images containing multiple crops as the input of this crop feature perception module. And outputs the corresponding crop target detection results and instance segmentation result images. Observing the crop detection result image, each crop in the original image is correctly identified. In addition, for the obtained instance segmentation image, the corresponding leaf features, disease spot features and withered parts of each crop are also accurately segmented. Due to the introduction of residual coding blocks and residual decoding blocks in the module, this module also maintains excellent detection effects for smaller instance features. Therefore, experiments show that this crop feature perception module can successfully complete the accurate detection of crops and obtain instance segmentation images with clear boundaries. The segmentation results of this implementation experiment are shown in the figure. Figure 3 shown.

[0106] In order to verify the improvement effect of the crop matching and feature reconstruction strategy proposed in the present invention on crop detection and crop feature instance segmentation. A performance comparison experiment was carried out in the same crop environment with the Mask-RT-DETR model and the YOLACT model. In an actual crop planting field, the model proposed in the present invention was used to process the crop planting field and record the number of crop plants, leaves, fruits, flowers, lesions and withered parts of the crops detected. Similarly, the number of crop plants, leaves, fruits, flowers, lesions and withered parts of the crops detected by the Mask-RT-DETR model and the YOLACT model were recorded. The detection rate is obtained by dividing the number of features obtained by the above detection by the actual number of features existing in the farmland.

[0107] Depend on Figure 4 It can be seen that the detection rates calculated by the model proposed in the present invention on six different characteristics of crops, leaves, fruits, flowers, lesions and withering are the highest among the three models, indicating that the method proposed in the present invention has the best performance among the three models.

[0108] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

[0109] Although the above describes the specific implementation methods of the present invention, it does not limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for detecting and evaluating crop growth based on multi-angle images, characterized in that: The following steps are involved: S1, real-time acquisition of Z-frame crop sequence images above the crop area; S2, inputting the collected image into the trained crop feature perception model to obtain crop detection results and instance image segmentation results, wherein the instance image segmentation results are feature instance images of the crop during growth; The crop feature perception model uses the first Q layers of the VGG19 network, as well as multiple convolutional layers and maximum pooling layers, to perform multi-level feature extraction. It then uses residual coding blocks and self-attention layers to perform adaptive feature fusion. The resulting adaptive fused features are then fed into the constructed crop detection head and feature instance segmentation network, respectively. Specifically: First, the front Q layer network in VGG19 is used as the VGG19 feature extraction module to perform preliminary feature extraction on the crop image to obtain the initial feature information data. ; Then, a convolution layer with a convolution kernel size of 3*3 and a maximum pooling layer are used to extract multi-level features, and the output features of the first 3*3 convolution layer in the model are defined as shallow features , the output features of the second 3*3 convolutional layer are defined as the middle layer features , the output of the third 3*3 convolutional layer is defined as the deep feature ; Then the deep features Input the third residual coding block to perform feature extraction processing to obtain features , and the features and mid-level features After addition, the second residual coding block is input to extract the fusion features to obtain the features , and then the features and shallow features After addition, the first residual coding block is input to perform fusion feature processing to obtain the feature ; Finally, the feature Input the self-attention layer to perform multi-level feature fusion processing to obtain features ; The features Send it to the crop detection head to obtain the target detection result of the crop ; The features The feature instance segmentation network is fed into three different channels, each of which performs upsampling operations and is processed in combination with the residual decoding block to obtain three sets of features. The three sets of features are fed into the channel attention layer for adaptive weighted processing, and the weighted results are fed into The final processing is performed in the activation function to obtain the instance segmentation image ; S3, based on the obtained crop detection results, use the crop coarse-grained matching method to match each crop in the video, and match and integrate to obtain the common crops; at the same time, based on the instance segmentation results, each instance of each crop is matched using the feature fine-grained matching method; by performing crop coarse-grained matching frame by frame and feature fine-grained matching of each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are counted, and the final result is Growth characteristics of individual crops; S4, based on the growth characteristics of all crops in the obtained land, uses the k-means clustering algorithm to obtain the overall growth characteristics of the crops, obtains the detailed growth characteristics of the crops based on the data analysis method, and calculates the crop growth evaluation results based on the above two types of characteristics.

2. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, wherein: The instance segmentation results include five categories: leaves, flowers, fruits, lesions, and withered.

3. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, wherein: The first residual coding block, the second residual coding block and the third residual coding block are specifically: First, a convolution layer with a convolution kernel size of 3*3 is used for feature perception, and then the features are regularized using the BN normalization layer to enhance the generalization ability of the model and reduce the interference caused by instance overlap. The activation function is used to activate the features, and three 3*3 convolutional layers are used to extract features again. At this time, the residual connection is introduced to directly add the output of the first 3*3 convolutional layer in the residual coding block and the output of the last 3*3 convolutional layer in the module, and the added features are reactivated through the ReLU activation function.

4. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, wherein: The feature instance segmentation network is specifically: In the first pass, two deconvolution layers with a convolution kernel size of 3*3 are used to perform feature Implement upsampling processing, then perform feature dimension increase operation through the first residual decoding block, and obtain the feature ; In the second path, two deconvolution layers with a convolution kernel size of 5*5 are first used to perform feature Upsampling is performed to obtain feature information , then the features and features After addition, it is sent to the second residual decoding block for processing to obtain the feature ; In the third path, two deconvolution layers with a convolution kernel size of 7*7 are used to perform feature Upsampling is performed to obtain feature information , and then the feature information and After addition, it is sent to the third residual decoding block for processing to obtain the feature ; Then the feature All of them are sent to the channel attention layer for adaptive weighted processing, and the weighted processing results are sent to Instance segmentation image obtained in the activation function .

5. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, wherein: The specific process of S3 is as follows: Crop coarse-grained matching: for the Frame and Crop detection results of frame images and ,in , and get the crop detection results according to the position of the rectangular frame Chinese Communist Party The image rectangle corresponding to each plant , and crop testing results Chinese Communist Party The image rectangle corresponding to each plant ; Afterwards, the pre-trained Inceptionv3 model is used to extract features from all rectangular block images and obtain Characteristics of each plant ,as well as Characteristics of each plant ; Then, the crop similarity threshold is preset , and calculate the cosine similarity between the two sets of features. When the similarity between two features in different image frames exceeds When the two crops are matched as the same crop, the first The crop in the frame image is the same as the Accurate matching between crops in frame images; In addition, for the Crops that are not matched in the frame image are considered as newly detected crops, and all crops are numbered accordingly; Fine-grained feature matching: Directed gradient histogram is used to directly calculate the first Frame and Shape structure features of each instance in the frame image , ; And calculate the LBP of the joint pixel block of each instance and the adjacent instances to obtain the environmental features of each instance , ; Afterwards, for the Shape structure features of each instance in the frame image , and Shape structure features of each instance in the frame image , calculate the shape similarity between two sets of data based on cosine similarity; similarly, for the Environmental features of each instance in the frame image , and Environmental features of each instance in the frame image , calculate the environmental similarity between two sets of data based on cosine similarity; Preset shape similarity threshold and environment similarity threshold , when the shape similarity of two instances in different image frames exceeds , and the environmental similarity exceeds , it indicates that the two instances are the same matching instance in different video frames, thus completing the first The instance in the frame image is the same as the Accurate matching between instances in the frame image; in addition, for the Instances that are not matched in the frame image are considered as newly detected instances and all instances are numbered accordingly; Acquisition of growth characteristics of all crops: based on the obtained crop detection results , using the crop coarse-grained matching method to match each crop in the video, and matching integration to obtain the common crops; at the same time, based on the instance segmentation results , use the feature fine-grained matching method to match each instance of each crop; by performing crop coarse-grained matching frame by frame and feature fine-grained matching of each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are counted to obtain the growth characteristics of the corresponding crop , and finally get Growth characteristics of crops .

6. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, wherein: The specific process of using the k-means clustering algorithm to obtain the overall growth characteristics of crops in S4 is as follows: S41, based on the obtained growth characteristics of Total crops , set the number of initialized cluster centers to ] Indicates rounding operation and randomly selects middle features as cluster centers, and the obtained c cluster centers are defined as ; S42, according to the obtained Different cluster centers , calculate in sequence Each crop characteristic in distance The Mahalanobis distance of each cluster center point and the characteristics of each crop Assign to the nearest cluster center point. When all crop features are assigned, a temporary cluster is formed. ; S43, based on the c temporary clusters obtained in the above S42 process , calculate the geometric mean of each crop feature in each temporary cluster in turn, get the crop mean feature of c temporary clusters, and use it as the new cluster center point ; S44, a new cluster center point is obtained according to the above S43 process , calculate the Euclidean distance between the center points of each cluster , and set the judgment threshold , when the Euclidean distance between the two cluster centers is When , the two clusters are merged into one cluster; the merged Cluster centers ; S45, will Cluster centers Re-initialize the cluster center point and repeat the iterative process S42, S43, S44 After the iteration is completed, the number of clusters remaining after the last iteration is recorded as .

7. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, wherein: The method based on data analysis to obtain detailed growth characteristics of crops is specifically as follows: According to the growth characteristics of each crop The number of leaves, fruits, flowers, spots and withered parts contained in the total number of leaves are used to analyze the growth of the crop itself; the average number of leaves of the total crop is calculated respectively. , average number of fruits , average number of crop flowers , average number of crop lesions and finally the average number of crop blights .

Images