Crop growth detection and evaluation method based on multi-angle image

Abstract

The invention provides a crop growth detection and evaluation method based on a multi-angle image, and belongs to the technical field of crop growth evaluation based on computer data processing. Firstly, crop images are collected; extracting multi-scale features based on a crop feature perception model, enhancing overlapped boundary recognition by using a residual coding block, optimizing segmentation precision through a self-attention mechanism and multi-channel deconvolution, and realizing crop detection and instance segmentation; further designing a cross-frame matching strategy, firstly roughly matching and tracking a plant by using a detection frame feature similarity, then fusing a shape gradient and a local texture feature fine-grained alignment instance, and integrating growth indexes frame by frame; and finally, a k-means clustering algorithm is utilized to obtain overall growth characteristics of the crops, detailed growth characteristics of the crops are obtained based on data analysis, and a crop growth evaluation result is obtained through calculation according to the two types of characteristics. According to the invention, an accurate monitoring system from data acquisition, feature perception to growth evaluation is formed, and evaluation of crop growth conditions is efficiently and accurately realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of crop growth assessment based on computer data processing, and particularly relates to a method for detecting and assessing crop growth based on multi-angle images. Background Art

[0002] Currently, agricultural planting is gradually developing towards large-scale, and agricultural production is also accelerating the realization of automation and intelligence. In this transformation process, the real-time and accurate determination of crop growth status has become the key technical support for the smooth implementation of smart agriculture. The traditional method of relying on manual assessment of crop growth status can, to a certain extent, ensure the assessment quality. However, in the face of large-scale and industrial modern agricultural production, the manual method faces severe challenges. With the expansion of the planting scale, the number of crops to be monitored has increased sharply, resulting in the fact that manual determination not only consumes a large amount of human resources but also incurs high time and economic costs, seriously restricting the promotion efficiency of agricultural intelligence. In recent years, with the significant progress of computer vision technology, the efficient assessment of large-scale crop growth can be achieved through computer vision technology, providing a reliable basis for determining the growth status of crops for large-scale and industrial intelligent planting to promote the further development of smart agriculture.

[0003] The existing crop growth detection methods mainly include the following three types: Practical judgment method based on expert experience: The detector directly observes the leaf color, leaf morphology, flowering situation, and pest and disease signs of the crop, and judges the growth status of the crop based on the accumulated experience and professional knowledge. This method has a certain degree of intuitiveness and flexibility. However, the subjectivity of this judgment method is relatively strong. Due to the differences in knowledge reserves and practical experience among different detectors, it is difficult to unify the judgment criteria, resulting in inaccurate and unstable detection results. In addition, the detection efficiency of this method is too low, and it is difficult to achieve rapid detection in the face of large areas of farmland; Image analysis method based on agricultural remote sensing images: The growth status of crops is analyzed using agricultural remote sensing images. First, remote sensing images of crop areas in different periods and different bands are obtained through remote sensing platforms such as satellites. Then, image preprocessing is carried out to correct the geometric distortion and radiation distortion of the images, remove noise interference, and improve the image quality. Subsequently, the type, health status, growth stage, etc. of the crops are judged by combining the characteristics such as the color, texture, and shape of the crops. Although the judgment efficiency of this method is relatively high, the cost of obtaining high-resolution remote sensing images is relatively high, thus limiting the application scenarios of this method; in addition, this method is greatly affected by the weather, and cloudy and foggy weather will reduce the image quality or even make it impossible to obtain images; and this method is prone to misjudgment in complex planting patterns and mixed vegetation areas, and it is difficult to ensure a high analysis accuracy rate; Model simulation method based on actual planting parameters: Based on the physiological and ecological processes of crop growth, a crop growth model is established. By inputting meteorological parameters, soil parameters, and crop variety parameters, the growth and development process of the crop is simulated, so as to effectively evaluate the growth and development of the crop. However, a large number of field tests are required to obtain and calibrate the crop growth model parameters required by the present invention, and the whole process is complicated, time-consuming, laborious and costly. On the other hand, the natural environment is complex and changeable, and it is difficult for the model to accurately simulate the sudden environmental changes, resulting in the inability to maintain a high evaluation accuracy under unstable environmental conditions.

[0004] Therefore, the existing methods all have different degrees of limitations in crop growth detection, and it is difficult to achieve efficient and accurate evaluation of the growth and development of large-area planted crops. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a method for crop growth detection and evaluation based on multi-angle images, which is characterized by including the following steps: S1, real-time collect Z-frame crop sequence images above the crop area; S2, input the collected images into the trained crop feature perception model to obtain crop detection results and instance segmentation results, and the instance image segmentation result is the feature instance map during crop growth; S3, based on the obtained crop detection results, use the crop coarse-grained matching method to match each crop in the video, and match and integrate to obtain a total of crops; at the same time, based on the obtained instance segmentation results, use the feature fine-grained matching method to match each instance of each crop; by performing crop coarse-grained matching and feature fine-grained matching of each crop frame by frame, obtain the complete instance of each crop; finally, count the growth characteristics of each crop, and finally obtain the growth characteristics of crops; S4, based on the growth characteristics of all crops in the land obtained, use the k-means clustering algorithm to obtain the overall growth characteristics of the crops, use the data analysis method to obtain the detailed growth characteristics of the crops, and calculate the crop growth evaluation results according to the above two types of characteristics.

[0006] Preferably, the instance segmentation results include five categories: leaves, flowers, fruits, disease spots, and withering.

[0007] Preferably, the crop feature perception model uses the first Q layers of the VGG19 network, as well as multiple convolutional layers and max pooling layers for multi-level feature extraction, and then uses the residual encoding block and self-attention layer to complete the adaptive fusion of features; and sends the obtained adaptive fusion features into the constructed crop detection head and feature instance segmentation network respectively.

[0008] Preferably, the crop feature perception model is specifically: First, use the first Q layers of the VGG19 network as the VGG19 feature extraction module to perform preliminary feature extraction on the crop image to obtain initial feature information data ; Then, use a convolutional layer with a convolution kernel size of 3*3 and a max pooling layer to perform multi-level feature extraction. Define the output feature of the first 3*3 convolutional layer in the model as the shallow feature , define the output feature of the second 3*3 convolutional layer as the middle feature , and define the output of the third 3*3 convolutional layer as the deep feature ; Subsequently, input the deep feature into the third residual encoding block for feature extraction processing to obtain a feature , and add the feature and the middle feature and input the sum into the second residual encoding block for fused feature extraction to obtain a feature . Subsequently, add the feature and the shallow feature and input the sum into the first residual encoding block for fused feature processing to obtain a feature ; Finally, input the feature into the self-attention layer for multi-level feature fusion processing to obtain a feature ; Send the feature to the crop detection head to obtain the object detection result of the crop ; Send the feature to the feature instance segmentation network, which includes three different paths, perform upsampling operations respectively, and combine with the residual decoding block for processing to obtain three groups of features. The three groups of features are sent to the channel attention layer for adaptive weighted processing, and the result after weighted processing is sent to the activation function for final processing to obtain the instance segmentation image .

[0009] Preferably, the first residual encoding block, the second residual encoding block, and the third residual encoding block are specifically: First, a convolutional layer with a convolution kernel size of 3*3 is used for feature perception. Subsequently, the BN normalization layer is used to regularize the features to enhance the generalization ability of the model and reduce the interference caused by instance overlap. After that, use The activation function activates the features and uses three 3×3 convolutional layers to extract features again. At this time, a residual connection is introduced, and the output of the first 3×3 convolutional layer in the residual encoding block is directly added to the output of the last 3×3 convolutional layer in the module, and the added features are re-activated through the Relu activation function.

[0010] Preferably, the feature instance segmentation network is specifically: In the first path, first use two deconvolution layers with a convolutional kernel size of 3×3 to perform upsampling processing on the features, and then perform feature dimension elevation operation through the first residual decoding block to obtain features ; in the second path, first use two deconvolution layers with a convolutional kernel size of 5×5 to perform upsampling processing on the features to obtain feature information , and then add the features and the features and send the sum into the second residual decoding block for processing to obtain features ; in the third path, use two deconvolution layers with a convolutional kernel size of 7×7 to perform upsampling processing on the features to obtain feature information , and then add the feature information and and send the sum into the third residual decoding block for processing to obtain features ; then send the features into the channel attention layer for adaptive weighting processing, and send the result of the weighting processing into the activation function to obtain the instance segmentation image .

[0011] Preferably, the specific process of S3 is: Coarse-grained crop matching: For the th frame and the th frame of the crop detection results and , where , respectively obtain the crop detection results in the total image rectangular blocks corresponding to the plants, and the crop detection results in the total image rectangular blocks corresponding to the plants ; ; After that, use the pre-trained Inceptionv3 model to extract features from all rectangular block images, and respectively obtain features corresponding to the plants , and Features corresponding to a plant ; Subsequently, a preset crop similarity threshold is set, and the cosine similarity between two sets of features is calculated pairwise. When the similarity between two features in different image frames exceeds , these two crops are matched as the same plant, thus completing the precise matching between the crops in the th frame of the image and the crops in the th frame of the image; In addition, for the crops that are not matched in the th frame of the image, they are regarded as newly detected crops, and corresponding numbers are assigned to all crops; Fine-grained feature matching: The shape structure features of each instance in the th frame and the th frame of the image are directly calculated using the histogram of oriented gradients , ; And the LBP of the joint pixel block of each instance and its adjacent instances is calculated to obtain the environmental features of each instance , ; After that, for the shape structure features of each instance in the th frame of the image , and the shape structure features of each instance in the th frame of the image , the shape similarity between the two sets of data is calculated pairwise based on the cosine similarity; Similarly, for the environmental features of each instance in the th frame of the image , and the environmental features of each instance in the th frame of the image , the environmental similarity between the two sets of data is calculated pairwise based on the cosine similarity; A preset shape similarity threshold and an environmental similarity threshold are set. When the shape similarity between two instances in different image frames exceeds , and the environmental similarity exceeds , it indicates that the two instances are the same matching instance in different video frames, thus completing the precise matching between the instances in the th frame of the image and the instances in the th frame of the image; In addition, for the instances that are not matched in the th frame of the image, they are regarded as newly detected instances, and corresponding numbers are assigned to all instances; Obtaining the growth characteristics of all crops: Based on the obtained crop detection results , each crop in the video is matched using the crop coarse-grained matching method, and a total of Individual crops; meanwhile, based on the obtained instance segmentation results , each instance of each crop is matched using a feature fine-grained matching method; by performing coarse-grained matching of crops frame by frame and fine-grained matching of features for each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are statistically analyzed to obtain the growth characteristics of the corresponding crop , and finally obtain the growth characteristics of individual crops .

[0012] Preferably, the specific process of using the k-means clustering algorithm to obtain the overall growth characteristics of crops in S4 is as follows: S41, based on the growth characteristics of the Total individual crops obtained , set the initial number of cluster centers to denotes the rounding operation, and randomly select from features as cluster center points, and define the obtained c cluster center points as ; S42, according to the obtained different cluster center points , calculate the Mahalanobis distance between each crop feature in the Total individual crops and each cluster center point in turn, and assign each crop feature to the cluster center point with the closest distance. After all Total individual crop features are assigned, form temporary clusters ; ; S43, based on the c temporary clusters obtained in the above S42 process, calculate the geometric mean of each crop feature in each temporary cluster in turn to obtain the crop mean features of the c temporary clusters, and use them as the new cluster center points ; S44, according to the new cluster center points obtained in the above S43 process, calculate the Euclidean distance between pairwise cluster center points, and set a judgment threshold . When the Euclidean distance between two cluster center points, merge these two clusters into one cluster; obtain the merged cluster center points ; S45, use the cluster center points as the initial cluster center points again, and repeat the iteration of the S42, S43, and S44 processes ​Next, after the iterative operation is completed, record the number of clusters remaining after the last iteration as .

[0013] Preferably, the method for obtaining the detailed growth characteristics of crops based on data analysis is specifically as follows: According to the growth characteristics of each crop Analyze the growth of the crop itself based on the number of leaves, fruits, flowers, disease spots, and the number of withered parts included; calculate the average number of leaves of Total plants , the average number of fruits , the average number of crop flowers , the average number of crop disease spots And finally the average number of withered parts of the crop .

[0014] Compared with the prior art, the present invention has the following beneficial effects: (1) A feature perception model for large-area crops is proposed: A multi-level feature extraction module is designed to obtain multi-level features of crop images, and the designed residual coding block is further combined to enhance the detection accuracy of the model for overlapping instances. Subsequently, a multi-level feature fusion mechanism, an object detection head, and an instance segmentation network are combined to quickly locate the crop position and obtain its instance segmentation map; in the crop feature perception model proposed by the present invention, the added residual coding block can reduce the influence caused by instance overlap, and combined with the multi-level feature fusion design, it ensures the accuracy of crop target extraction. In addition, the combination of the residual decoding block and the multi-path deconvolution layer ensures the processing accuracy of the instance segmentation image, and good detection accuracy can also be maintained in a complex farmland environment; (2) A crop matching and feature reconstruction strategy is constructed: Based on the object detection results and instance segmentation maps corresponding to different video frames predicted by the model, the matching of the same crop in different video frames is realized based on the cosine similarity, and the comprehensive reconstruction of crop feature instances is realized based on the histogram of oriented gradients and the local binary model feature histogram to obtain the comprehensive growth characteristics of all crops and eliminate the interference caused by object occlusion; the accurate matching of the same plant crop in different images is realized; (3)A crop growth situation assessment unit is designed: To effectively evaluate the growth and development of all crops in the land, the present invention obtains the growth differences of all crops in the land by designing an improved K-means clustering algorithm, combines data analysis to obtain the specific growth and development characteristics of the crops, and analyzes the overall development situation of the crops by combining the above two pieces of information. The constructed cluster center initialization strategy and dynamic merging mechanism can significantly enhance the ability to capture crop growth differences, intuitively reflect the growth uniformity of crops through the number of clusters, and also combine specific growth characteristic parameters of the crops to further improve the rationality of the evaluation. Using the above two evaluation methods can significantly improve the accuracy of growth situation evaluation and has good practicability. Brief Description of the Drawings

[0015] Figure 1 It is the overall technical route flowchart of the present invention.

[0016] Figure 2 It is the overall framework diagram of the crop feature perception model of the present invention.

[0017] Figure 3 It is the schematic diagram of the crop target detection and instance segmentation results.

[0018] Figure 4 It is the model performance comparison diagram in the embodiment. Detailed Embodiment

[0019] The present invention proposes a method for crop growth detection and evaluation based on multi-angle images. This method collects crop images by an unmanned aerial vehicle according to a zigzag trajectory and constructs a growth feature data set, combines rectangular box annotation and instance annotation tools to generate annotation information with crop positions and crop growth characteristics; extracts multi-scale features based on an improved deep model, uses a residual coding block to enhance the recognition of overlapping boundaries, optimizes the segmentation accuracy through a self-attention mechanism and multi-path deconvolution, and realizes crop detection and instance segmentation; further designs a cross-frame matching strategy, first roughly matches and tracks plants based on the similarity of detection box features, and then fuses shape gradients and local texture features for fine-grained alignment of instances, and integrates growth indicators such as leaves, flowers, and fruits frame by frame; finally, uses an improved clustering algorithm to dynamically initialize scattered cluster centers, combines Mahalanobis distance division and Euclidean distance merging to optimize the cluster group, evaluates the growth differences through the number of clusters, and at the same time statistically calculates the mean value of health parameters such as disease spots and withering, and calculates the farmland growth score by comprehensively considering the population distribution and individual status, forming a full-chain precise monitoring system from data collection, feature perception to growth evaluation, and providing intelligent decision-making support for agricultural management.

[0020] The following combines specific embodiments to specifically illustrate the specific implementation process of the present invention.

[0021] The overall process of this embodiment is as Figure 1 shown: S1, Construction of Crop Growth Feature Dataset: First, a drone is deployed above the crop planting land, and an image acquisition device is mounted on its belly. The drone flies in a zigzag flight trajectory to collect crop images in real time. Then, the collected images are respectively subjected to crop annotation and instance annotation. Specifically, Labelimg is used to perform rectangular box annotation on the crops in the images, including the position and size information of the crops in the images, so as to obtain crop annotation labels. LabelMe is used to perform instance annotation on the features such as leaves, flowering, fruits, disease spots, and withering of each crop in the crop images to obtain crop growth feature instance maps. Finally, the above steps of image acquisition, crop annotation, and instance annotation are repeated to obtain a total of L sets of crop growth feature instance annotation images, completing the construction of the crop growth feature dataset ; S2, Construction of Crop Feature Perception Model: In the constructed crop feature perception model, the first Q layers of the VGG19 network, as well as multiple convolutional layers and max-pooling layers, are used to implement multi-level feature extraction. Subsequently, the residual encoding block and the self-attention layer are used to complete the adaptive fusion of features. The obtained adaptively fused features are respectively sent into the constructed crop detection head and the feature instance segmentation network to obtain crop detection results and instance segmentation images, and the dataset constructed in S1 is used for model training to obtain the finally trained crop feature perception model; S3, Construction of Crop Matching and Feature Reconstruction Strategy: Data is collected in an actual crop planting field, and the collected images are sent into the crop feature perception model described in S2 for processing. Crop coarse-grained matching strategies and feature fine-grained matching strategies are constructed to re-match the crop detection results and instance segmentation results obtained from different video frame images to obtain comprehensive growth features of all crops in the land; S4, Construction of Crop Growth Situation Evaluation Unit: Based on the comprehensive growth features of all crops in the land obtained in S3, the k-means clustering algorithm is used to obtain the overall growth features of the crops, and the data analysis method is used to obtain the detailed growth features of the crops. Further calculations are performed based on the above two types of features to obtain the growth evaluation results of the crops.

[0022] I. Construction of Crop Growth Feature Dataset To achieve the automatic evaluation of the growth situation of crops in farmland, the present invention needs to ensure the automatic detection of all crops in the farmland and the automatic extraction of crop growth and development features. For this purpose, it is first necessary to construct a crop growth feature dataset, including the collection of crop growth images from different angles, as well as the calibration of crop position information and the instance annotation of crop growth features; Collection of crop growth images from different angles: In this embodiment, a drone is deployed above the crop planting land, and an image acquisition device is mounted on the belly of the drone to collect images containing crop growth information in real time When using a drone for image acquisition, the drone performs image acquisition along a zigzag flight trajectory to obtain images of crop growth information at different angles , avoiding the problem of crop overlap that exists when performing image acquisition at a single angle, and ensuring the robustness of crop growth information image acquisition; Initial crop annotation of the image: Since in the image , in addition to crops, there are also irrigation facilities, weeds, unused soil blocks, and other interfering objects. Accurately identifying the crops in the land is the basis for analyzing the growth conditions of the crops. Therefore, based on the acquired crop images , in this embodiment, the image annotation software Labelimg is used to perform rectangular box annotation on all the crops in the crop image , obtaining the position information and size information of the crops in the image, and finally obtaining the crop rectangular box label data ; Instance annotation of crop growth characteristics: Based on the acquired crop images, in addition to performing the above-mentioned crop annotation to obtain the position information of the crops, the present invention aims to predict and evaluate the growth and development conditions of the crops. For this purpose, in this embodiment, further instance annotation is performed on the growth characteristics of each crop. Since the number of leaves, the number of flowers, the number of fruits, the distribution information of disease spots, and the withering distribution of the crops are all useful information for measuring the growth and development of the crops. The present invention uses tools to carry out crop instance annotation. When performing instance annotation, there are a total of five categories including leaves, flowers, fruits, disease spots, and withering, and finally obtaining the crop growth characteristic instance annotation image ; Construction of crop growth characteristic dataset: Repeating the above process, a total of groups of images of crop growth information , as well as the corresponding crop rectangular box label data and crop growth characteristic instance annotation images are obtained. These data together constitute the crop growth characteristic dataset .

[0023] II. Regarding the construction of the feature perception model The present invention has developed a crop feature perception model to achieve automatic detection of crop plants and instance segmentation of crop features. This model is mainly composed of a residual encoding block, a residual decoding block, and an attention mechanism. Its overall architecture diagram is as shown in Figure 2 , and the specific process of building the model includes the following steps: S2-1, for the crop growth information image , first use the first Q layers of the network in VGG19 as the VGG19 feature extraction module of the present invention to perform preliminary feature extraction on the crop image to obtain initial feature information data 。In the present invention, since the model needs to be able to process crop images at different angles and there will be light interference during the actual use of the model; therefore, to ensure that the model can accurately identify crops and accurately perform instance segmentation under complex conditions, for the initial feature information data , further use a convolutional layer with a convolution kernel size of 3*3 and a max-pooling layer for multi-level feature extraction, and define the output feature of the first 3*3 convolutional layer in the model as shallow features , define the output feature of the second 3*3 convolutional layer as middle features , and define the output of the third 3*3 convolutional layer as deep features ; S2-2. When collecting images in actual crop planting land, there will be a situation where multiple instances of the same crop overlap, resulting in a blurred boundary between each instance, making it difficult to accurately segment each instance. To ensure that the network can accurately obtain the feature representation of the crop in various instance overlap situations, the present invention designs a residual encoding block to further adapt to the instance boundary blur problem caused by instance overlap. In the residual encoding block, first, a convolutional layer with a convolution kernel size of 3*3 is used for further feature perception, and then the BN normalization layer is used to further regularize the features to enhance the generalization ability of the model to further reduce the interference caused by instance overlap. After that, an activation function is used for feature activation, and three 3*3 convolutional layers are used to achieve further feature extraction. At this time, a residual connection is introduced, and the output of the first 3*3 convolutional layer in the residual encoding block is directly added to the output of the last 3*3 convolutional layer in the module, and the added features are further activated through the Relu activation function to complete the construction of the residual encoding block; S2-3. Based on the residual encoding block designed in S2-2 and the shallow features , middle features , deep features obtained in S2-1, further perform feature fusion processing to enhance the detection accuracy of crops in various scenarios. First, input the deep features into the third residual encoding block for feature extraction processing to obtain features , and add the features and the middle features and input them into the second residual encoding block for fused feature extraction to obtain features . Subsequently, add the features and the shallow features and input them into the first residual encoding block for fused feature processing to obtain features . Finally, the features Features are obtained through multi-level feature fusion processing in the input self-attention layer This multi-level feature fusion processing method can further enhance the robustness of the model and reduce the impact of rain and sunlight interference on crop feature perception; S2-4, based on the features obtained in S2-3 , input it into the crop detection head to obtain the object detection result , and input it into the feature instance segmentation network to obtain the instance segmentation image ; The specific process includes: (1) Object detection process After obtaining the features , input them into the crop detection head to obtain the object detection result of the crop . In the crop detection head, an average pooling layer is first used to further compress the features to meet the requirements of subsequent object detection. Then, further convolution processing is performed through two sequentially connected convolutional layers to further enhance the feature representation ability. After being processed by the BN normalization layer and the Relu activation function, the final object detection result of the crop is obtained ; (2) Instance segmentation process After obtaining the features , the features are fed into three different paths for upsampling operations. Specifically: In the first path, two transposed convolutional layers with a kernel size of 3*3 are first used to perform upsampling processing on the features , and then further feature dimension elevation operations are performed through the residual decoding block 1 to obtain the features ; In the second path, two transposed convolutional layers with a kernel size of 5*5 are first used to perform upsampling processing on the features to obtain the feature information . Then, the features and the features are added together and then fed into the residual decoding block 2 for processing to obtain the features ; In the third path, two transposed convolutional layers with a kernel size of 7*7 are used to perform upsampling processing on the features to obtain the feature information . Then, the feature information and are added together and then fed into the residual decoding block 3 for processing to obtain the features . Subsequently, the features are all fed into the channel attention layer for adaptive weighted processing. And the result of the weighted processing is fed into the activation function for final processing to obtain the finally predicted instance segmentation image ; Among them, the network structure of the residual decoding block is as follows: First, a 3×3 transposed convolutional layer is used to perform upsampling of the feature data; subsequently, a Dropout layer is used to further improve the robustness of the model when generating instance segmentation maps; then, it passes through 3 sequentially connected 3×3 transposed convolutional layers to further achieve upsampling of the features. To ensure that a clearly bounded instance segmentation map can still be generated when there are instance overlaps, in the designed residual decoding block, the output of the first 3×3 transposed convolutional layer is directly added to the output of the last 3×3 transposed convolutional layer in the module to introduce a residual connection, and the added features are further activated through the Silu activation function, thus completing the construction of the residual decoding block; S2-5. Using the crop growth feature dataset constructed in S1 , the above crop feature perception model is trained. During the training process, the Adam optimizer is used to optimize various network parameters in the model, and finally, the trained crop feature perception model is obtained.

[0024] III. Crop Matching and Feature Reconstruction Strategy To eliminate the crop occlusion problem existing in the single-angle image acquisition and obtain comprehensive crop growth feature information, the present invention constructs a crop matching and feature reconstruction strategy, aiming to match the same plant crops in different video frames and further reconstruct the crop growth features extracted in different video frames to obtain comprehensive crop growth feature data; S3-1. Image acquisition and corresponding plant detection and instance segmentation: A drone equipped with a camera flies uniformly above the crops, and a total of crop sequence images are evenly acquired at an interval of 0.5 s ; and they are input into the crop feature perception model trained in S2, and finally, the corresponding crop detection results , and instance segmentation results are obtained; then, the present invention performs coarse-grained crop matching and fine-grained feature matching on the crop detection and instance segmentation results frame by frame; S3-2. Coarse-grained crop matching: The coarse-grained matching is based on the crop detection results , aiming to accurately match the same plant crops in different frames of images before and after; specifically, for the crop detection results of the th frame and the and th frame images, where , according to the positions of the rectangular frames, a total of image rectangular blocks corresponding to the plants in the crop detection results , and the crop detection results the Communist Party of China image rectangle blocks corresponding to individual plants ; After that, use the pre-trained Inceptionv3 model to extract features from all the rectangle block images, and respectively obtain features corresponding to individual plants , and features corresponding to individual plants ; Subsequently, preset the crop similarity threshold , and calculate the cosine similarity between two groups of features pairwise. When the similarity between two features in different image frames exceeds , then match these two crops as the same plant, thus completing the precise matching between the crops in the rd frame image and the crops in the th frame image; In addition, for the crops that are not matched in the th frame image, regard them as newly detected crops, and assign corresponding numbers to all crops; S3-3, Feature fine-grained matching: Based on the crop coarse-grained matching process in S3-2, the same plants in the rd frame and the th frame images can be accurately matched; After that, on the basis of successful matching of the same plants, the present invention is based on the instance segmentation results of the rd frame and the th frame images and to perform matching and integration of crop growth characteristics; First of all, since each instance in the instance segmentation is an irregular pixel block, the pixel blocks corresponding to the instances included in ; the pixel blocks corresponding to the instances included in ; And, taking advantage of the fact that the histogram of oriented gradients (HOG) and the local binary pattern feature histogram (LBP) can directly reflect the feature distribution of pixel blocks, the present invention directly calculates the shape and structure features of each instance in the rd frame and the th frame images using HOG ; And calculate the LBP of the joint pixel blocks of each instance and its adjacent instances, so as to obtain the environmental features of each instance ; After that, for the shape and structure features of each instance in the th frame image , and the shape structure features of each instance in the th frame image , calculate the shape similarity between every two sets of data based on the cosine similarity; similarly, for the environmental features of each instance in the th frame image , and the environmental features of each instance in the th frame image , calculate the environmental similarity between every two sets of data based on the cosine similarity; Preset shape similarity threshold and environmental similarity threshold , when the shape similarity between two instances in different image frames exceeds , and the environmental similarity exceeds , it indicates that the two instances are the same matching instance in different video frames, thus completing the precise matching between the instance in the th frame image and the instance in the th frame image; in addition, for the instances that are not matched in the th frame image, they are regarded as newly detected instances, and corresponding numbers are assigned to all instances; S3-4, Frame-by-frame matching and integration of growth features: Based on the crop detection results obtained in S3-1 , use the crop coarse-grained matching method in S3-2 to match each crop in the video, and match and integrate to obtain a total of crops; at the same time, based on the instance segmentation results obtained in S3-1 , use the feature fine-grained matching method in S3-3 to match each instance of each crop; by performing crop coarse-grained matching and feature fine-grained matching of each crop frame by frame, the complete instances of each crop can be obtained; Finally, according to the complete instance results of the crops, count the number of leaves Lea, the number of flowers Flo, the number of fruits Fru, the number of disease spots Sic, and the number of withered parts Wit of each crop to obtain the growth characteristics of the corresponding crop , where represents the growth characteristics of the th crop, and ; S3-5, Obtaining the growth characteristics of all crops: Based on the frame-by-frame matching and integration method of growth features in S3-4, the growth characteristics of the crops can be obtained

[0025] IV. Evaluation of crop growth Based on the finally obtained growth characteristics of the crops in the farmland , evaluate the overall growth of the crops planted in this farmland; specifically, first obtain the overall growth characteristics of the crops based on K-means, and then obtain the detailed growth characteristics of the crops based on data analysis; finally, use the overall growth characteristics and detailed growth characteristics of the crops to evaluate the growth of the crops. The specific process includes: S4-1, Obtaining the overall growth characteristics of crops: When there are large differences in the growth status of crops in the farmland and the growth trend is unevenly distributed, it often indicates that there is a problem of uneven overall growth in this plot, manifested as vigorous growth of crops in some areas, while in other areas, there are phenomena such as slow growth, abnormal leaf color, or lagging development. Therefore, in order to evaluate the uneven growth of crops as a whole, the present invention designs an improved K-means clustering algorithm to analyze the growth characteristics of all crops in the field for automatic clustering, and reflects the uniformity of the overall crop growth through the number of cluster results; moreover, the more the number of clusters divided, the less unified the growth of the crops in this piece of land, and the fewer the number of clusters divided, the more unified the growth of the crops in this piece of land; the specific steps include: (1) Based on the growth characteristics of the Total crops obtained , set the initial number of cluster centers as indicating the rounding operation. And randomly select from features as the cluster center points, and define the obtained c cluster center points as ; (2) According to the obtained different cluster center points , calculate the Mahalanobis distance between each crop feature among the Total crops and each cluster center point in turn, and assign each crop feature to the nearest cluster center point. After all Total crop features are assigned, form a temporary cluster ; (3) Based on the c temporary clusters obtained in the above step (2), calculate the geometric mean of each crop feature in each temporary cluster in turn, obtain the crop mean features of the c temporary clusters, and use them as the new cluster center points ; (4) According to the new cluster center points obtained in the above step (3), calculate the Euclidean distance between every two cluster center points, and set a judgment threshold . When the Euclidean distance between two cluster center points is ​When that occurs, merge these two clusters into one cluster; obtain the merged cluster center points ; (5) Use the cluster center points as the initial cluster center points again, and repeat the iteration processes of (2), (3), and (4) times. After the iteration operation is completed, record the number of remaining clusters after the last iteration as ; S4-2, Crop detailed growth feature acquisition: In addition, the present invention further analyzes the growth condition of the crop itself according to the number of leaves, the number of fruits, the number of flowers, the number of disease spots, and the number of withered parts included in the growth characteristics of each crop. Specifically, calculate the average number of leaves of Total plants of crops , the average number of fruits , the average number of crop flowers , the average number of crop disease spots , and finally the average number of crop withered parts ; ; S4-3 Crop growth condition evaluation: Based on the number of clusters calculated in the S4-1 process, and the crop detailed growth characteristics (including , , , , ) obtained in the S4-2 process, the present invention conducts an overall evaluation of the crop growth condition; First, since reflects the overall crop growth uniformity, calculate the growth uniformity score : ; Among them, is the preset maximum reasonable number of clusters; since , so , and the larger it is, the more uniform the growth; Secondly, the crop detailed growth characteristics reflect the health of the crop itself (the more the number of leaves, fruits, and flowers, and the fewer the number of crop disease spots and crop withered parts, the healthier the crop), so calculate the growth uniformity score : ; And, the larger it is, the better the health condition of the crop and the more ideal the overall growth state; Finally, based on and Conduct an integrated evaluation, and let professionals select the key segmentation values for scoring, and obtain the final evaluation result through Table 1.

[0026] Table 1 Crop Growth Evaluation Table

[0027] V. Experimental Result Verification To verify the effectiveness of the crop feature perception module proposed in the present invention in crop object detection and crop feature instance segmentation. In this embodiment, a set of actual images containing multiple crops is selected as the input of this crop feature perception module, and the corresponding crop object detection results and instance segmentation result images are output. Observe the crop detection result image, and each crop in the original image is correctly identified. In addition, for the obtained instance segmentation image, the corresponding leaf features, lesion features, and withered parts in each crop are also accurately segmented. Due to the introduction of residual encoding blocks and residual decoding blocks in the module, this module also maintains excellent detection effects for smaller instance features. Therefore, the experiment shows that this crop feature perception module can successfully complete the accurate detection of crops and obtain instance segmentation images with clear boundaries. The experimental segmentation results are as Figure 3 shown.

[0028] To verify the improvement effect of the crop matching and feature reconstruction strategies proposed in the present invention on crop detection and crop feature instance segmentation, performance comparison experiments are carried out with the Mask-RT-DETR model and the YOLACT model in the same crop environment. In an actual crop field, use the model proposed in the present invention to process the crop field and record the number of detected crops, the number of leaves, the number of fruits, the number of flowers, the number of lesions, and the number of withered parts of the crops. Similarly, record the number of detected crops, the number of leaves, the number of fruits, the number of flowers, the number of lesions, and the number of withered parts of the crops detected by the Mask-RT-DETR model and the YOLACT model. Divide the above-mentioned detected feature numbers by the actual number of features in the farmland to obtain the detection rate.

[0029] As Figure 4 can be seen, the detection rates calculated by the model proposed in the present invention for six different features of crops, leaves, fruits, flowers, lesions, and withering are the highest among the three models, indicating that the method proposed in the present invention has the best performance among the three models.

[0030] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0031] Although the specific implementation manners of the present invention are described above, they are not limitations on the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A crop growth detection and evaluation method based on multi-angle images, characterized in that: The following steps are involved: S1, real-time acquisition of Z-frame crop sequence images above the crop area; S2, inputting the collected image into the trained crop feature perception model to obtain crop detection results and instance image segmentation results, wherein the instance image segmentation results are feature instance images of crops during growth; S3, based on the obtained crop detection results, use the crop coarse-grained matching method to match each crop in the video, and match and integrate to obtain the common crops; at the same time, based on the instance segmentation results, each instance of each crop is matched using the feature fine-grained matching method; by performing crop coarse-grained matching frame by frame and feature fine-grained matching of each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are counted, and finally the Growth characteristics of individual crops; S4, based on the growth characteristics of all crops in the land, the k-means clustering algorithm is used to obtain the overall growth characteristics of the crops, and the detailed growth characteristics of the crops are obtained based on the data analysis method, and the growth evaluation results of the crops are calculated based on the above two types of characteristics.

2. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, characterized in that: The instance segmentation results include five categories: leaves, flowers, fruits, lesions, and withered.

3. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, characterized in that: The crop feature perception model uses the first Q layers of the VGG19 network as well as multi-layer convolutional layers and maximum pooling layers to perform multi-level feature extraction, and then uses residual coding blocks and self-attention layers to complete adaptive fusion of features; and the adaptive fusion features are respectively sent to the constructed crop detection head and feature instance segmentation network.

4. The method for detecting and evaluating crop growth based on multi-angle images as claimed in claim 3, characterized in that: The crop feature perception model is specifically: First, the first Q layer network in VGG19 is used as the VGG19 feature extraction module to perform preliminary feature extraction on the crop image to obtain the initial feature information data. ; Then, a convolution layer with a convolution kernel size of 3*3 and a maximum pooling layer are used to extract multi-level features, and the output features of the first 3*3 convolution layer in the model are defined as shallow features , the output features of the second 3*3 convolutional layer are defined as the middle-level features , the output of the third 3*3 convolutional layer is defined as the deep feature ; Then the deep features Input the third residual coding block to extract features and obtain features , and the features and mid-level features After addition, the second residual coding block is input to extract the fusion features to obtain the features , and then the features and shallow features After addition, the first residual coding block is input to perform fusion feature processing to obtain the feature ; Finally, the feature Input the self-attention layer to perform multi-level feature fusion processing to obtain features ; The characteristics Send it to the crop detection head to obtain the target detection result of the crop ; The characteristics The feature instance segmentation network includes three different channels, which perform upsampling operations respectively and combine with the residual decoding block to obtain three sets of features. The three sets of features are sent to the channel attention layer for adaptive weighted processing, and the weighted results are sent to The final processing is performed in the activation function to obtain the instance segmentation image .

5. The method for detecting and evaluating crop growth based on multi-angle images as claimed in claim 4, characterized in that: The first residual coding block, the second residual coding block and the third residual coding block are specifically: First, a convolutional layer with a convolution kernel size of 3*3 is used for feature perception, and then the features are regularized using a BN normalization layer to enhance the generalization ability of the model and reduce the interference caused by instance overlap. The activation function is used to activate the features, and three 3*3 convolutional layers are used to extract the features again. At this time, the residual connection is introduced to directly add the output of the first 3*3 convolutional layer in the residual coding block to the output of the last 3*3 convolutional layer in the module, and the added features are reactivated through the Relu activation function.

6. The method for detecting and evaluating crop growth based on multi-angle images according to claim 4, characterized in that: The feature instance segmentation network is specifically: In the first pass, two deconvolution layers with a convolution kernel size of 3*3 are used to perform feature extraction. The upsampling process is implemented, and then the feature dimension is increased through the first residual decoding block to obtain the feature ; In the second path, two deconvolution layers with a convolution kernel size of 5*5 are first used to perform feature Upsampling is performed to obtain feature information , then the feature and Features After addition, it is sent to the second residual decoding block for processing to obtain the feature ; In the third path, two deconvolution layers with a convolution kernel size of 7*7 are used to perform feature Upsampling is performed to obtain feature information , and then the feature information and After addition, it is sent to the third residual decoding block for processing to obtain the feature ; Then the feature They are all sent to the channel attention layer for adaptive weighted processing, and the weighted processing results are sent to Instance segmentation image obtained in the activation function .

7. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, characterized in that: The specific process of S3 is as follows: Crop coarse-grained matching: Frame and Crop detection results of frame images and ,in , and obtain crop detection results according to the position of the rectangular frame Chinese Communist Party The image rectangle corresponding to each plant , and crop detection results Chinese Communist Party The image rectangle corresponding to each plant ; Afterwards, the pre-trained Inceptionv3 model is used to extract features from all rectangular block images and obtain Characteristics of each plant ,as well as Characteristics of each plant ; Then, the crop similarity threshold is preset , and calculate the cosine similarity between the two sets of features. When the similarity between two features in different image frames exceeds When , the two crops are matched as the same crop, thus completing the The crop in the frame image is Accurate matching between crops in frame images; in addition, for the Crops that are not matched in the frame image are considered as newly detected crops, and all crops are numbered accordingly; Feature fine-grained matching: Directed gradient histogram is used to directly calculate the first Frame and Shape structure features of each instance in the frame image , ; And calculate the LBP of the joint pixel block of each instance and the adjacent instances to obtain the environmental features of each instance , ; Afterwards, for the Shape structure features of each instance in the frame image , and Shape structure features of each instance in the frame image , based on the cosine similarity, the shape similarity between the two sets of data is calculated; similarly, for the Environmental features of each instance in the frame image , and Environmental features of each instance in the frame image , based on cosine similarity, calculate the environmental similarity between two sets of data; Preset shape similarity threshold Similarity threshold with environment , when the shape similarity of two instances in different image frames exceeds , and the environmental similarity exceeds , indicating that the two instances are the same matching instance in different video frames, thus completing the first The instance in the frame image is the same as the Accurate matching between instances in frame images; in addition, Instances that are not matched in the frame image are considered as newly detected instances, and all instances are numbered accordingly; Acquisition of growth characteristics of all crops: based on the obtained crop detection results , using the crop coarse-grained matching method to match each crop in the video, and matching and integrating to obtain the common crops; at the same time, based on the instance segmentation results , use the feature fine-grained matching method to match each instance of each crop; by performing crop coarse-grained matching frame by frame and feature fine-grained matching of each crop, a complete instance of each crop is obtained; finally, the growth characteristics of each crop are counted to obtain the growth characteristics of the corresponding crop , and finally get Growth characteristics of crops .

8. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, characterized in that: The specific process of using the k-means clustering algorithm to obtain the overall growth characteristics of crops in S4 is as follows: S41, based on the growth characteristics of the Total crops obtained , set the number of initialized cluster centers to ] Indicates rounding operation and randomly selects middle features as cluster centers, and the obtained c cluster centers are defined as ; S42, according to the obtained Different cluster centers , calculate successively Each crop characteristic distance The Mahalanobis distance of each cluster center point and each crop feature Assign to the nearest cluster center point. When all the total crop features are assigned, a temporary cluster is formed. ; S43, based on the c temporary clusters obtained in the above S42 process , calculate the geometric mean of each crop feature in each temporary cluster in turn, obtain the crop mean features of c temporary clusters, and use them as the new cluster center point ; S44, a new cluster center point is obtained according to the above S43 process , calculate the Euclidean distance between the center points of each cluster , and set the judgment threshold , when the Euclidean distance between the centers of two clusters , merge the two clusters into one cluster; get the merged Cluster Centers ; S45, Cluster Centers Re-initialize the cluster center point and repeat the iterative process S42, S43, S44 After the iteration is completed, the number of clusters remaining after the last iteration is recorded as .

9. The method for detecting and evaluating crop growth based on multi-angle images according to claim 1, characterized in that: The method based on data analysis to obtain detailed growth characteristics of crops is specifically as follows: According to the growth characteristics of each crop The number of leaves, fruits, flowers, spots, and withered parts contained in the analysis of the crop growth; the average number of leaves of the Total crop is calculated. , average number of fruits , average number of crop flowers , average number of crop lesions and the final average number of crop blights .

Images