A method for automatically detecting cotton seedling growth and estimating emergence rate during the cotton seedling stage

Abstract

The present invention discloses a method for automatically detecting cotton seedling growth and estimating seedling emergence rate during the seedling stage, comprising the following steps: constructing a lightweight instance segmentation network; collecting cotton seedling information; using Labelme software to mark the cotton seedling area corresponding to the cotton seedling information to create a data set; using the data set to train the lightweight instance segmentation network to obtain pre-trained weights; selecting a sampling area in a cotton field and collecting video data using an unmanned aerial vehicle; writing a test script, loading the pre-trained weights, and detecting the video data of the sampling area; saving the video segmentation results frame by frame as images; looping through the images to obtain a cotton seedling outline matrix; and statistically analyzing the growth of the cotton seedlings based on the cotton seedling outline matrix; using the DeepSort algorithm to count the cotton seedlings in the target detection frame in the detection result; and modeling and estimating the cotton seedling emergence rate based on the cotton seedling counts. The present invention has high detection accuracy and strong practicality, providing a data basis for ensuring cotton field yield and cotton quality.

Description

Technical Field

[0001] The invention relates to the field of automatic detection of cotton seedling growth and emergence rate, and in particular to a method for automatic detection of cotton seedling growth and emergence rate estimation in the cotton seedling stage. Background Art

[0002] Xinjiang is a vast region with a sparse population and a vast area of ​​arable land. In recent years, inefficient manual labor and a shortage of agricultural workers have caused the labor cost of cotton planting to continue to rise. Therefore, cotton has gradually switched from manual picking to mechanical picking, and the planting model has also changed accordingly. Machine-picked cotton planting adopts a "double-row mode" with a row spacing of (66+10) cm, that is, 6 rows per film, 66 cm spacing outside the row, and 10 cm spacing inside the row; the plant spacing is 10 cm, and the sowing density is 666.7m 2 / 18,000 plants. Machine-picked cotton is sown using a precision, film-mulched mechanical sowing method, employing a one-hole, one-seed, spot-seeding method. Consequently, uneven emergence and short seedlings hinder normal growth, leading to reduced cotton yields. A prerequisite for refined cotton management is rapid and accurate monitoring of cotton seedling conditions. Currently, information on cotton seedling conditions in Xinjiang, including emergence rates and growth, is primarily collected through field visits, relying on farmers' subjective judgment based on their own experience. This method is not only inefficient, but also inaccurate, dependent on individual farmers' experience. This hinders timely and effective remedial measures and fails to meet current agricultural development requirements. Therefore, there is an urgent need to develop a method for rapidly monitoring cotton seedling growth and emergence rates per mu to ensure healthy seedling growth. This is of great practical significance for ensuring cotton yield and cotton quality, and promoting increased agricultural production and income.

[0003] The present invention mainly uses the moment features of the image and the Deepsort multi-target tracking algorithm. In recent years, due to the excellent properties of moments and moment invariants, moment features have been widely used in pattern recognition, image recognition and other aspects. There is almost no image feature that can be directly compared with the moment feature. The moment set extracted from the digital image describes the global characteristics of the image shape and provides a large amount of geometric feature information of different types of the image, such as size, position, direction, shape, etc. The first-order moment is related to the shape, and the second-order moment shows the degree of expansion of the curve around the average value of the straight line. The invariant moment is a statistical feature of the image, which satisfies the invariance of translation, scaling, and rotation and has a wide range of applications in the field of image recognition. The Deepsort algorithm applies deep appearance features to the problem of multi-target tracking. It introduces an appearance feature extraction network based on sort target tracking. The network is obtained by offline training on the pedestrian re-identification dataset. During the target tracking process, the appearance features of the target are extracted for nearest neighbor matching, which can effectively improve the target tracking effect under occlusion. Summary of the Invention

[0004] This invention aims to address the shortcomings of existing technologies for monitoring cotton seedling growth and emergence rates by proposing a method for automatically detecting cotton seedling growth and estimating emergence rates during the seedling stage. This method not only accurately locates the position and size of cotton seedlings, but also automatically monitors the emergence rate in the cotton field and the proportion of seedlings with poor growth. The detection is highly accurate, practical, safe, and reliable, providing data support for ensuring cotton field yield and cotton quality.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the seedling stage of cotton comprises the following steps:

[0007] S1: Build a lightweight instance segmentation network;

[0008] S2: Collect cotton seedling information;

[0009] S3: using Labelme software to mark the cotton seedling area corresponding to the cotton seedling information to create a data set;

[0010] S4: Using the data set to train the lightweight instance segmentation network to obtain pre-trained weights;

[0011] S5: Select sampling areas in the cotton field and collect video data using drones;

[0012] S6: Load the pre-trained weights and detect the video data of the sampling area;

[0013] S7: Segment the detected video and save the video segmentation results as images frame by frame;

[0014] S8: looping through the images in S7 to obtain a cotton seedling outline matrix, and calculating the growth of the cotton seedlings based on the cotton seedling outline matrix;

[0015] S9: Using the DeepSort algorithm, count the cotton seedlings in the target detection box in the detection result;

[0016] S10: Based on the cotton seedling counts, modeling is performed to estimate the cotton seedling emergence rate.

[0017] Preferably, in S1, the method for constructing the lightweight instance segmentation network includes:

[0018] S11: Construct lightweight convolution SCS Conv;

[0019] S12: Based on the lightweight convolution SCS Conv, construct a lightweight convolution block SCS Block;

[0020] S13: Using the lightweight convolution block SCS Block, ordinary two-dimensional convolution, convolution with CBMA and Maxpooling, construct a feature extraction network Backbone;

[0021] S14: Based on the lightweight convolution SCS Conv, construct a lightweight convolution block SCS bottleneck;

[0022] S15: Use the lightweight convolution block SCS bottleneck and upsampling layer to build Neck;

[0023] S16: Utilize the feature extraction network Backbone, the Neck, the target detection part, and the semantic segmentation part to construct a lightweight instance segmentation network.

[0024] Preferably, in S2, the method for collecting the cotton seedling information includes:

[0025] S21: Use a drone to build a shooting platform, with the camera lens pointing vertically downward, using hovering shooting mode to capture images of cotton at the 3-4 leaf stage;

[0026] S22: extracting a static image from the captured cotton image at an interval of 1 frame / s to complete the collection of the cotton seedling information.

[0027] Preferably, in S3, the method for preparing the data set includes:

[0028] S31: Use the image annotation tool Labelme software to select the edge of the cotton seedling area for annotation;

[0029] S32: The labeled images are randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1 to complete the preparation of the dataset.

[0030] Preferably, in S4, the training method of the lightweight instance segmentation network includes:

[0031] Using the deep learning development framework Pytorch, the lightweight instance segmentation network structure is constructed and the lightweight instance segmentation network model is trained;

[0032] Setting evaluation indicators of the lightweight instance segmentation network model, recall rate, precision rate, average accuracy rate, average intersection-over-union rate and F1 value score for evaluation;

[0033] The lightweight instance segmentation network model is trained with an initial learning rate of 0.0001, and is trained using stochastic gradient descent with a decay rate of 0.9 and a batch size of 2. The training is performed for a total of 150 epoch iterations.

[0034] After the training of the lightweight instance segmentation network model is completed, the loss, global correct, precision, recall, F1 value score, and average intersection-over-union ratio of each epoch are saved.

[0035] Preferably, in S8, the method of processing the image in S7 to obtain the cotton seedling outline matrix includes:

[0036] S81: Perform image preprocessing on the RoI maps obtained by network segmentation, first converting the RoI maps into grayscale images, and then converting the RoI maps into binary images;

[0037] S82: using the Otsu threshold method on the binary image obtained from S81 to obtain the outline of the cotton seedling;

[0038] S83: Save the outline of each cotton seedling as a matrix, and then save all matrices as an outline list;

[0039] S84: Use OpenCV to calculate the area of ​​each outline in the outline list, delete the noise points, and save the normal cotton seedling outline matrix.

[0040] Preferably, in S9, the method of counting cotton seedlings in the target detection frame in the detection result includes:

[0041] S91: Read the position of the cotton seedling detection frame of the current frame and the depth features of the image blocks of each detection frame;

[0042] S92: Filter the detection frame and depth features according to the confidence level;

[0043] S93: Perform non-maximum suppression on the detection frame to eliminate the situation where there are multiple frames on the target;

[0044] S94: Use Kalman filter to predict the position of cotton seedlings in the current frame;

[0045] S95: The predicted cotton seedling positions are matched with the trajectories, and the cotton seedling counting is completed.

[0046] Preferably, in said S10, the method of modeling and estimating the cotton seedling emergence rate includes:

[0047] S101: Select a plum blossom pattern for sampling based on the survey area and planting environment;

[0048] S102: Based on the sampling results, when investigating the emergence rate of precision seeding of machine-picked cotton, calculation is performed using holes as units. Each hole is considered to have one seedling emerging. The emergence rate is calculated by dividing the number of seedlings emerging by the number of seeding holes:

[0049]

[0050] Compared with the prior art, the beneficial effects of the present invention are as follows: the SCSconv convolution combination designed by the present invention not only solves the problem of low feature extraction and fusion capabilities of depth-separable convolution, but also greatly improves the speed of the detector. The moments of an image can describe the global characteristics of a shape, wherein the first-order central moment is used to describe the area of ​​the shape, and the second-order central moment represents the moment of inertia around the vertical and horizontal center lines. The present invention uses the segmentation results output by the neural network to calculate the moments of each cotton seedling outline. The center of gravity of each outline is obtained by the moment, and the Euclidean distance between adjacent centers of gravity is calculated to eliminate the problem of multiple outlines for one plant. The first-order central moment is then used to count the growth of the cotton seedlings. The Deepsort algorithm is one of the algorithms with better performance in multi-target tracking algorithms. The present invention uses the Deepsort algorithm to predict the target detection frame output by the neural network, realizes real-time statistics of cotton seedlings, and then establishes a relationship model between the sampling area and the number of cotton seedlings. The seedling emergence rate is calculated by the area of ​​the sampling area and the result output by the Deepsort algorithm. It provides a research basis and strategy for future cotton seedling growth detection and real-time statistics of seedling emergence rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 This is a flow chart of a method for automatically detecting cotton seedling growth and estimating emergence rate at the cotton seedling stage according to a first embodiment of the present invention;

[0053] Figure 2 This is a diagram of the SCSConv convolution structure in Example 1 of the present invention;

[0054] Figure 3 This is a structural diagram of an SCS block according to an embodiment of the present invention;

[0055] Figure 4 This is a diagram of the SCS bottleneck structure in Example 1 of the present invention;

[0056] Figure 5 This is a diagram of a segmentation network model of an example of an embodiment of the present invention;

[0057] Figure 6 This is a cotton field sampling area selection diagram according to Example 1 of the present invention;

[0058] Figure 7 This is a data preprocessing flow chart of Example 1 of the present invention;

[0059] Figure 8 This is a curve diagram of the change in model training loss in Example 1 of the present invention;

[0060] Figure 9 This is a graph showing the learning rate change curve for model training in Example 1 of the present invention;

[0061] Figure 10 This is a flowchart of the implementation of the DeepSort algorithm in Example 1 of the present invention;

[0062] Figure 11 This is a schematic diagram of calculating the lengths of cotton holes in accordance with the first embodiment of the present invention;

[0063] Figure 12 This is a flow chart for realizing the growth of cotton seedlings according to the first embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0065] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Example 1

[0067] like Figure 1 As shown, the present invention provides a method for automatically detecting the growth of cotton seedlings and estimating the emergence rate during the seedling stage, comprising the following steps:

[0068] S1: Build a lightweight instance segmentation network;

[0069] S2: Collect cotton seedling information;

[0070] S3: Use Labelme software to mark the cotton seedling areas corresponding to the cotton seedling information and create a data set;

[0071] S4: Use the dataset to train a lightweight instance segmentation network to obtain pre-trained weights;

[0072] S5: Select sampling areas in the cotton field and collect video data using drones;

[0073] S6: Write a test script, load the pre-trained weights, and detect the video data in the sampling area;

[0074] S7: Segment the detected video and save the video segmentation results as images frame by frame;

[0075] S8: looping through the images in S7 to obtain a cotton seedling outline matrix, and calculating the growth of the cotton seedlings based on the cotton seedling outline matrix;

[0076] S9: Using the DeepSort algorithm, count the cotton seedlings in the target detection box in the detection result;

[0077] S10: Based on the cotton seedling counts, model and estimate the cotton seedling emergence rate.

[0078] In this embodiment, in S1, the method for constructing a lightweight instance segmentation network includes:

[0079] S11: Construct lightweight convolution SCS Conv;

[0080] S12: Based on the lightweight convolution SCS Conv, build a lightweight convolution block SCS Block;

[0081] S13: Use lightweight convolution blocks (SCS blocks), ordinary two-dimensional convolutions, convolutions with CBMA, and maxpooling to build a feature extraction network backbone.

[0082] S14: Based on lightweight convolution SCS Conv, build lightweight convolution block SCS bottleneck;

[0083] S15: Use lightweight convolutional block SCS bottleneck and upsampling layer to build Neck;

[0084] S16: Use the feature extraction network Backbone, Neck, object detection part, and semantic segmentation part to build a lightweight instance segmentation network.

[0085] In this embodiment, if Figure 2 As shown, in S11, the construction method of the lightweight convolution SCS Conv includes:

[0086] S111: extracting the first feature of the target to be detected by performing ordinary convolution on the feature information;

[0087] S112: The first feature obtained by S111 is subjected to depthwise separable convolution in two steps. The first step, DepthwiseConvolution, first undergoes the first convolution operation, and DW is performed entirely in a two-dimensional plane. The number of convolution kernels is the same as the number of channels in the previous layer (channels and convolution kernels correspond one to one), so a three-channel image generates three feature maps after the operation; the second step, Pointwise Convolution, is very similar to the conventional convolution operation. The size of its convolution kernel is 1×1×M, where M is the number of channels in the previous layer. Therefore, the convolution operation here will perform a weighted combination of the map of the previous step in the depth direction to generate a new feature map and obtain the second feature;

[0088] S113: Assume that the input layer is divided into M groups and the total number of channels is M×n. First, split the channel dimension into two dimensions (M,n), then transpose these two dimensions into (n,M), and finally reshape it into a dimension M×n.

[0089] S114: Superimpose the first feature extracted by ordinary convolution with the second feature obtained by depthwise separable convolution, so that the first feature can penetrate into every part of the information extracted by depthwise separable convolution, and complete the construction of lightweight convolution SCSConv.

[0090] In this embodiment, if Figure 3 As shown, in S12, the method for constructing the lightweight convolution block SCS Block includes:

[0091] S121: Split the input channel C1 into two paths, and perform a two-dimensional convolution with a kernel size of 3x3 and a step size of 128 on each path to obtain two feature matrices.

[0092] S122: The two feature matrices obtained in S121 are simultaneously input into two SCS convs with a convolution kernel size of 3x3 and a step size of 128 for processing. This step not only increases the depth and width of the network but also reduces the parameters;

[0093] S123: The above features and the features in S11 are passed through a convolution layer with a convolution kernel size of 1x1. The purpose is to reduce the dimension (reduce the depth), reduce the model training parameters, reduce the amount of calculation, and complete the construction of the lightweight convolution block SCS Block.

[0094] In this embodiment, in S13, the method for constructing the feature extraction network Backbone includes:

[0095] S131: Apply a normal convolution with a kernel size of 3x3 and a stride of 64 to the image with an input size of (512, 512, 3), and obtain an output feature vector of (512, 512, 64);

[0096] S132: The feature vector obtained in S131 is passed through the SCS block and the convolution layer with CBMA to extract the features, and then passed through the Maxpooling layer with a convolution kernel size of 2x2 and a stride of 2 to adjust the height and width of the features;

[0097] S133: Repeat S131 and S132 three times to obtain three feature maps;

[0098] S134: The three feature maps are subjected to ordinary convolution with a convolution kernel size of 3x3 and a stride of 512 to obtain a feature vector with an output of (32, 32, 512), completing the construction of the feature extraction network Backbone.

[0099] In this embodiment, if Figure 4 As shown, in S14, the method for constructing the lightweight convolution block SCS bottleneck includes:

[0100] S141: The input information is subjected to a 2D convolution with a convolution kernel size of 3x3 and a step size of 128 to obtain a feature matrix;

[0101] S142: Input the feature matrix obtained in S141 into two SCSconvs with a convolution kernel size of 3x3 and a step size of 128 for processing to obtain a feature matrix;

[0102] S143: The above features are combined with the features in S11 through a convolution layer with a convolution kernel size of 1x1. On the one hand, the features are fused, and on the other hand, in order to reduce the dimension (reduce the depth), reduce the model training parameters, and reduce the amount of calculation, the lightweight convolution block SCS bottleneck is constructed.

[0103] In this embodiment, if Figure 5 As shown in Table 1, the deployment methods of the lightweight instance segmentation network include:

[0104] 1) This paper constructs a lightweight instance segmentation convolutional neural network model.

[0105] 2) Adjust the height and width of the input image to 512x512.

[0106] 3) Input the image obtained in step 2) into Backbone to obtain 3 output feature maps.

[0107] 4) The three output feature maps are input into the neck. In the neck part, SCSbottleneck and upsampling layers of different sizes are stacked to form three sizes of detection blocks. After passing through the neck, the information of the three feature maps is finally fused together and input into the target prediction branch and segmentation prediction branch respectively.

[0108] 5) In the target prediction branch, the prediction head divides the feature map into a 9x9 grid. For each grid i, (x+1) prediction scores are generated based on the number of detected targets x. The four adjacent grids are checked. If the class with the highest prediction score is the same as that of the surrounding grids, the largest connected domain is taken. Based on the size of the connected domain and with its center of mass as the center, a prediction box with a 1:1, 2:1, or 1:2 ratio is generated.

[0109] 6) By calculating the IoU values ​​between all predicted boxes and the true boxes, the NMS algorithm deletes all predicted boxes with IoU < 0.65 to obtain RoI maps.

[0110] 7) In the segmentation prediction branch, the feature map is first subjected to two-dimensional convolution to adjust the shape of the feature map.

[0111] 8) Upsample the result from step 7) and enlarge the feature map.

[0112] 9) After 1x1 convolution and Softmat layer, the segmented score maps are obtained.

[0113] 10) Add the two branches together to get the instance segmentation result.

[0114] Table 1

[0115]

[0116] In this embodiment, in S2, the method for collecting cotton seedling information includes:

[0117] S21: This image was taken in Shihezi, Xinjiang, in mid-May using a DJI drone set up as the shooting platform at an altitude of 80-100 cm. The camera resolution was set to 4608 pixels by 3456 pixels, with a focal length of 15 mm. The camera lens was pointed vertically downward in hovering mode, capturing images of cotton plants at the 3- to 4-leaf stage.

[0118] S22: 5,000 static images are extracted from the captured cotton images at an interval of 1 frame / s to complete the collection of cotton seedling information. In order to diversify the sample images in the training set, avoid image redundancy, minimize overfitting, and enhance the stability of the model, the static images are manually screened to establish a sample set of 1,000 images.

[0119] In this embodiment, if Figure 7 As shown, in order to expand the data set, the present invention uses methods such as horizontal or vertical flipping of the original input image and adjusting the image hue saturation to perform random data enhancement during the model training process to improve the robustness and accuracy of the model.

[0120] During each training iteration, the original image is subjected to the above steps in sequence. Each step has a 33% probability of triggering a change. HSV random transformations are restricted to a certain range: hue (H) transformations are limited to less than 20°, saturation (S) transformations are limited to less than 30%, and value (V) transformations are limited to less than 20%.

[0121] After the above data enhancement, the image size is adjusted and the final image size is: 512X512X3.

[0122] In this embodiment, in S3, the method for preparing the data set includes:

[0123] S31: Use the image annotation tool Labelme software to select the edge of the cotton seedling area for annotation;

[0124] S32: The labeled images are randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1 to complete the production of the dataset.

[0125] In this embodiment, the specific steps for selecting network evaluation indicators are as follows:

[0126] Step 9.1: Set the model's evaluation metrics, including recall, precision, mean accuracy (MAP), mean intersection over union (MIoU), and F1-score. The definitions of the above evaluation metrics are as follows:

[0127]

[0128]

[0129]

[0130] The calculation formulas for recall and precision are shown in Equations (1) and (2). In these formulas, TP represents a positive sample, while the actual sample is positive; FP represents a positive sample, while the actual sample is negative; FN represents a negative sample, while the actual sample is positive; and TN represents a negative sample, while the actual sample is negative. The calculation formula for average intersection-over-union is shown in Equation (3).

[0131]

[0132]

[0133] The calculation formula of F-Score is shown in formula (4), where β controls the importance of Precision and Recall. When β = 1, as shown in formula (5), the corresponding F1-score indicates that Precision and Recall are equally important. F1-score is an important indicator for evaluating network performance. The larger the F1-score value, the better the network effect.

[0134] In this embodiment, if Figure 8 、 Figure 9 As shown, in S4, the training method of the lightweight instance segmentation network includes:

[0135] In the Linux system environment, use the deep learning development framework Pytorch to build a lightweight instance segmentation network structure and train the lightweight instance segmentation network model;

[0136] Set the evaluation indicators of the lightweight instance segmentation network model, including recall rate, precision rate, average accuracy rate, average intersection-over-union rate and F1 value score for evaluation;

[0137] The initial learning rate of the lightweight instance segmentation network model is 0.0001. The stochastic gradient descent method is used for training, the decay rate is set to 0.9, the batch size is set to 2, and the training is carried out for 150 epoch iterations.

[0138] After the lightweight instance segmentation network model training is completed, the loss, global correct, precision, recall, F1 value score, and average intersection-over-union ratio of each epoch are saved.

[0139] In this embodiment, in S5, the method for selecting the sampling area includes:

[0140] The sampling was carried out in a five-point manner, which is consistent with the method used by local plant protection workers to investigate the emergence rate.

[0141] like Figure 6 As shown in the figure, the specific steps for selecting the cotton field data collection area are as follows:

[0142] S51: Compare and analyze the advantages and disadvantages of different sampling areas, including the reliability of data coverage after sampling, the difficulty of sampling, etc.

[0143] S52: Select the simplest and most commonly used 5-point sampling method.

[0144] S53: Check the planting characteristics of cotton fields in Xinjiang and set the size of the cotton field sampling area to 2.28m×2.28m.

[0145] S54: In order to make the data closer to the actual situation, a larger cotton field was selected.

[0146] S55: Place 5 sampling points at the northwest corner, northeast corner, southwest corner, southeast corner, and center of the cotton field.

[0147] The machine-picked cotton in Xinjiang adopts the typical (66+10) cm wide-narrow row planting pattern, that is, 6 rows per film, with inner and outer row spacing of 10 cm and 66 cm respectively, and a total width of 2.28 m. Therefore, the sample size for the local survey was set as 2.28 m x 2.28 m.

[0148] In this embodiment, in S8, the method of processing the image in S7 to obtain the cotton seedling outline matrix includes:

[0149] S81: Perform image preprocessing on the RoI maps obtained by network segmentation, first converting the RoI maps into grayscale images, and then converting the RoI maps into binary images;

[0150] S82: Using the Otsu threshold method on the binary image obtained from S81, the outline of the cotton seedling is obtained;

[0151] S83: Save the outline of each cotton seedling as a matrix, and then save all matrices as an outline list;

[0152] S84: Use OpenCV to calculate the area of ​​each outline in the outline list, delete the noise points, and save the normal cotton seedling outline matrix.

[0153] In this embodiment, the specific calculation steps of the contour matrix of the cotton seedling leaf surface are as follows:

[0154] 1) Perform image preprocessing on the output of the lightweight instance segmentation network.

[0155] 2) Use the Otsu threshold method to detect connected contours in binary images.

[0156] 3) Calculate the area of ​​each connected contour. If the area is smaller than the threshold, it is a noise point. The corresponding contour is deleted and the remaining contours are saved.

[0157] 4) To prevent a cotton seedling from being divided into multiple plants, first calculate the centroid coordinates of each contour and save them.

[0158] 5) Loop through the coordinates of the centers of gravity in pairs and calculate the Euclidean distance between the two points:

[0159]

[0160] Where x1 and y1 are the coordinate values ​​of the center of gravity of contour 1; x1 and y1 are the coordinate values ​​of the center of gravity of contour 2;

[0161] 6) If the calculation result d i Less than 10, indicating that the d i The two corresponding contours are the same cotton seedling, and the two matrix numbers corresponding to the calculated center of gravity are saved.

[0162] 7) Calculate the area of ​​each saved contour matrix, that is, calculate the 0th order moment of the preprocessed image, and save the calculation results.

[0163] 8) According to the saved matrix serial number of the same plant with different areas, take out the corresponding contour matrix and calculate its area and center of gravity.

[0164] 9) Cover the area and center of gravity of the position corresponding to one of the serial numbers with the calculation results, and delete the contour matrix, area, and center of gravity of the other serial number.

[0165] 10) Delete the saved contour matrix.

[0166] 11) Calculate the mean and standard error of the mean for the processed area information.

[0167] 12) Count the number of cotton seedlings with a value less than the mean value - the standard error of the mean in the treated area and calculate the percentage of cotton seedlings with poor growth.

[0168] See Figure 11 、 Figure 12 As shown, the specific steps of monitoring the growth of cotton seedlings in the steps of the present invention are as follows:

[0169] 1) Perform image preprocessing on the output of the instance segmentation model: first convert the image into a grayscale image, and then convert the grayscale image into a binary image.

[0170] 2) Use the Otsu threshold method to detect connected contours in binary images.

[0171] 3) Opencv detects connected contours and deletes noise points with contour areas less than 10 pixels. The rest are the contours of the cotton seedlings, and the contour information is stored in the outline list in a matrix format.

[0172] 4) Loop out the matrix information in the outline.

[0173] 5) To prevent a single cotton seedling from being split into multiple plants, first calculate the centroid of each contour:

[0174]

[0175] Where u 0,0is the zero-order moment of each contour; u 1,0 、u 1,0 is the first-order moment of each contour. The calculation formula of the above moments is as follows:

[0176]

[0177]

[0178]

[0179] Where i is the horizontal coordinate value of the cotton seedling outline in the two-dimensional coordinates in the binary image; y is the vertical coordinate value of the cotton seedling outline in the two-dimensional coordinates in the binary image.

[0180] 6) The serial number, centroid coordinates, and contour matrix of each contour are saved in the centroid list.

[0181] 7) Arrange the centroid values ​​in ascending order according to the vertical coordinate value.

[0182] 8) Loop through each centroid to find its centroid coordinates and calculate the Euclidean distance between the two points:

[0183]

[0184] Where x1 and y1 are the coordinate values ​​of the center of gravity of contour 1; x1 and y1 are the coordinate values ​​of the center of gravity of contour 2.

[0185] 9) If the calculation result d i Less than 10, indicating that the d i The two corresponding contours are the same cotton seedling. The serial numbers corresponding to the above contours are saved in the form of tuples (m, n) to the copy list.

[0186] 10) Calculate the area of ​​each contour matrix in the list, that is, calculate the 0th-order moment of the preprocessed image.

[0187] 11) Loop and save the calculation results to centroid.

[0188] 12) Based on the tuples in the copy list, extract the cotton seedlings of the same plant with different areas from the centroid and calculate their areas and centers of gravity:

[0189] Area=(centroid[m,4]+centroid[n,4]) / 2 (12)

[0190] X=(centroid[m,1]+centroid[n,1]) / 2

[0191] Y=(centroid[m,2]+centroid[n,2]) / 2

[0192] centroid is a 5-column list. The first column is the contour number; the second column is the horizontal coordinate of the contour centroid; the third column is the vertical coordinate of the contour centroid; the fourth column is the contour matrix; and the fifth column is the contour area. So the area of ​​the mth contour is: centroid[m,4], and so on.

[0193] Delete n rows of data and overwrite the corresponding values ​​in m with Area, X, and Y in step 9

[0194] 14) Delete the 4th column of data in centroid

[0195] 15) For the processed centroid data in column 4, which is the area information, calculate the mean and standard error of the mean:

[0196] mean=1 / x·(centroid[0,4]+centroid[1,4]+...+centroid[x-1,4]) (13)

[0197]

[0198]

[0199] Where x is the number of cotton seedling outlines, δ is the standard deviation of the cotton seedling outlines, and SE is the standard error of the mean.

[0200] 16) Count the number of data in the fourth column of centroid that are less than mean-SE and calculate the percentage of cotton seedlings with poor growth.

[0201] In this embodiment, in S9, the method of counting cotton seedlings in the target detection frame in the detection result includes:

[0202] S91: Read the position of the cotton seedling detection frame of the current frame and the depth features of the image blocks of each detection frame;

[0203] S92: Filter the detection frame and depth features according to the confidence level;

[0204] S93: Perform non-maximum suppression on the detection frame to eliminate the situation where there are multiple frames on the target;

[0205] S94: Use Kalman filter to predict the position of cotton seedlings in the current frame;

[0206] S95: The predicted cotton seedling positions are matched with the trajectories, and the cotton seedling counting is completed.

[0207] In this embodiment, in S10, the method for modeling and estimating the cotton seedling emergence rate includes:

[0208] S101: Select a plum blossom pattern for sampling based on the survey area and planting environment;

[0209] S102: Based on the sampling results, when investigating the emergence rate of precision seeding of machine-picked cotton, the calculation is performed using holes as the unit. The emergence rate is calculated by dividing the number of emerged seedlings by the number of seeding holes:

[0210]

[0211] See Figure 10 As shown in Table 2, the specific steps of DeepSort in the steps of the present invention to achieve cotton seedling counting are as follows:

[0212] The DeepSort algorithm uses a wide residual network (WRNS) model with two convolutional layers followed by six residual blocks. A global feature map of dimension 128 is computed in the "Dense 10" layer. Finally, features are projected onto a unit hypersphere using batch normalization and l2 normalization to be compatible with the cosine appearance metric. As a result, the DeepSort algorithm not only has a small number of parameters but also a simple structure, making it easier to implement and control on hardware.

[0213] Table 2

[0214]

[0215]

[0216] (2) Using Kalman filtering to estimate the status of cotton seedlings. The specific steps are as follows:

[0217] 1) Establish a Kalman state estimation model and use an 8-dimensional space to represent the state of the target at a certain moment:

[0218]

[0219] Where (u, v) represents the center coordinates of the bounding box, r is the aspect ratio, and h is the height. The remaining four variables represent the corresponding velocity information in the image coordinate system.

[0220] 2) Kalman filtering is divided into two processes: prediction and update. The state prediction process is as follows:

[0221] X(k|k-1)=ΦX(k-1|k-1)+BU(k) (18)

[0222] Where, represents the system state at time k, U(k) is the control quantity of the system at time k, Φ is the state transition matrix, and B is the system parameter. X(k|k-1) is the target state information of the previous frame, which is expressed in 8-dimensional space as: [x, y, ration, height, 0, 0, 0, 0]

[0223] 3) The covariance prediction in the recursive process of Kalman filtering is as follows:

[0224] P(k|k-1)=ΦP(k-1|k-1)Φ T +Q (19)

[0225] Where P(k-1|k-1) is the covariance matrix corresponding to the previous state X(k-1|k-1), P(k|k-1) is the covariance matrix corresponding to the current state X(k|k-1), and Q is the covariance matrix of the system noise. This step updates the covariance matrix of the current state using the previous state.

[0226] (2) The steps for matching cotton seedling trajectories in the Deepsort algorithm are as follows:

[0227] 1) Use the Mahalanobis distance between the detection box and the tracker prediction box to describe the association of motion information:

[0228] d (1) (i,j)=(d j -y i ) T S i -1 (d j -y i ) (20)

[0229] Among them, d j Indicates the position of the jth detection box, (y i ,S i ) represents the projection of the i-th trajectory distribution to the measurement space.

[0230] 2) If the association d (1) The Mahalanobis distance is less than the specified threshold t (1) , then the association of setting the motion state is successful:

[0231] b i,j (1) =1[d (1) (i,j)≤t (1) ] (twenty one)

[0232] 3) When the detection is under occlusion, the Mahalanobis distance has a large accuracy error, so the minimum cosine distance is used:

[0233]

[0234] 4) In the minimum cosine distance, a binary variable is used to measure whether there is association:

[0235] b i,j (2) =1[d (2) (i,j)≤t (2) ] (twenty three)

[0236] 5) In the final association process, the associations in the above two cases are weighted:

[0237] C i，j =λd (1) (i, j)+(1-λ)d (2) (i, j) (24)

[0238] The hyperparameter λ controls the impact of each metric on the combined associated cost.

[0239] (3) The specific steps for modeling and estimating the emergence rate are as follows:

[0240] 1) In order to quickly estimate the emergence rate of large-scale cotton fields in Xinjiang, the present invention takes 5 sampling areas as the research objects, each sampling area has a side length of 2.28m, and according to the planting characteristics of cotton fields in Xinjiang: one film has 6 rows, the spacing outside the row is 66cm, the spacing inside the row is 10cm; the plant spacing is 10cm, and the locomotive walking row spacing is 76cm.

[0241] 2) From 1), we can get that the number of cotton holes in N sampling areas with side length L is:

[0242]

[0243] Where L is the side length of the sampling area, which is usually measured in meters. To keep the units consistent with the spacing within rows and between plants, L·100 is expressed in centimeters.

[0244] 3) According to the above formula, the total number of cotton holes in the sampling area is: 660

[0245] 4) Seedling emergence rate of sampled cotton fields:

[0246]

[0247] Where n is the number of cotton plants in the five sampling areas detected by the Deepsort algorithm, and Y is the number of cotton seedlings in the cotton field calculated in the above steps.

[0248] This paper designs a lightweight instance segmentation neural network for cotton seedling segmentation. The network utilizes the SCSconv convolutional combination designed in this paper. The convolutional blocks constructed in this paper not only address the shortcomings of depthwise separable convolutional methods in terms of poor feature extraction and fusion capabilities, but also significantly improve detector speed. Image moments can describe global shape properties, with first-order central moments describing the area of ​​the shape and second-order central moments representing the moments of inertia around the vertical and horizontal centerlines. This paper uses the segmentation results output by the neural network to calculate the moments of each cotton seedling's outline. The moments determine the center of gravity of each outline and calculate the Euclidean distance between adjacent centers of gravity, eliminating the problem of multiple outlines per plant. The first-order central moments are then used to calculate cotton seedling growth statistics. The Deepsort algorithm is one of the most performant multi-target tracking algorithms. This paper uses the Deepsort algorithm to predict the target detection boxes output by the neural network, enabling real-time cotton seedling counting. A relationship model between the sampling area and the number of cotton seedlings is then established. The seedling emergence rate is calculated using the sampling area and the results output by the Deepsort algorithm. This provides a research foundation and strategy for future cotton seedling growth detection and real-time seedling emergence rate statistics.

[0249] The embodiments described above are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Without departing from the spirit of the present invention, various modifications and improvements made to the technical solutions of the present invention by persons skilled in the art should fall within the scope of protection defined by the claims of the present invention.

Claims

1. A method for automatically detecting the growth of cotton seedlings and estimating the emergence rate during the cotton seedling stage, characterized in that: The following steps are involved: S1: Build a lightweight instance segmentation network; S2: Collect cotton seedling information; S3: using Labelme software to mark the cotton seedling area corresponding to the cotton seedling information to create a data set; S4: Using the data set to train the lightweight instance segmentation network to obtain pre-trained weights; S5: Select sampling areas in the cotton field and collect video data using drones; S6: Load the pre-trained weights and detect the video data of the sampling area; S7: Segment the detected video and save the video segmentation results as images frame by frame; S8: looping through the images in S7 to obtain a cotton seedling outline matrix, and calculating the growth of the cotton seedlings based on the cotton seedling outline matrix; S9: Using the DeepSort algorithm, count the cotton seedlings in the target detection box in the detection result; S10: Based on the cotton seedling counts, modeling and estimating the cotton seedling emergence rate; In S1, the method for constructing the lightweight instance segmentation network includes: S11: Construct lightweight convolution SCS Conv; S12: Based on the lightweight convolution SCS Conv, construct a lightweight convolution block SCS Block; S13: Using the lightweight convolution block SCS Block, ordinary two-dimensional convolution, convolution with CBMA and Maxpooling, construct a feature extraction network Backbone; S14: Based on the lightweight convolution SCS Conv, construct a lightweight convolution block SCS bottleneck; S15: Use the lightweight convolution block SCS bottleneck and upsampling layer to build Neck; S16: Utilize the feature extraction network Backbone, the Neck, the target detection part, and the semantic segmentation part to construct a lightweight instance segmentation network.

2. The method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the cotton seedling stage according to claim 1, wherein: In S2, the method for collecting the cotton seedling information includes: S21: Use a drone to build a shooting platform, with the camera lens pointing vertically downward, using hovering shooting mode to capture images of cotton at the 3-4 leaf stage; S22: extracting a static image from the captured cotton image at an interval of 1 frame / s to complete the collection of the cotton seedling information.

3. The method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the cotton seedling stage according to claim 1, wherein: In S3, the method for preparing the data set includes: S31: Use the image annotation tool Labelme software to select the edge of the cotton seedling area for annotation; S32: The labeled images are randomly divided into a training set, a validation set, and a test set in a ratio of 7:2:1 to complete the preparation of the dataset.

4. The method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the cotton seedling stage according to claim 1, wherein: In S4, the training method of the lightweight instance segmentation network includes: Using the deep learning development framework Pytorch, the lightweight instance segmentation network structure is constructed and the lightweight instance segmentation network model is trained; Setting evaluation indicators of the lightweight instance segmentation network model, recall rate, precision rate, average accuracy rate, average intersection-over-union rate and F1 value score for evaluation; The lightweight instance segmentation network model is trained with an initial learning rate of 0.0001, and is trained using stochastic gradient descent with a decay rate of 0.9 and a batch size of 2. The training is performed for a total of 150 epoch iterations. After the training of the lightweight instance segmentation network model is completed, the loss, global correct, precision, recall, F1 value score, and average intersection-over-union ratio of each epoch are saved.

5. The method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the cotton seedling stage according to claim 1, wherein: In S8, the method of processing the image in S7 to obtain the cotton seedling outline matrix includes: S81: Perform image preprocessing on the RoImaps obtained by network segmentation, first converting the RoImaps into grayscale images, and then converting the RoImaps into binary images; S82: using the Otsu threshold method on the binary image obtained from S81 to obtain the outline of the cotton seedling; S83: Save the outline of each cotton seedling as a matrix, and then save all matrices as an outline list; S84: Use OpenCV to calculate the area of ​​each outline in the outline list, delete the noise points, and save the normal cotton seedling outline matrix.

6. The method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the cotton seedling stage according to claim 1, wherein: In S9, the method for counting cotton seedlings in the target detection frame in the detection result includes: S91: Read the position of the cotton seedling detection frame of the current frame and the depth features of the image blocks of each detection frame; S92: Filter the detection frame and depth features according to the confidence level; S93: Perform non-maximum suppression on the detection frame to eliminate the situation where there are multiple frames on the target; S94: Use Kalman filter to predict the position of cotton seedlings in the current frame; S95: The predicted cotton seedling positions are matched with the trajectories, and the cotton seedling counting is completed.

7. The method for automatically detecting the growth of cotton seedlings and estimating the emergence rate at the cotton seedling stage according to claim 1, wherein: In said S10, the method for modeling and estimating the cotton seedling emergence rate includes: S101: Select a plum blossom pattern for sampling based on the survey area and planting environment; S102: Based on the sampling results, when investigating the emergence rate of precision seeding of machine-picked cotton, calculation is performed using holes as units. Each hole is considered to have one seedling emerging. The emergence rate is calculated by dividing the number of seedlings emerging by the number of seeding holes:

Images