Farmland crop row identification method based on machine vision

Abstract

The invention discloses a farmland crop row identification method based on machine vision, and the method comprises the following steps: 1, building a crop row data set: collecting data, and carrying out the marking of an image through LabelImg software; in order to improve the generalization ability and effect of the model, mosaic data enhancement and mixed data enhancement are carried out on the image; label smoothing is carried out on the image, a hard label of data is converted into a soft label, and the generalization ability of the model is improved; step 2, a farmland crop row identification method: identifying and positioning crops by using a YOLO-G target detection algorithm so as to obtain the central point position of the crops; clustering the crop rows by using a clustering algorithm, and obtaining the number of the crop rows in the image and which crops exist in each crop row; and fitting a complex row line by using a least square method. According to the invention, farmland crop rows can be accurately identified, key technology and theoretical basis are provided for field visual navigation of the intelligent agricultural machine, and the intelligent agricultural machine is prevented from damaging crops during field autonomous operation.

Description

Technical Field

[0001] The invention relates to a method for identifying farmland crop rows, and in particular to a method for identifying farmland crop rows based on machine vision. Background Art

[0002] With the development of machine vision, visual navigation has become a research hotspot in the field of automatic navigation of agricultural machinery due to its low cost and strong adaptability. Crop row recognition is an important part of visual navigation. The difficulty of crop row recognition varies for different crops. The recognition of paddy field crop rows is easily affected by factors such as reflections, while the recognition of dry field crop rows is more easily affected by factors such as weeds. Object detection algorithms can accurately identify and locate crops in environments with many reflections and weeds, but some mainstream object detection algorithms have problems such as large number of parameters.

[0003] After identifying and locating the crops, the location information of the crops can be used to further identify the crop rows. Crop row identification can help intelligent agricultural machinery avoid crushing and damaging seedlings when operating autonomously in the field. Therefore, it is crucial to provide accurate crop row information for intelligent agricultural machinery. At present, the main methods for crop row identification are Hough transform method, vanishing point method, green pixel accumulation method and least squares method. Although the crop row recognition algorithm based on Hough transform can identify multiple crop rows at the same time, this method has disadvantages such as large amount of calculation. The vanishing point method requires an environment with fewer weeds and is only applicable when there are vanishing points. The disadvantage of the green pixel accumulation method is that it is very easily affected by weeds. There are many weeds in the dry corn field environment, which is not suitable for this method. The least squares method is fast and has a simple algorithm, so it is more suitable for crop row recognition methods. Summary of the invention

[0004] Aiming at the problem that crop row recognition in farmland is easily affected by factors such as light and weeds, resulting in low crop row recognition accuracy, the present invention provides a farmland crop row recognition method based on machine vision, improves the accuracy of crop row recognition in farmland through research on crop recognition and positioning algorithm, crop row clustering algorithm and crop row fitting algorithm, provides key technology for field visual navigation of intelligent agricultural machinery, thereby preventing intelligent agricultural machinery from damaging crops when operating autonomously in the field.

[0005] The objective of the present invention is achieved through the following technical solutions:

[0006] A method for identifying crop rows in farmland based on machine vision comprises the following steps:

[0007] Technical Solution 1:

[0008] Step 1: Build a rice crop row dataset

[0009] Step 1.1, collect data 7 days, 14 days and 21 days after rice transplanting. In order to increase the complexity of the data, the rice data 21 days after transplanting was collected under strong light, and the images were labeled using LabelImg software;

[0010] Step 1.2: In order to improve the generalization ability and effect of the model, mosaic data enhancement and mixed data enhancement are performed on the image;

[0011] Step 1.3: Smooth the image labels and convert the hard labels of the data into soft labels to improve the generalization ability of the model;

[0012] Step 2: Farmland crop row identification method

[0013] Step 2.1, use the YOLO-G target detection algorithm to identify and locate crops, so as to obtain the center point position of the crops, where: YOLO-G consists of three parts: the backbone network, the enhanced feature extraction network and the detection head. The backbone network replaces the backbone network of YOLOv7 with the lightweight convolutional neural network GhostnetV2, thereby reducing the number of parameters and the amount of calculation of the algorithm; the enhanced feature extraction network continues to use the PANet structure and SPPCSPC of YOLOv7, and then further reduces the number of parameters and the amount of calculation by reducing the number of feature map channels of the intermediate layer generated in the enhanced feature extraction network; a channel attention mechanism is added before the detection head to balance the importance relationship between each channel; the Focal Loss loss function is used to coordinate the matching between positive and negative samples;

[0014] Step 2.2: Use the DBSCAN clustering algorithm to cluster the crop rows, obtain the number of crop rows in the image and the crops in each crop row;

[0015] Step 2.3: Use the least squares method to fit the crop lines.

[0016] Technical Solution 2:

[0017] Step 1: Build a corn crop row dataset

[0018] Step 1.1, collect data of corn 7 days, 14 days and 21 days after emergence. In order to increase the complexity of the data, the corn image 21 days after emergence contains many weeds, and the image is labeled using LabelImg software;

[0019] Step 1.2: In order to improve the generalization ability and effect of the model, mosaic data enhancement and mixed data enhancement are performed on the image;

[0020] Step 1.3: Smooth the image labels and convert the hard labels of the data into soft labels to improve the generalization ability of the model;

[0021] Step 2: Crop row identification method

[0022] Step 2.1, use the YOLO-G target detection algorithm to identify and locate crops, so as to obtain the center point position of the crops, where: YOLO-G consists of three parts: the backbone network, the enhanced feature extraction network and the detection head. The backbone network replaces the backbone network of YOLOv7 with the lightweight convolutional neural network GhostnetV2, thereby reducing the number of parameters and the amount of calculation of the algorithm; the enhanced feature extraction network continues to use the PANet structure and SPPCSPC of YOLOv7, and then further reduces the number of parameters and the amount of calculation by reducing the number of feature map channels of the intermediate layer generated in the enhanced feature extraction network; a channel attention mechanism is added before the detection head to balance the importance relationship between each channel; the Focal Loss loss function is used to coordinate the matching between positive and negative samples;

[0023] Step 2.2, cluster the crop rows using the mean shift algorithm or the nearest neighbor propagation clustering algorithm to obtain the number of crop rows in the image and the crops in each crop row;

[0024] Step 2.3: Use the least squares method to fit the crop lines.

[0025] Compared with the prior art, the present invention has the following advantages:

[0026] (1) The present invention proposes a lightweight target detection algorithm YOLO-G. The YOLO-G algorithm is an improved algorithm based on YOLOv7. After adding four improved components, the number of parameters is only 23.61% of the YOLOv7 before the improvement, the amount of calculation is only 9.81% of the YOLOv7 before the improvement, and the number of frames detected per second increases by 10.49. At the same time, the rice AP of the YOLO-G algorithm is 96.29%, and the corn AP is 95.53%, which proves that the YOLO-G algorithm can better identify and locate rice and corn crops while being lightweight.

[0027] (2) The present invention uses the mean shift algorithm, the nearest neighbor propagation clustering algorithm and the DBSCAN clustering algorithm with the least squares method and the random sampling consensus algorithm to conduct experiments. The experiment shows that the method with the highest accuracy in rice crop row recognition is the YOLO-G algorithm, the DBSCAN clustering algorithm with the least squares method, with an accuracy of 94.49%. The method with the highest accuracy in corn crop row recognition is the YOLO-G algorithm, the mean shift algorithm with the least squares method, with an accuracy of 97.59%.

[0028] (3) The present invention can accurately identify crop rows in farmland, provide key technologies and theoretical basis for field visual navigation of intelligent agricultural machinery, and prevent intelligent agricultural machinery from damaging crops when operating autonomously in the field. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Provide a technical roadmap for machine vision-based crop row recognition methods;

[0030] Figure 2 For DFC attention;

[0031] Figure 3 It is the GhostNetV2 structure;

[0032] Figure 4 For transition modules and CBS structures;

[0033] Figure 5 It is the SPPCSPC structure;

[0034] Figure 6 Stack blocks for multiple branches;

[0035] Figure 7 It is the channel attention mechanism;

[0036] Figure 8 It is the YOLO-G structure;

[0037] Fig. 9 This is an example of clustering crop rows using the DBSCAN algorithm;

[0038] Fig.10 The DBSCAN algorithm clusters crop rows process;

[0039] Fig.11 Two fitting algorithms were used to identify rice crop rows for DBSCAN;

[0040] Fig.12 Two fitting algorithms were used to identify corn crop rows for DBSCAN;

[0041] Fig.13 The mean shift algorithm is paired with two fitting algorithms to identify rice crop rows;

[0042] Fig.14 The mean shift algorithm is paired with two fitting algorithms to identify corn crop rows;

[0043] Fig.15 The neighbor propagation algorithm is paired with two fitting algorithms to identify rice crop rows;

[0044] Fig.16 The neighbor propagation algorithm is paired with two fitting algorithms to identify corn crop rows. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0046] The present invention provides a method for identifying crop rows in farmland based on machine vision, such as Figure 1 As shown, the specific steps are as follows:

[0047] Step 1: Data collection and data enhancement

[0048] 1.1 Data collection and annotation

[0049] The data collection area of ​​the present invention is Acheng Farm in Acheng District, Harbin City, Heilongjiang Province, China, where rice crops and corn crops are planted.

[0050] 1.1.1 Rice data collection

[0051] In order to study the recognition ability of crop row recognition methods for paddy field crop rows in natural environments, the present invention constructs a rice data set. The present invention collects rice seedling data for three periods, which are collected 7 days, 14 days and 21 days after rice transplanting. There may be bumps during the movement of agricultural machinery, resulting in changes in the shooting angle. In order to improve the robustness of the algorithm, two angles are used for data collection in each period, with a depression angle of 65 degrees and 75 degrees. Detailed data on the number of rice images and the number of rice seedlings in each period are shown in Table 1. There are 100 rice images in each period, including 50 images at each angle.

[0052] Table 1 Summary of rice data

[0053]

[0054] During training, the training set accounts for 80% of the total data set, the validation set accounts for 10% of the total data set, and the test set accounts for 10% of the total data set. The algorithm is trained using the training set, and the performance is verified using the validation set, thereby adjusting the algorithm's hyperparameters. Once the algorithm's hyperparameters are determined, the algorithm is evaluated using the test set.

[0055] 1.1.2 Corn data collection

[0056] In order to study the crop row recognition ability of the crop row recognition method for dry fields under natural environment, the present invention constructs a corn data set. The present invention collects corn seedling data for three periods, which are collected 7 days, 14 days and 21 days after the corn seedlings emerge. Two angles are used for data collection in each period, with a depression angle of 75 degrees and 85 degrees. The number of pictures and the number of corn plants in each period are shown in Table 2. For the data set division during training, the corn data set is the same as the rice data set, with the training set accounting for 80% of the total data set, the validation set accounting for 10% of the total data set, and the test set accounting for 10% of the total data set.

[0057] Table 2 Summary of corn data

[0058]

[0059] 1.1.3 Data Annotation

[0060] Use the Python tool library LabelImg to label the image. When labeling, you need to draw a real box to surround the target and mark the category of the target. After labeling, LabelImg generates an xml file, which mainly contains the category of the target, the position of the target real box, and the size of the target real box. The position and size are determined by four parameters, namely the upper left corner horizontal coordinate, upper left corner vertical coordinate, lower right corner horizontal coordinate, and lower right corner vertical coordinate of the real box. The width of the real box can be obtained by subtracting the horizontal coordinate of the upper left corner from the horizontal coordinate of the lower right corner, and the height of the real box can be obtained by subtracting the vertical coordinate of the upper left corner from the vertical coordinate of the lower right corner, thereby obtaining the size of the real box.

[0061] 1.2 Data Augmentation

[0062] In order to improve the generalization ability and effect of the model, mosaic data augmentation and mixed data augmentation are performed on the image.

[0063] 1.2.1 Mosaic Data Enhancement

[0064] (1) Randomly select four images from the training set each time.

[0065] (2) Perform operations such as flipping, scaling, and color change on the four images. After the operations are completed, the first image of the original image is placed at the upper left position, the second image is placed at the lower left position, the third image is placed at the lower right position, and the fourth image is placed at the upper right position.

[0066] (3) After the images are arranged, a fixed area of ​​the image is cropped using a matrix method, and then the four images are spliced ​​into one image.

[0067] 1.2.2 Hybrid Data Augmentation

[0068] First, two images are randomly selected from the training set, and then the two images are flipped, scaled, and color-changed, and finally the two images are stacked together. Hybrid data augmentation can expand the distribution of a given dataset, thereby improving the generalization ability of the network.

[0069] 1.3 Label Smoothing

[0070] Label smoothing is often used in classification tasks to prevent the model from overconfidently predicting labels during training, thereby improving the model's generalization ability. Label smoothing techniques convert hard labels into soft labels, making network optimization smoother.

[0071] 2. Improved lightweight crop recognition and positioning algorithm YOLO-G

[0072] 2.1 Improved YOLO-G algorithm

[0073] The YOLO-G target detection algorithm is an improved target detection algorithm based on YOLOv7. YOLO-G consists of three parts: backbone network, enhanced feature extraction network and detection head. Figure 8 As shown in the figure, the present invention replaces the backbone network of YOLOv7 with GhostnetV2, thereby reducing the number of parameters and the amount of calculation of the algorithm. The enhanced feature extraction network of YOLO-G continues to use the PANet structure and SPPCSPC of YOLOv7, and then further reduces the number of parameters and the amount of calculation by reducing the number of feature map channels of the intermediate layer generated in the enhanced feature extraction network. YOLO-G adds a channel attention mechanism before the detection head to balance the importance relationship between each channel, and YOLO-G uses the Focal Loss loss function to coordinate the matching between positive and negative samples.

[0074] 2.1.1 Backbone Network GhostNetV2

[0075] YOLO-G replaces the backbone network of YOLOv7 with GhostNetV2. GhostNetV2 features Ghost Module, DFC attention mechanism and Ghost Bottleneck.

[0076] The ghost module uses a series of cheap linear transformations to generate ghost feature maps. The ghost module consists of two parts. The first step is to use ordinary convolution to generate a feature map without redundancy. The second step is to generate a new ghost feature map through cheap linear transformation of the feature map generated in the first step. The feature maps generated in the two steps are then stacked together to form a complete feature map. Compared with ordinary convolution, the ghost module requires fewer parameters and less computation.

[0077] Half of the feature maps generated by the ghost module are captured by cheap linear operations, which prevents it from communicating information with other pixels. Although the ghost module greatly reduces the amount of computation, its ability to capture spatial information is also weakened. In order to increase the module's ability to capture spatial information, an attention mechanism is needed. Self-attention has the ability to capture spatial information, but the amount of self-attention computation is large. DFC attention (Decoupled Fully Connected) reduces the amount of computation by capturing the spatial information of the image in the horizontal and vertical directions respectively. The amount of computation of DFC attention is lower than that of the self-attention mechanism. Therefore, GhostnetV2 uses the DFC attention mechanism to make up for the problem that the ghost module has a weak ability to capture spatial information.

[0078] Although the computational complexity of DFC attention is lower than that of self-attention, it is still larger than that of ghost module. Therefore, GhostNetV2 reduces the size of the input feature map before using DFC attention. After using the DFC attention mechanism to make each value of the output result contain the information of all input features, it uses upsampling to restore the feature map to its original size. This operation greatly reduces the computational complexity of DFC attention. Figure 2 As shown in Figure 1, the DFC attention mechanism is used in parallel with the ghost module. Although the ghost module branch only captures the information of a few pixels around the target pixel, the DFC attention branch captures the information of the entire image. Then, the two branches are dot-multiplied, and the final output captures the information of the entire image, making up for the problem that the ghost module has a weak ability to capture spatial information.

[0079] GhostNetV2 has two kinds of ghost bottlenecks. Figure 3 As shown in Figure 1, the ghost bottleneck with a stride of 1 uses two ghost modules, of which the first ghost module and the DFC attention mechanism are executed in parallel. The first ghost module of the ghost bottleneck with a stride of 2 is used in parallel with the DFC attention mechanism, and a depthwise separable convolution with a stride of 2 is inserted before the second ghost module. The depthwise separable convolution is used to halve the size of the feature map. Figure 3 The complete structure of GhostNetV2 is also shown. GhostNetV2 consists of a normal convolution, a ghost bottleneck with a stride of 1, two ghost bottlenecks-1, ghost bottleneck-5, and ghost bottleneck-4. Ghost bottleneck-1 is a ghost bottleneck with a stride of 2 followed by a ghost bottleneck with a stride of 1, ghost bottleneck-5 is a ghost bottleneck with a stride of 2 followed by five ghost bottlenecks with a stride of 1, and ghost bottleneck-4 is a ghost bottleneck with a stride of 2 followed by four ghost bottlenecks with a stride of 1.

[0080] The detailed parameters of each operation of the YOLO-G backbone network GhostNetV2 are shown in Table 3. The input in the table indicates the input feature map size of this operation, the operation indicates whether to perform a normal convolution operation or a ghost bottleneck, the expansion size indicates the number of channels in the hidden layer of the ghost bottleneck, the number of output channels indicates the number of channels in the output feature map of this operation, the SE (Squeeze and Excitation) attention mechanism indicates whether to use the SE attention mechanism, and the step size indicates the step size of the convolution or ghost bottleneck. When the step size is 1, the output feature map size remains unchanged, and when the step size is 2, the output feature map size is halved.

[0081] GhostNetV2 extracts three preliminary valid feature maps for subsequent processing, namely the feature map of size 80×80×40 generated by ghost bottleneck-1, the feature map of size 40×40×112 generated by ghost bottleneck-5, and the feature map of size 20×20×160 generated by ghost bottleneck-4.

[0082] Table 3 Network structure of GhostNetV2

[0083]

[0084] 2.1.2 Strengthening the feature extraction network

[0085] YOLO-G's enhanced feature extraction network fuses the three effective feature maps generated by the backbone network to obtain feature maps with more scale information. YOLO-G continues to use the PANet structure of YOLOv7, and while performing upsampling feature fusion on feature maps, it also performs downsampling feature fusion on features.

[0086] YOLO-G uses the transition block for downsampling. Figure 4 As shown in the figure, the CBS (Convolution BatchNorm SiLU) operation means that the convolution operation is performed first, and then the feature map generated by the convolution is BatchNorm normalized and the SiLU activation function is used. The transition module uses two branches, one branch uses a maximum pooling and a CBS operation of size 1×1, and the other branch first uses a CBS operation of size 1×1, and then uses a CBS operation of size 3×3 and a stride of 2×2 convolution. Finally, the feature maps of the two branches are stacked together to obtain the final feature map.

[0087] The enhanced feature extraction network uses the SPPCSPC structure. Figure 5The complete structure of SPPCSPC is shown. The structure has two branches. The upper branch first performs three CBS operations, and then performs maximum pooling of 5×5, 9×9, and 13×13 on the obtained feature maps. The feature maps generated by these three maximum pooling operations and the residual edges without maximum pooling are stacked, and then the stacked feature maps are subjected to two CBS operations to obtain the feature map of the upper branch. The lower branch performs a CBS operation to obtain the feature map. The feature maps of the upper and lower branches are stacked together, and then the stacked feature maps are subjected to a CBS operation to obtain the final feature map. Since this structure uses a larger residual edge to assist in optimization and feature extraction, the receptive field can be expanded.

[0088] YOLO-G's enhanced feature extraction network also uses multi-branch stacking blocks. Figure 6 As shown, the upper branch first performs a CBS operation, and then divides the obtained feature map into two branches, one of which is directly sent to the end for stacking, and the other performs two CBS operations in succession. It is then divided into two branches, one of which directly sends the feature map to the end for stacking, and the other performs two consecutive CBS operations on the generated feature map. The upper branch generates a total of three feature maps for stacking. The lower branch performs a CBS operation to obtain a feature map. Finally, the four feature maps generated by the upper and lower branches are stacked together to obtain a new feature map. This part uses too much stacking, thereby forming a denser residual connection, making the model easier to optimize and alleviating the gradient vanishing problem caused by increasing the depth. The present invention reduces the number of channels of the feature map generated by the intermediate layer in the enhanced feature extraction network, thereby reducing the number of parameters and the amount of calculation of YOLO-G.

[0089] 2.1.3 Channel Attention Mechanism

[0090] In order to coordinate the importance relationship between channels, this paper proposes a channel attention mechanism. Figure 7As shown, the channel attention mechanism proposed in the present invention consists of a global average pooling layer, a fully connected layer, a ReLU layer, a Hard Sigmoid layer, and a residual edge. First, global average pooling is performed, which is an operation to average the two-dimensional feature map of each channel, that is, the two-dimensional feature map of each channel corresponds to a mean value. The size of the feature map generated by this operation is 1×1×C, where C is the number of channels of the input feature map. Then the 1×1×C feature map is input to the fully connected layer, and the number of neurons in the fully connected layer is also C, so that the size of the output obtained is still 1×1×C. Then the result is input to the ReLU layer and the Hard Sigmoid layer, and the size of the output feature map is still 1×1×C. The feature map is multiplied by the corresponding channel of the input feature map, that is, the feature map of each channel of the input feature map is multiplied by the value in the corresponding 1×1×C feature map, and finally a feature map of the same size as the original input is obtained. Finally, a residual connection operation is performed to obtain the final result of the channel attention mechanism. The channel attention mechanism of the present invention focuses on coordinating the importance relationship between channels. The traditional convolution process assumes that each channel has the same importance. Each channel of YOLO-G represents different attributes, such as detection category, location, etc. The importance between channels is different, so it is necessary to adjust the importance relationship between channels.

[0091] 2.1.4Focal Loss Function

[0092] YOLO-G proposed in the present invention uses the Focal Loss loss function, which is the loss function proposed in RetinaNet. Focal Loss has two important features: controlling the weights of positive and negative samples and controlling the weights of easy-to-classify and difficult-to-classify samples. In the crop images of the present invention, there are generally only a dozen or dozens of crops that need to be detected, but tens of thousands of candidate boxes will be generated during prediction, of which only those that match the crop targets are positive samples, and the rest are negative samples. Therefore, the difference in the number of positive and negative samples is very large, so Focal Loss is used to control the weights of positive and negative samples.

[0093]

[0094] CE(p t )=-a t log(p t ) (3)

[0095] The p in formula (1) represents the probability that the sample is a positive sample, and the a in formula (2) is an adjustable hyperparameter, and the range of a is 0 to 1. Formula (3) shows that Focal Loss can be adjusted by controlling the parameter a. t The value of controls the contribution of positive and negative samples to the loss, and at It is determined by the hyperparameter a.

[0096] FL(p t )=-a t (1-p t ) γ log(p t ) (4)

[0097] For positive samples, the greater the probability of predicting it as a positive sample, the easier it is to classify. Therefore, the larger the value of 1-p, the harder it is to classify the sample. For negative samples, the greater the probability of predicting it as a negative sample, the easier it is to classify. Therefore, the larger the value of p, the harder it is to classify. Therefore, 1-p t It can be determined whether the sample is easy to classify or difficult to classify. Formula (4) is the complete formula of the FocalLoss loss function, where γ is a hyperparameter. At this time, the contribution of easy-to-classify and difficult-to-classify samples to the loss can be controlled by setting the value of γ.

[0098] 2.2 Experimental Analysis of YOLO-G Algorithm

[0099] 2.2.1 YOLO-G training parameters

[0100] YOLO-G uses rice data and corn data for training and testing respectively. The algorithm is trained for 300 rounds, of which the first 50 rounds are frozen training. In the frozen training, the backbone network is frozen, and the parameters of the backbone network do not change. Only the parameters in the enhanced feature extraction network are changed. The batch size is set to 1 at this time because each picture in the data contains dozens of targets, so a smaller batch size is set. After the 50th round, the training is unfrozen. At this time, the algorithm will fine-tune all its parameters and the batch size is set to 2. The optimizer during training selects Stochastic Gradient Descent, and the momentum of the stochastic gradient descent optimizer is set to 0.937. Mosaic data enhancement is used in the first 210 rounds of the algorithm training process. Since the training images generated by mosaic data enhancement are out of the real distribution of natural images, mosaic data enhancement is only used in the first 210 rounds of training, and data enhancement is turned off in the last 90 rounds. In the first 210 rounds, each round has only a 50% probability of using mosaic data augmentation, and after mosaic data augmentation, there is a 50% probability of using mixed data augmentation.

[0101] 2.2.2YOLO-G Experimental Results and Analysis

[0102] Table 4 shows the parameter comparison of YOLO-G, YOLOv7 and SSD. The number of parameters of YOLO-G is 23.61% of that of the improved YOLOv7 and 37.19% of that of the SSD algorithm. The amount of computation of YOLO-G is only 9.81% of that of the improved YOLOv7 and 16.97% of that of the SSD algorithm. The number of detection frames per second of YOLO-G is 14.49 and 1.35 higher than that of YOLOv7 and SSD algorithms respectively. YOLO-G has lower parameter and computational complexity, as well as higher number of detection frames per second.

[0103] Table 4 Comparison of main parameters of each algorithm

[0104]

[0105] Rice and corn are typical paddy field crops and dry field crops, respectively. The present invention uses rice data and corn data to evaluate the performance of the YOLO-G algorithm. As shown in Table 5, YOLO-G achieved high precision, F1 value and AP value of crops on rice and corn data in each period. The precision, recall and F1 value in this table are all the results when the confidence threshold is set to 0.5.

[0106] Table 5 Experimental results of YOLO-G algorithm

[0107]

[0108] 2.3 Analysis of the effectiveness of YOLO-G algorithm improvements

[0109] 2.3.1 Comparison results and analysis of different backbone networks

[0110] As shown in Table 6, in terms of parameter quantity, the parameter quantities of YOLOv7-GhostNetV2, YOLOv7-GhostNetV1, YOLOv7-MobilenetV1, YOLOv7-MobilenetV2 and YOLOv7-MobilenetV3 are close and relatively small, and the difference between the largest and the smallest among them is only 2.657M. In terms of computational complexity, the computational complexity of YOLOv7-GhostNetV2, YOLOv7-GhostNetV1, YOLOv7-MobilenetV2 and YOLOv7-MobilenetV3 is close and relatively small; in terms of detection frames per second, the detection frames per second of YOLOv7-GhostNetV2, YOLOv7-GhostNetV1, YOLOv7-MobileNetV1, YOLOv7-MobileNetV2 and YOLOv7-MobileNetV3 are all greater than 15. If we only consider the number of parameters, the amount of calculation, and the number of detection frames per second, YOLOv7-GhostNetV2, YOLOv7-GhostNet, YOLOv7-MobileNetV1, YOLOv7-MobileNetV2, and YOLOv7-MobileNetV3 are all suitable.

[0111] The experimental data selected all three-stage rice data and all three-stage corn data, and the rice AP value was evaluated on all three-stage rice data and the corn AP value was evaluated on all three-stage corn data. In terms of rice AP value, YOLOv7-GhostNetV2 is the highest. In terms of corn AP value, YOLOv7-GhostNetV2 ranks second, only 0.34% lower than the original YOLOv7 network.

[0112] Therefore, considering the five indicators of parameter quantity, calculation amount, number of frames detected per second, rice AP value and corn AP value, GhostNetV2 is the most suitable for matching with the YOLOv7 algorithm, whether it is for rice recognition and positioning or corn recognition and positioning. Therefore, the YOLO-G algorithm based on YOLOv7 proposed in the present invention uses GhostNetV2 as the backbone network.

[0113] Table 6 Comparison of backbone network selections of YOLOv7

[0114]

[0115] 2.3.2 Ablation study

[0116] As shown in Table 7, after the algorithm replaced the backbone network with GhostnetV2, the number of parameters decreased by 5.366M, the amount of calculation decreased by 59.620G, and the amount of calculation was only 45.49% of that before the modification. The number of frames detected per second increased by 4.09, the rice AP increased by 0.45%, and the corn AP value decreased by 0.34%.

[0117] After further reducing the number of channels of the feature map generated by the intermediate layer of the enhanced feature extraction network in the algorithm, the number of parameters decreased by 23.394M, which was only 26.50% of the number before the modification, the amount of calculation decreased by 35.181G, and the number of frames detected per second increased by 10.92, but the rice AP decreased by 2.60%, and the corn AP value decreased by 3.22%.

[0118] Although the above two improved components greatly reduce the algorithm's parameter volume, computational complexity, and increase the number of detection frames per second, they also reduce rice AP and corn AP. In order to make the algorithm smaller, faster, and more accurate, the present invention replaces the loss function with the Focal Loss loss function and then adds a channel attention mechanism. These two improved components increase the algorithm's rice AP value by 0.96% and corn AP value by 2.47% while only increasing the algorithm's parameter volume by 0.345M and computational complexity by 0.003G.

[0119] In short, after adding four improved components, the algorithm proposed by the present invention has only 23.61% of the parameters of the YOLOv7 before the improvement, and only 9.81% of the calculation amount of the YOLOv7 before the improvement. The number of frames detected per second increased by 10.49, while the rice AP value only decreased by 1.19% and the corn AP value only decreased by 1.09%. This makes the algorithm proposed by the present invention greatly reduce the algorithm's parameter amount and calculation amount, and improve the algorithm's detection frame number per second, while only slightly reducing the rice AP and corn AP, which makes YOLO-G more suitable for crop recognition and positioning in farmland.

[0120] Table 7 Ablation study of each network design

[0121]

[0122] 3. Crop row identification method

[0123] The crop row recognition method requires two steps. The first step is to identify and locate crops and obtain the location of crops. The second step is to identify crop rows according to the location of crops. The present invention uses a target detection algorithm to identify and locate crops. After obtaining the location of crops, the crop rows are first clustered using a crop row clustering algorithm. Each category represents a crop row, thereby obtaining the number of crop rows in the image and which crops are in each crop row. Finally, a crop row fitting algorithm is used to fit the crop row lines.

[0124] 3.1 Crop row clustering algorithm

[0125] 3.1.1DBSCAN (Density-Based Spatial Clustering ofApplications withNoise) algorithm

[0126] The neighborhood of the DBSCAN clustering algorithm is a circle, so it is not suitable for direct application in crop row identification. Fig. 9 An image with a width and height greater than 2000 pixels is shown. The points in the image represent crops. Taking rice crops as an example, because the paddy field is very muddy, it is difficult for the rice transplanter to plant the rice seedlings straight in the paddy field. The transplanter may also miss the rice seedlings, which leads to uneven distribution of rice seedlings on the same crop row. The rice seedlings on the same crop row may be far apart. Fig. 9 For the red point in the middle, if its neighborhood with a radius of Epsilon can include the crop point below it in the same crop row, it will also include crops in other crop rows, resulting in clustering errors. If you want to prevent the Epsilon neighborhood of the blue point in the figure from including crops in other crop rows, you need to set a smaller Epsilon. Under this smaller Epsilon, the other purple point cannot include its adjacent point in the same crop row below it into its own Epsilon neighborhood, resulting in clustering errors.

[0127] In order to allow the clustering algorithm to achieve better results in clustering crop rows, the present invention divides the vertical coordinates of all crops in the image by 1000. At this time, the horizontal coordinate range of the crops in the crop image is still 0 to 2500, but the vertical coordinate range is reduced to 0 to 2.5. If Epsilon is set to 110, the range of the Epsilon neighborhood for the image is no longer a circle, but an approximate rectangle. Because 110 is much larger than 2.5, the new neighborhood of each point can easily cover the top and bottom of the image. Another hyperparameter minimum point is set to 4. If the number of data points in the Epsilon neighborhood of a point exceeds 4, then the point is regarded as a core point, where the point itself is included in the calculation of the number, and if the center of other data points is in the neighborhood, the point is regarded as included.

[0128] Fig.10 The complete process of clustering the crop row on the far left of the image using the DBSCAN clustering algorithm is shown. First, Figure a) shows that a core point is randomly selected. The two vertical lines around the red point are its Epsilon neighborhood. Since there are more than 4 points in this neighborhood, the red point is the core point. Then Figure b) shows that all data points in the neighborhood of the red core point are classified as the same category as the red point. Figure c) shows that another core point is randomly selected from the red point category. The neighborhood of the specially marked red point in the figure has 6 data points, so it is the core point. Finally, Figure d) shows that all other points in the neighborhood of the newly selected core point are classified as the red category. Although there are other core points in the red category, since there are no other points except the red point in the neighborhood of these core points, the clustering of the crop row on the far left is completed. According to this process, other crop rows can be clustered in turn.

[0129] 3.1.2 Mean Shift Algorithm

[0130] (1) First, randomly select an unclustered point A from the data set as the search center point, and then find all data points that appear in the area with a radius (bandwidth) of h centered on A.

[0131] (2) Use the kernel function to calculate the offset vector of the center of the data. The center point moves along the offset vector. The moving distance is the length of the offset vector. Repeat this step. In this process, every time a data point appears in an area with a radius of h centered on the offset center point, it is marked with the center mark.

[0132] (3) When the center point stops moving, it reaches the area with the highest density. If there are other center points around the current center point and the distance is less than the threshold, the two center points are merged and the labels corresponding to the data points are also merged accordingly. Otherwise, the center point is considered a new cluster. Repeat the above steps until all data points are marked. If a data point is marked by multiple categories, the data point is assigned to the category with which it is marked the most times.

[0133] 3.1.3 Affinity Propagation Algorithm

[0134] (1) Initialize the attraction matrix and the belonging matrix to 0 matrices.

[0135] (2) Update the attraction matrix. As shown in formula (5), the new attraction matrix r t+1 Need to use the reference matrix S and the attraction matrix r of the previous iteration t and the attribution matrix at calculate.

[0136]

[0137] (3) Update the attribution matrix. As shown in formula (6), the new attribution matrix a t+1 Need to use the attraction matrix r t+1 calculate.

[0138]

[0139] (4) Use the decay coefficient λ to decay the two matrices. Formulas (7) and (8) show how to decay the matrix using the matrix of this iteration, the matrix of the previous iteration, and the decay coefficient, where the decay coefficient is a hyperparameter and the decay coefficient is used to avoid numerical oscillations during the update process.

[0140] r t+1 (x,z)=λr t (x,z)+(1-λ)r t+1 (x,z) (7)

[0141] a t+1 (x,z)=λa t (x,z)+(1-λ)a t+1 (x,z) (8)

[0142] The proximity propagation clustering algorithm continuously iterates and updates the attraction and belonging of each data point until multiple high-quality cluster centers are generated, and the remaining data points are assigned to their corresponding classes. The proximity propagation clustering algorithm needs to update the attraction and belonging of each data point in each iteration, so the algorithm is more complex.

[0143] 3.2 Crop row fitting algorithm

[0144] After clustering crops using the crop row clustering algorithm, the number of crop rows in the image and the crops in each crop row can be obtained, and then each crop row can be fitted using the crop row fitting algorithm. The present invention selects two crop row fitting algorithms, namely the least squares method and the random sampling consensus algorithm.

[0145] 3.3 Results and analysis of crop row recognition method

[0146] 3.3.1 Results and analysis of crop row recognition based on DBSCAN algorithm

[0147] Table 8 shows the crop row recognition accuracy of the DBSCAN clustering algorithm combined with the least squares method on rice and corn data at different periods. This combination achieved good results on both rice and corn data, with the crop row recognition accuracy of rice data at different periods exceeding 93% and the crop row recognition accuracy of corn data at different periods exceeding 95%. This proves that the DBSCAN clustering algorithm combined with the least squares method can better identify rice and corn crop rows.

[0148] Table 8 Accuracy of DBSCAN with least squares method

[0149]

[0150] Table 9 shows the crop row recognition accuracy of the DBSCAN clustering algorithm combined with the random sampling consensus algorithm on rice and corn data in different periods. The recognition result of this combination on corn crop rows is much lower than that of the DBSCAN clustering algorithm combined with the least squares method. The reason is that the DBSCAN clustering algorithm can better cluster the crops in each crop row on corn data, while the random sampling consensus algorithm sometimes regards the correct crops as noise points, resulting in incorrect crop row recognition.

[0151] Table 9 Accuracy of DBSCAN with random sampling consensus algorithm

[0152]

[0153] Fig.11 The following example shows how the DBSCAN clustering algorithm is used with the least squares method and the random sampling consensus algorithm to identify rice crop rows. The DBSCAN clustering algorithm divides different crop rows into different categories. The crops in the same color detection box in the figure represent the same category, that is, they belong to the same crop row. The crop row fitting algorithm fits row lines for different crop rows respectively, thereby identifying all crop rows. The red lines in the figure represent the crop row lines fitted by the crop row fitting algorithm. In Figure b), the DBSCAN algorithm is not disturbed by the discrete rice seedlings on the far left when clustering the blue crop row.

[0154] Fig.12 This example shows how the DBSCAN algorithm is used with the least squares method and the random sampling consensus algorithm to identify corn crop rows. The image shows that corn crops are more sparsely planted, with fewer plants in the same row and farther distances between adjacent rows, making them easier to cluster. There are only three or four corn plants in a row in the image, and both combinations can successfully fit it.

[0155] 3.3.2 Results and analysis of crop row recognition based on mean shift algorithm

[0156] Table 10 shows the crop row recognition accuracy of the mean shift clustering algorithm combined with the least squares method on rice and corn data at different periods. This combination has a poor effect on rice data 7 days after transplanting, and a good effect on rice data 14 days and 21 days after transplanting, indicating that the robustness of this method is weak on rice data with dense transplanting. This combination has achieved a very high crop row recognition accuracy on corn data at different periods, which is higher than the accuracy of the DBSCAN clustering algorithm combined with the least squares method for identifying corn crop rows introduced above. The corn crop row recognition accuracy of the mean shift algorithm combined with the least squares method in each period exceeded 96.00%, indicating that this method can effectively identify corn crop rows.

[0157] Table 10 Accuracy of mean shift algorithm combined with least squares method

[0158]

[0159] Table 11 shows the recognition results of the mean shift clustering algorithm combined with the random sampling consensus algorithm on rice and corn data in different periods. The mean shift clustering algorithm combined with the random sampling consensus algorithm has poor results on rice and corn data, and the accuracy of this combination is lower than that of the mean shift clustering algorithm combined with the least squares method. The reason is that the random sampling consensus algorithm sometimes mistakes crops in crop rows as noise points, resulting in recognition errors.

[0160] Table 11 Accuracy of mean shift algorithm combined with random sampling consensus algorithm

[0161]

[0162] Fig.13 Examples of fitting rice crop rows using the mean shift clustering algorithm with the least squares method and the random sampling consensus algorithm are shown. The least squares method usually works well when the crop rows are successfully clustered, but it is affected when they are incorrectly clustered. The advantage of the random sampling consensus algorithm is that it can find the incorrectly clustered crop points and then treat them as noise points, thereby clustering them according to the correct crop points. The disadvantage of the random sampling consensus algorithm is that it sometimes classifies the correctly clustered crop points as noise points, resulting in crop row recognition failure. b) The two crop rows on the far left in the figure both failed to be clustered, but the random sampling consensus algorithm still successfully fits these two crop rows. The reason is that the random sampling consensus algorithm treats the incorrectly clustered crops as noise points and only uses the correctly clustered crops to fit the crop rows.

[0163] Fig.14 The example of the mean shift clustering algorithm with the least squares method and the random sampling consensus algorithm for fitting corn crop rows is shown. The mean shift clustering algorithm with the least squares method works well on sparsely planted corn crops and can usually effectively identify crop rows. Fig.14 Both combinations achieved good recognition results.

[0164] 3.3.3 Crop row recognition results and analysis based on the nearest neighbor propagation algorithm

[0165] Table 12 shows the recognition results of the nearest neighbor propagation clustering algorithm combined with the least squares method on rice and corn data in different periods. The crop recognition results of this method on corn are significantly better than those on rice, because rice is planted densely and the distance between adjacent crop rows is close, making clustering more difficult. The nearest neighbor propagation clustering algorithm combined with the least squares method can better identify corn crop rows, but the effect of identifying rice crop rows is poor.

[0166] Table 12 Accuracy of the nearest neighbor propagation algorithm combined with the least squares method

[0167]

[0168] Table 13 shows the recognition results of the nearest neighbor propagation clustering algorithm combined with the random sampling consensus algorithm on rice and corn data in different periods. The result is still that the recognition of corn crop rows is more accurate, while the recognition effect of rice crop rows is poor. The main reason is that the nearest neighbor propagation clustering algorithm has a poor clustering effect on dense rice.

[0169] Table 13 Accuracy of the neighbor propagation algorithm combined with the random sampling consensus algorithm

[0170]

[0171] Fig.15 The example of the nearest neighbor propagation clustering algorithm with the least squares method and the random sampling consensus algorithm fitting rice crop rows is shown. In Figure b), the nearest neighbor propagation clustering algorithm mistakenly classifies the rice plant in the upper left corner into the green crop row, but the random sampling consensus algorithm still successfully clusters it.

[0172] Fig.16 The example of fitting corn crop rows using the neighbor propagation clustering algorithm with the least squares method and the random sampling consensus algorithm is shown. There are four corn crop rows in Figure a) and three corn crop rows in Figure b), and the neighbor propagation clustering algorithm successfully clusters them, and then the crop row fitting algorithm successfully fits the crop rows.

Claims

1. A method for identifying crop rows in farmland based on machine vision, characterized in that The method comprises the following steps: Step 1: Build a rice crop row dataset Step 1.1, collect data 7 days, 14 days and 21 days after rice transplanting, and use LabelImg software to annotate the images; Step 1.2: In order to improve the generalization ability and effect of the model, mosaic data enhancement and mixed data enhancement are performed on the image; Step 1.3: Smooth the image labels and convert the hard labels of the data into soft labels to improve the generalization ability of the model; Step 2: Farmland crop row identification method Step 2.1, use the YOLO-G target detection algorithm to identify and locate crops, so as to obtain the center point position of the crops, where: YOLO-G consists of three parts: the backbone network, the enhanced feature extraction network and the detection head. The backbone network replaces the backbone network of YOLOv7 with the lightweight convolutional neural network GhostnetV2, thereby reducing the number of parameters and the amount of calculation of the algorithm; the enhanced feature extraction network continues to use the PANet structure and SPPCSPC of YOLOv7, and then further reduces the number of parameters and the amount of calculation by reducing the number of feature map channels of the intermediate layer generated in the enhanced feature extraction network; a channel attention mechanism is added before the detection head to balance the importance relationship between each channel; the FocalLoss loss function is used to coordinate the matching between positive and negative samples; Step 2.2, clustering the crop rows using a clustering algorithm to obtain the number of crop rows in the image and the crops in each crop row; Step 2.3: Use the least squares method to fit the crop lines.

2. The method for identifying crop rows in farmland based on machine vision according to claim 1, characterized in that In step 1.1, data collection for each period is performed using depression angles of 65 degrees and 75 degrees.

3. The method for identifying crop rows in farmland based on machine vision according to claim 1, characterized in that In step 1.1, in order to increase the complexity of the data, the rice data 21 days after transplanting were collected under strong light.

4. The method for identifying crop rows in farmland based on machine vision according to claim 1, characterized in that In step 2.2, the clustering algorithm is the DBSCAN clustering algorithm.

5. A method for identifying crop rows in farmland based on machine vision, characterized in that The method comprises the following steps: Step 1: Build a corn crop row dataset Step 1.1, collect data of corn 7 days, 14 days and 21 days after emergence. In order to increase the complexity of the data, the corn image 21 days after emergence contains many weeds, and the image is labeled using LabelImg software; Step 1.2: In order to improve the generalization ability and effect of the model, mosaic data enhancement and mixed data enhancement are performed on the image; Step 1.3: Smooth the image labels and convert the hard labels of the data into soft labels to improve the generalization ability of the model; Step 2: Crop row identification method Step 2.1, use the YOLO-G target detection algorithm to identify and locate crops, so as to obtain the center point position of the crops, where: YOLO-G consists of three parts: the backbone network, the enhanced feature extraction network and the detection head. The backbone network replaces the backbone network of YOLOv7 with the lightweight convolutional neural network GhostnetV2, thereby reducing the number of parameters and the amount of calculation of the algorithm; the enhanced feature extraction network continues to use the PANet structure and SPPCSPC of YOLOv7, and then further reduces the number of parameters and the amount of calculation by reducing the number of feature map channels of the intermediate layer generated in the enhanced feature extraction network; a channel attention mechanism is added before the detection head to balance the importance relationship between each channel; the FocalLoss loss function is used to coordinate the matching between positive and negative samples; Step 2.2, clustering the crop rows using a clustering algorithm to obtain the number of crop rows in the image and the crops in each crop row; Step 2.3: Use the least squares method to fit the crop lines.

6. The method for identifying crop rows in farmland based on machine vision according to claim 5, characterized in that In step 1.1, data collection for each period is performed using depression angles of 75 degrees and 85 degrees.

7. The method for identifying crop rows in farmland based on machine vision according to claim 5, characterized in that In step 1.1, in order to increase the complexity of the data, the corn image 21 days after emergence contains many weeds.

8. The method for identifying crop rows in farmland based on machine vision according to claim 5, characterized in that In step 2.2, the clustering algorithm is a mean shift algorithm or a neighbor propagation clustering algorithm.

Images