Crop row center line extraction method

By introducing dynamic sparse attention mechanism, hollow convolution and multi-scale fusion module into the crop row centerline detection algorithm, the problems of low detection accuracy and high model complexity in the existing technology are solved, and the skewed crop rows are realized Accurate detection and efficient utilization of computing resources.

CN118941956APending Publication Date: 2024-11-12MECHANICS RES & DESIGN ACAD SICHUAN PROV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202410996100.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing crop row centerline detection algorithm has low detection accuracy when the planting density is high and the crop is skewed, and the high model complexity leads to high computing resources consumption, making it difficult to deploy on agricultural machinery equipment.

Method used

The crop image detection model is constructed using convolution module based on dynamic sparse attention mechanism, multi-scale fusion module with hollow convolution and weight sharing, upsampling module for prediction kernel and feature recombination, and object detection and evaluation module for two-dimensional Gaussian distribution. , improve detection accuracy and reduce model complexity.

Benefits of technology

It improves the accuracy and detection speed of crop row centerline detection, can effectively identify skew, shift and crop leaf occlusion, improves the small target detection effect, and is suitable for different agricultural scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941956A_ABST
    Figure CN118941956A_ABST
Patent Text Reader

Abstract

The invention discloses a crop row center line extraction method, which comprises the following steps: collecting crop image data, labeling crop images, and making a training set and a verification set; a crop image detection model is constructed, model hyper-parameters and a learning optimizer are set, and the crop image detection model comprises a feature extraction structure, a feature fusion structure and a regression prediction structure; model hyper-parameters are adjusted, and training set images are input into the crop image detection model for training; judging whether the crop image detection model converges and meets detection index requirements, if yes, inputting the test image into the crop image detection model, and outputting a detection result; otherwise, adjusting model hyper-parameters and continuing training; and fitting a line center line of a compound according to a detection result output by the crop image detection model. According to the method, skew, displacement and crop row center lines with mutually shielded crop leaves can be effectively identified, the detection effect on small targets is improved, and the problem of missing detection in different scenes of an existing algorithm is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to the technical field of automatic navigation of agricultural machinery, and in particular to a method for extracting the center line of a crop row. Background Art

[0002] In order to enable intelligent agricultural machinery to adjust its operating route according to actual working conditions during operation, it is necessary for it to have the ability of autonomous real-time navigation. At present, the main methods for automatic navigation of agricultural machinery are the Global Navigation Satellite System (GNSS), Inertial Navigation System (INS) and Machine Vision (MV) navigation system. However, the global satellite navigation system has shortcomings such as poor positioning accuracy and lack of environmental perception function in extreme environments or unstructured agricultural environments. Using only the global satellite navigation positioning system for navigation will fail to meet the requirements of precise operation due to insufficient local accuracy.

[0003] Since field crops are usually cultivated in straight lines and the crop rows are parallel to each other, the detection of the centerline of crop rows is often used as an important benchmark for visual navigation of agricultural machinery to make up for the lack of local accuracy of methods such as GPS, improve the accuracy of autonomous navigation of agricultural machinery, and meet the needs of precision agriculture. However, when crop rows are densely planted, adjacent rows of crops block each other, and some crops are skewed, lodged, or shifted, the existing visual detection methods for crop row centerline detection still have problems such as poor robustness and low accuracy. In order to solve the defects of mainstream methods, it is urgent to develop a crop row centerline detection algorithm to improve the detection accuracy.

[0004] There are two types of existing crop row centerline detection algorithms. One type is the traditional algorithm, which performs morphological filtering on the image, identifies crops, and then performs crop row recognition. For example, Zhai Zhiqiang et al. proposed crop row recognition and tracking using binocular vision and adaptive Kalman filtering. The other type is based on deep learning algorithms. Based on deep learning, the convolutional layer continuously abstracts and learns image features, extracts image features, and detects targets on this basis. For example, Wang Shanshan et al. proposed rice seedling row detection based on feature point neighborhood Hough transform, and rice crop row detection based on Faster RCNN. The Faster RCNN network was used to train the data set after data enhancement, and the trained model was used to detect the target crops in the image, detect the rice seedling feature points, and use the least squares method to perform straight line fitting on the obtained feature points to obtain the rice crop row centerline. The above algorithms all complete the extraction of crop row centerlines, but this type of method is mainly suitable for situations where the crop row interval is large and the crops are planted neatly. The accuracy of the crop row centerline extraction algorithm for crops with high planting density and crooked growth needs to be improved. Summary of the invention

[0005] The existing crop row centerline extraction algorithm has the following problems:

[0006] 1) Detection accuracy needs to be improved. The existing algorithm is suitable for crops with large spacing between crop rows and neat planting. For crops with high planting density and crooked growth, the existing method cannot accurately detect the center line of the crop row, which is not conducive to practical application in agricultural scenarios.

[0007] 2) The complexity of the model needs to be reduced. Although complex models can improve the accuracy of target detection, the addition of additional modules will also increase the resources required for model calculation, reduce the detection speed, and make it difficult to deploy on agricultural machinery that is sensitive to computing resources.

[0008] In order to solve the above problems, the present invention proposes a crop row centerline extraction method, which can effectively identify the crop row centerlines that are skewed, shifted, and occluded by crop leaves, improve the detection effect of small targets, and improve the missed detection problem that exists in different scenarios of existing algorithms.

[0009] The technical solution adopted by the present invention is as follows:

[0010] A crop row centerline extraction method, comprising:

[0011] Collect crop image data, annotate crop images, and create training sets and validation sets;

[0012] Constructing a crop image detection model, setting model hyperparameters and a learning optimizer; the crop image detection model includes a feature extraction structure, a feature fusion structure, and a regression prediction structure, the feature extraction structure includes a convolution module based on a dynamic sparse attention mechanism and a multi-scale fusion module based on dilated convolution and weight sharing, the feature fusion structure includes an upsampling module based on a prediction kernel and feature recombination, and the regression prediction structure includes a target detection evaluation module based on a two-dimensional Gaussian distribution;

[0013] Adjust the model hyperparameters and input the training set images into the crop image detection model for training;

[0014] Determine whether the crop image detection model converges and meets the detection index requirements. If so, input the test image into the crop image detection model and output the detection result; otherwise, adjust the model hyperparameters and continue training;

[0015] Fit the crop row centerline based on the detection results output by the crop image detection model.

[0016] Furthermore, the convolution module based on the dynamic sparse attention mechanism includes a convolution layer, a batch normalization layer, an activation function layer and a dynamic sparse attention mechanism unit connected in sequence.

[0017] Furthermore, the dynamic sparse attention mechanism unit is configured as follows:

[0018] On a two-dimensional level, the input feature map X∈R H×W×C Divide into S×S non-overlapping regions, extract regional features and reorganize to obtain feature maps And derive the vector through linear mapping

[0019] Q=X r ×W q

[0020] K=X r ×W k

[0021] V=X r ×W v

[0022] Where H is the height of the input feature map, W is the width of the input feature map, C is the number of channels in the input feature map, and R is a real number set; W q ,W k ,W v are different projection weight matrices respectively; q is the query, which is used to find information related to the input data; k is the key, which is used to calculate the correlation between the query and the input data; v is a value, which is weighted to calculate the output according to the correlation between the query and the key;

[0023] By constructing an adjacency matrix, we find two semantically related regions: we take the average of the vectors Q and K of each region to get the region-level vector Through the vector K r The transpose of and vector Q r Perform matrix multiplication to obtain the adjacency matrix A representing the regional correlation r :

[0024] A r =Q r (K r ) T

[0025] For each feature map region, the top m regions with a similarity higher than a preset value are found through the adjacency matrix, called topm, and only the topm of each region is retained to filter out the least relevant information; each row in the adjacency matrix represents the current region, and each column of the current row represents the degree of semantic association between other regions and the current region, and the column indexes of these regions are saved in a vector, defined as Ir :

[0026] I r =topmIndex(A r )

[0027] So through the vector I r Utilize attention mechanism at fine-grained level.

[0028] Furthermore, each vector Q will be calculated with the vector K related to different regions in the feature map. Since the vector K is in a non-continuous region in the feature map, direct calculation will result in low memory utilization. Therefore, the vectors K and V related to the vector Q are combined:

[0029] K g =gather(K,I r )

[0030] V g =gather(V,I r )

[0031] Get the attention mechanism value calculation formula:

[0032] O=Attention(Q,K g ,V g )+LCE(V)

[0033] Among them, LCE is a local context enhancement function based on deep convolution.

[0034] Furthermore, the multi-scale fusion module based on dilated convolution and weight sharing includes:

[0035] The spatial pyramid pooling unit is configured to divide the input features into multiple grids of different sizes, each grid corresponds to a receptive field of a set size, and generates regional features of different sizes; the pooling operation is used to extract the features in each grid and concatenate them to form a feature vector of fixed length;

[0036] The receptive field enhancement unit is configured to extract information of different scales through the dilated convolution layer and perform weighted operations through the acquisition weighted layer to balance the information representation of different branches.

[0037] Furthermore, the atrous convolution layer includes multiple atrous convolution branches and convolution kernels with different expansion rates to maintain the original resolution of the features; the acquisition weighted layer aggregates the information of different atrous convolution branches through weighted operations to represent them with richer features.

[0038] Furthermore, the upsampling module based on prediction kernel and feature recombination is configured as follows:

[0039] For any given input feature map X, if the upsampling rate is σ, the output feature map X O The position of any pixel in is represented by l O =(i O ,j O ), corresponding to the pixel position l = (i, j) of the input feature map X, where i = i O / σ,j=j O / σ; using N(X l ,k) represents the area with the pixel position l as the center and the area k×k; the upsampling kernel prediction module ψ is used to predict N(X l ,k) The upsampling kernel of the region:

[0040]

[0041] Among them, k encoder Indicates the size of the convolution kernel when the convolution operation is used during upsampling.

[0042] Furthermore, for a given input feature map X∈R H×W×C , compressing the number of channels to the preset value C m , where H is the height of the input feature map, W is the width of the input feature map, C is the number of channels in the input feature map, and R is a real number set; for the compressed input feature map, use k encoder ×k encoder The convolution kernel's convolution layer is aligned to implement the convolution operation, and the number of channels of its output feature map is k up 2 ×σ 2 , where σ is the upsampling rate, k uprepresents the number of reorganization kernels, both of which are hyperparameters;

[0043] Expand the feature map output by the convolution in the channel dimension and normalize the expanded elements to obtain a matrix containing the sampling kernel. According to the obtained sampling, the input feature map X∈R H×W×C Perform feature recombination:

[0044]

[0045] in, Represents feature reorganization, the matrix obtained by prediction The input feature map X l Neighborhood N(X l ,k up ) and the matrix of the corresponding position Do the dot product to complete the upsampling of this area.

[0046] Furthermore, the target detection evaluation module based on two-dimensional Gaussian distribution is configured as follows:

[0047] The bounding box is modeled with a two-dimensional Gaussian distribution, and the similarity between the corresponding Gaussian distributions is calculated based on the normalized Wasserstein distance, and applied to the loss function of the model to obtain the loss function L based on the normalized Wasserstein distance. NWD ;

[0048] L NWD =1-NWD(N a ,N b )

[0049] Among them, NWD is the normalized Wasserstein distance, N a is the Gaussian distribution of the prediction box, N b is the Gaussian distribution of the real target box;

[0050] Combined with the SIoU loss function, the target loss function is calculated:

[0051]

[0052] Among them, IoU_ratio is the proportion of SIoU loss function in the entire loss function.

[0053] Furthermore, fitting the crop centerline according to the detection result output by the crop image detection model includes:

[0054] After obtaining the center coordinate point of each crop through the crop image detection model, the identified center coordinate point is fitted into a straight line; a strip region of interest is set along the longitudinal direction of the image, and the center coordinate points of the crops in the same strip region of interest are extracted as a group to ensure that the center coordinate points of the crops in the same column are grouped together; the least squares method is used to fit the center points of the crops in the same group into a straight line, which is the center line of the crop row.

[0055] The beneficial effects of the present invention are:

[0056] 1. The present invention embeds a convolutional module Bi-CBS based on a dynamic sparse attention mechanism in the backbone network, filters background interference in the feature map in a dynamic perception manner, and retains important detail features in local areas, thereby enhancing the network's ability to learn crop features. The present invention uses a convolutional module Bi-CBS with dynamic query capabilities to achieve more flexible computing allocation and content perception. By retaining fine-grained detail information, the network model's detection performance for the core of lettuce is improved.

[0057] 2. The present invention introduces a multi-scale fusion module RFE-SPPF based on dilated convolution and weight sharing into the FPN (feature extraction) structure of the network, ensuring that the network can effectively retain the target structure information after multiple downsampling, which is conducive to the detection of small targets. The present invention uses the multi-scale fusion module RFE-SPPF to improve the receptive field of the feature map in the network while reducing the loss of feature structure information during the downsampling process, thereby improving the accuracy of small target detection.

[0058] 3. The present invention introduces an upsampling module CARAFE based on prediction kernel and feature recombination. The upsampling operation makes full use of the contextual information in the feature extraction process, expands the receptive field of the underlying features, and enhances the network's ability to extract small target features. The present invention uses an upsampling operator based on prediction kernel and feature recombination to detect the core of lettuce, which is prone to missed detection and false detection due to the high similarity between the target and the background. By effectively aggregating contextual information between different levels, better detection performance is achieved.

[0059] 4. The present invention introduces the target detection evaluation module NWD-SIoU based on two-dimensional Gaussian distribution and the target detection evaluation method NWD-SIoU based on Wasserstein distance, and applies it to the loss function of the network to solve the problem that the traditional IoU method is sensitive to the position of small targets. The present invention uses a target detection evaluation method based on two-dimensional Gaussian distribution, which can reduce the sensitivity of traditional evaluation methods to small target position deviations and improve network detection performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 The present invention is a flow chart of a crop row centerline extraction method.

[0061] Figure 2 It is a structural schematic diagram of a crop image detection model of the present invention.

[0062] Figure 3 This is the schematic diagram of the BRA attention mechanism.

[0063] Figure 4 This is the Bi-CBS convolution module structure diagram.

[0064] Figure 5 This is the RFE module structure diagram.

[0065] Figure 6 It is the structure diagram of the multi-scale fusion module.

[0066] Figure 7 This is the schematic diagram of the CARAFE upsampling operator. DETAILED DESCRIPTION

[0067] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, the specific implementation methods of the present invention are now described. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the embodiments described are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0068] Example 1

[0069] like Figure 1 As shown, this embodiment provides a crop row centerline extraction method, including:

[0070] Collect crop image data, annotate crop images, and create training sets and validation sets;

[0071] Constructing a crop image detection model, setting model hyperparameters and a learning optimizer; the crop image detection model includes a feature extraction structure, a feature fusion structure, and a regression prediction structure, the feature extraction structure includes a convolution module based on a dynamic sparse attention mechanism and a multi-scale fusion module based on dilated convolution and weight sharing, the feature fusion structure includes an upsampling module based on a prediction kernel and feature recombination, and the regression prediction structure includes a target detection evaluation module based on a two-dimensional Gaussian distribution;

[0072] Adjust the model hyperparameters and input the training set images into the crop image detection model for training; determine whether the crop image detection model converges and meets the detection index requirements. If so, input the test image into the crop image detection model and output the detection result; otherwise, adjust the model hyperparameters and continue training;

[0073] Fit the crop row centerline based on the detection results output by the crop image detection model.

[0074] Specifically, the crop row centerline extraction method of this embodiment is described in detail by taking the core of mature lettuce as an example.

[0075] 1. Dataset Construction

[0076] The collected images are processed and a visual image annotation tool (such as Labelme) is selected to annotate the collected images. In this embodiment, 800 images of lettuce planting under strong light and weak light conditions are collected during shooting. Among them, 600 images are used for training and verification of the recognition algorithm, of which 450 are used for training and 150 are used for verification. In addition, there are 200 unlabeled images for testing the recognition effect of the algorithm.

[0077] 2. Building a crop image detection model

[0078] Taking the widely used target detection framework YOLOv5 as an example, the crop image detection model construction method of this embodiment is specifically described. This embodiment embeds a dynamic convolution module in the backbone network of the target detection framework YOLOv5, filters the background interference in the feature map in a dynamic perception manner, and retains important detail features in local areas, thereby enhancing the network's learning ability for the lettuce core features. At the same time, a multi-scale fusion module based on hole convolution and weight sharing is introduced into the FPN (feature extraction) structure of the network to ensure that the network can effectively retain the target structure information after multiple downsamplings, which is beneficial to the detection of small targets such as the lettuce core. In addition, the CARAFE upsampling operation is introduced to make full use of the contextual information in the feature extraction process, expand the receptive field of the underlying features, and enhance the network's ability to extract small target features. Furthermore, a target detection evaluation method NWD-SIoU based on Wasserstein distance is proposed and applied to the network's loss function, which solves the problem that the traditional IoU method is sensitive to the position of small targets. The algorithm structure is as follows Figure 2 shown.

[0079] (1) Convolutional module based on dynamic sparse attention mechanism

[0080] In the color lettuce image, the core of the lettuce contains fewer pixels and will be disturbed by the outer lettuce leaves during the recognition process. Therefore, this embodiment designs a convolution module based on the dynamic sparse attention mechanism Bi-Level Routing Attention (BRA) and adds it to the yolov5 backbone network. The BRA attention mechanism module is mainly divided into three parts, and its principle is as follows: Figure 3 shown.

[0081] For the input feature map X∈R H×W×C, divide it into S×S non-overlapping regions on a two-dimensional level, extract regional features and reorganize to obtain feature maps And derive the vector through linear mapping

[0082] Q=X r ×W q

[0083] K=X r ×W k

[0084] V=X r ×W v

[0085] Where H is the height of the input feature map, W is the width of the input feature map, C is the number of channels in the input feature map, and R is a real number set; W q ,W k ,W v They are different projection weight matrices respectively; q is the query, which is used to find information related to the input data; k is the key, which is used to calculate the correlation between the query and the input data; v is a numerical value, which is weighted to calculate the output according to the correlation between the query and the key.

[0086] This embodiment finds two semantically related regions by constructing an adjacency matrix: taking the average of vectors Q and K of each region to obtain the region-level vector Through the vector K r The transpose of and vector Q r Perform matrix multiplication to obtain the adjacency matrix A representing the regional correlation r :

[0087] A r =Q r (K r ) T

[0088] For each feature map region, the top m regions with a similarity higher than a preset value are found through the adjacency matrix, which are called top m regions. Only the top m regions of each region are retained to filter out the least relevant information. Each row in the adjacency matrix represents the current region, and each column of the current row represents the degree of semantic association between other regions and the current region. The column indexes of these regions are saved in a vector, which is defined as I r :

[0089] I r =topmIndex(A r )

[0090] So through the vector I rThe attention mechanism is used at a fine-grained level. Each vector Q is calculated with the related vector K in different regions of the feature map. Since the vector K is in a non-continuous region of the feature map, direct calculation will result in low memory utilization. Therefore, the vector K and V related to the vector Q are combined:

[0091] K g =gather(K,I r )

[0092] V g =gather(V,I r )

[0093] Get the attention mechanism value calculation formula:

[0094] O=Attention(Q,K g ,V g )+LCE(V)

[0095] Among them, LCE (Local Context Enhancement) is a local context enhancement function based on deep convolution.

[0096] This embodiment designs a convolution module Bi-CBS based on BRA, and its structure is as follows: Figure 4 Replacing the convolution in the backbone network with Bi-CBS has fewer parameters and better detection effect than using the BRA attention mechanism alone, because Bi-CBS better integrates the network with the BRA module, thereby utilizing the dynamic perception ability of the BRA attention mechanism to enable the network to filter out the information least relevant to the target in the coarse-grained area while increasing a small amount of computation, and retain the detailed information in the fine-grained area, achieving a more powerful ability to extract the core features of lettuce.

[0097] (2) Multi-scale feature fusion module based on dilated convolution and weight sharing

[0098] This embodiment introduces the Receptive Field Enhancement (RFE) module to solve the problem that the convolutional neural network may lose pixel information after multiple layers of convolution, especially in the detection task of small targets such as the core of lettuce. This module is based on the design concept of dilated convolution and weight sharing. Through multi-scale fusion, it increases the receptive field of the feature map while maintaining the resolution, thereby improving the accuracy of small target detection. Its structure is as follows Figure 5 shown.

[0099] The RFE module consists of two main parts: the dilated convolution layer and the acquisition weighted layer. In the dilated convolution layer, we use three dilated convolution branches with different dilation rates (dilation rates are 1, 2, and 3) and a normal 3×3 convolution kernel to extract information of different scales. This strategy can effectively increase the receptive field without losing the resolution of the feature map. In the acquisition weighted layer, the information of different branches is aggregated through weighted operations to obtain richer feature representations. In addition, in order to further enhance the learning ability of the model and avoid the problem of gradient explosion, residual connections are introduced to better optimize the training effect of the model.

[0100] In order to meet the needs of lettuce core detection for receptive field enhancement and feature fusion, based on the proposed RFE module, this embodiment embeds it into the spatial pyramid pooling module SPPF of YOLOV5 and names it RFE-SPPF. The structure of RFE-SPPF is shown in the figure below: Figure 6 As shown. For the input features, SPPF first divides them into multiple grids of different sizes, each grid corresponds to a receptive field of a specific size, and generates regional features of different sizes. Next, a pooling operation is used to extract the features within each grid, and these features are spliced ​​together to form a feature vector of fixed length. This feature vector is then input into the RFE module. The RFE module includes three dilation convolution branches with different expansion rates to maintain the original resolution of the features. Finally, the information of the three branches is weighted to balance the information representation of different branches.

[0101] (3) Upsampling module based on prediction kernel and feature reorganization

[0102] The feature extraction of lettuce images is affected by factors such as the small size of the lettuce core area and the low image resolution, resulting in less effective information that can be used for lettuce core identification. In this case, effectively aggregating context information and understanding scene and object structure information are crucial for small target detection in the lettuce core. In order to effectively aggregate upper and lower information, this embodiment introduces an upsampling operator based on prediction kernel and feature recombination.

[0103] For any given input feature X, assuming the upsampling rate is σ, the output feature map X O .X O The position of any pixel in is represented by l O =(i O ,j O ), which corresponds to the pixel position l = (i, j) of the input feature map, where i = i O / σ,j=j O / σ. In addition, this embodiment uses N(X l,k) represents the area of ​​k×k with pixel position l as the center. The upsampling kernel prediction module ψ is used to predict N(X l ,k) The upsampling kernel W of the region l , the process is shown in the following formula, and the upsampling kernel prediction module is detailed as follows Figure 7 shown.

[0104]

[0105] Among them, k encoder Represents the convolution kernel size used in the convolution operation during upsampling.

[0106] For a given input feature X∈R H×W×C , first compress the number of channels to C m To reduce the computational overhead of subsequent processes. For the compressed feature map, use k encoder ×k encoder The convolution kernel's convolution layer is aligned to implement the convolution operation, and the number of channels of its output feature map is k up 2 ×σ 2 , where σ is the upsampling rate, k up Represents the number of reorganization kernels, both of which are hyperparameters.

[0107] Expand the feature map output by the convolution in the channel dimension and normalize the expanded elements to obtain a matrix containing the sampling kernel. According to the obtained sampling, the input feature map X∈R H×W×C Perform feature recombination:

[0108]

[0109] in, Represents feature reorganization, the matrix obtained by prediction The input feature map X l Neighborhood N(X l ,k up ) and the matrix of the corresponding position Do the dot product to complete the upsampling of this area.

[0110] (4) Object detection evaluation module based on two-dimensional Gaussian distribution

[0111] The task of extracting the core of lettuce in lettuce images not only faces the problems of low resolution and few extractable features, but also faces the problem that the traditional IoU detection algorithm is not sensitive to small-scale target detection. Since the sensitivity of IoU to objects of different scales varies greatly, the label assignment ultimately has the following defects: In the object detector based on the anchor box, the IoU threshold (θ p ,θ n) to distribute positive and negative samples to support network training iterations. First, the sensitivity of IoU to small-sized objects means that even a small deviation in the position of the object may cause the anchor box label to flip, making the features of positive and negative samples similar and making it difficult for the network to converge. In addition, this will also affect the non-maximum suppression (NMS) and regression loss functions of the network, resulting in poor detection of small objects by the original network.

[0112] In view of the above problems, this embodiment introduces a new measurement method to calculate the similarity between the target box and the regression box. First, the bounding box is modeled as a two-dimensional Gaussian distribution, and a new measurement method based on Wasserstein distance is proposed, named Normalized Wasserstein Distance (NWD) to calculate the similarity between the corresponding Gaussian distributions. The similarity obtained in this way will not change greatly due to the slight position change of the small target bounding box. It is applied to the loss function of the model to obtain the loss function L based on NWD. NWD As shown in the following formula.

[0113] L NWD =1-NWD(N a ,N b )

[0114] Where: N a is the Gaussian distribution of the prediction box, N b is the Gaussian distribution of the true target box.

[0115] Since the core of lettuce is not entirely small-scale targets, if it is completely based on the NWD loss function, the model fitting speed will be significantly increased. To address this problem, this embodiment designs a new loss function NWD-SIOU based on NWD and SIoU, which can not only improve the effect of small target detection, but also alleviate the decrease in network fitting speed.

[0116]

[0117] Where: IoU_ratio is the proportion of SIoU in the entire loss function.

[0118] 3. Crop row centerline fitting

[0119] After obtaining the center coordinates of each lettuce, the identified coordinate points need to be fitted into a straight line. In this embodiment, a strip region of interest (ROI) is set along the longitudinal direction of the image, and the center coordinates of the lettuces in the same ROI are extracted as a group to ensure that the center coordinate points of the same row of crops are grouped together. Finally, the least squares method is used to fit the center points of the crops in the same group into a straight line, which is the center line of the crop row.

[0120] 4. Algorithm training and testing

[0121] In terms of the experimental environment, this example uses the PyTorch deep learning framework as the software tool and builds a machine equipped with A workstation with CoreTM I5-9400F CPU@2.90Ghz, NVIDIAGeforce GTX 1080GPU, and 32G running memory is used as the hardware device. During the experiment, this embodiment uses the SGD optimizer and sets the training parameters according to Table 1 to improve the convergence speed of the model.

[0122] Table 1 - Training parameters

[0123] Parameter name Parameter information Learning rate (Lr0) 0.01 Cosine annealing parameter (Lrf) 0.01 Learning rate momentum 0.937 Batch size 8

[0124] To ensure the fairness of the experiment, the number of training rounds for each experiment was set to 300 epochs, and four evaluation indicators, including precision (Precision, P), recall (Recall, R), average precision (Average Precision, AP) and GFLOPs, were selected to evaluate the performance of the model.

[0125] Precision (P): refers to the probability of correct detection among all detected targets.

[0126] Recall (R): refers to the probability of correct identification among all positive samples.

[0127] Average Precision (AP): The area under the two-dimensional curve with precision as the vertical axis and recall as the horizontal axis. It combines precision and recall to more comprehensively measure the performance of the model.

[0128] GFLOPs (Floating Point Operations Per Second): represents the number of floating point operations performed by the model in one second during the inference process, which can be used to evaluate the computational complexity of the model.

[0129] V. Experimental Analysis

[0130] In order to demonstrate the superiority of the algorithm proposed in this embodiment in terms of detection accuracy and speed, it is compared with the current mainstream target detection model, and the lettuce dataset is tested under the same experimental environment and parameter conditions. The model comparison experimental results are shown in Table 2. From the results in Table 2, it can be found that the algorithm in this embodiment achieves the best effect.

[0131] Table 2 - Model comparison experiment

[0132] Model AP R FPS / (frame / s) YOLOv5s 0.525 0.511 61.4 YOLOv5n 0.462 0.473 70.9 YOLOv7 0.493 0.486 41.2 YOLOv8n 0.529 0.515 56.2 YOLOv8s 0.541 0.533 49.8 Faster-RCNN 0.496 0.504 7.25 ours 0.586 0.574 42.9

[0133] The test results of YOLOv5s, YOLOv7, YOLOv8s and the improved model under strong light and weak light conditions were compared. Each model missed the detection of the core of the back row lettuce, and the confidence of the detected target was also very low. In some locations, repeated detection occurred. The core of the back row lettuce is small, and the original model has poor detection effect on it. The improved model recognizes all the cores of the back row lettuce, and the detection confidence is also higher than the original model overall. The detection effect of each model on the target edge of the image is not ideal, and the improved algorithm detects more targets. Comprehensive detection effect comparison and the above analysis show that compared with the existing algorithm, the improved algorithm has improved the detection effect of small targets, and effectively improved the problem of missed detection in different scenarios of the existing algorithm.

[0134] Comparison of the crop row centerline obtained from the model results with the actual crop row centerline shows that each column of the crop row centerline obtained from the original model detection results deviates from the actual centerline, especially the circled part in the figure, where the deviation is quite serious. The crop row centerline obtained from the improved model detection results has a smaller deviation from the actual centerline, which is more in line with the actual situation.

[0135] Example 2

[0136] This embodiment is based on embodiment 1:

[0137] This embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the crop row centerline extraction method of embodiment 1 is implemented. The computer program may be in source code form, object code form, executable file, or some intermediate form.

[0138] Example 3

[0139] This embodiment is based on embodiment 1:

[0140] This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the crop row centerline extraction method of Embodiment 1. The computer program may be in source code form, object code form, executable file, or some intermediate form, etc. The storage medium includes: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the storage medium does not include electric carrier signals and telecommunication signals.

[0141] It should be noted that, for the above method embodiments, for the sake of simplicity of description, they are expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

Claims

1. A method for extracting the center line of a crop row, characterized in that: include: Collect crop image data, annotate crop images, and create training sets and validation sets; Build a crop image detection model, set model hyperparameters and learn optimizer; The crop image detection model includes a feature extraction structure, a feature fusion structure and a regression prediction structure. The feature extraction structure includes a convolution module based on a dynamic sparse attention mechanism and a multi-scale fusion module based on a dilated convolution and weight sharing. The feature fusion structure includes an upsampling module based on a prediction kernel and feature reorganization. The regression prediction structure includes a target detection evaluation module based on a two-dimensional Gaussian distribution. Adjust the model hyperparameters and input the training set images into the crop image detection model for training; Determine whether the crop image detection model converges and meets the detection index requirements. If so, input the test image into the crop image detection model and output the detection result; Otherwise, adjust the model hyperparameters and continue training; Fit the crop row centerline based on the detection results output by the crop image detection model.

2. A crop row centerline extraction method according to claim 1, characterized in that: The convolution module based on the dynamic sparse attention mechanism includes a convolution layer, a batch normalization layer, an activation function layer and a dynamic sparse attention mechanism unit connected in sequence.

3. A crop row centerline extraction method according to claim 2, characterized in that: The dynamic sparse attention mechanism unit is configured as follows: On a two-dimensional level, the input feature map X∈R H×W×C Divide into S×S non-overlapping regions, extract regional features and reorganize to obtain feature maps And derive the vector through linear mapping Q=X r ×W q K=X r ×W k V=X r ×W v Where H is the height of the input feature map, W is the width of the input feature map, C is the number of channels in the input feature map, and R is a real number set; W q ,W k ,W v are different projection weight matrices respectively; q is the query, which is used to find information related to the input data; k is the key, which is used to calculate the correlation between the query and the input data; v is a value, which is weighted to calculate the output according to the correlation between the query and the key; By constructing an adjacency matrix, we find two semantically related regions: we take the average of the vectors Q and K of each region to get the region-level vector Through the vector K r The transpose of and vector Q r Perform matrix multiplication to obtain the adjacency matrix A representing the regional correlation r : A r =Q r (K r ) T For each feature map region, the top m regions with a similarity higher than a preset value are found through the adjacency matrix, which are called top m regions. Only the top m regions of each region are retained to filter out the least relevant information. Each row in the adjacency matrix represents the current region, and each column of the current row represents the degree of semantic association between other regions and the current region. The column indexes of these regions are saved in a vector, which is defined as I r : I r =topmIndex(A r ) So through the vector I r Utilize attention mechanism at fine-grained level.

4. A crop row centerline extraction method according to claim 3, characterized in that: Each vector Q will be calculated with the vector K related to different regions in the feature map. Since the vector K is in a non-continuous region in the feature map, direct calculation will result in low memory utilization. Therefore, the vectors K and V related to the vector Q are combined: K g =gather(K,I r ) V g =gather(V,I r ) Get the attention mechanism value calculation formula: O=Attention(Q,K g ,V g )+LCE(V) Among them, LCE is a local context enhancement function based on deep convolution.

5. A crop row centerline extraction method according to claim 1, characterized in that: The multi-scale fusion module based on dilated convolution and weight sharing includes: The spatial pyramid pooling unit is configured to divide the input features into multiple grids of different sizes, each grid corresponds to a receptive field of a set size, and generates regional features of different sizes; the pooling operation is used to extract the features in each grid and concatenate them to form a feature vector of fixed length; The receptive field enhancement unit is configured to extract information of different scales through the dilated convolution layer and perform weighted operations through the acquisition weighted layer to balance the information representation of different branches.

6. A crop row centerline extraction method according to claim 5, characterized in that: The atrous convolution layer includes multiple atrous convolution branches and convolution kernels with different expansion rates to maintain the original resolution of the features; the acquisition weighted layer aggregates the information of different atrous convolution branches through weighted operations to represent them with richer features.

7. A crop row centerline extraction method according to claim 1, characterized in that: The upsampling module based on prediction kernel and feature recombination is configured as follows: For any given input feature map X, if the upsampling rate is σ, the output feature map X O The position of any pixel in is represented by l O =(i O ,j O ), corresponding to the pixel position l = (i, j) of the input feature map X, where i = i O / σ,j=j O / σ; using N(X l ,k) represents the area with the pixel position l as the center and the area k×k; the upsampling kernel prediction module ψ is used to predict N(X l ,k) The upsampling kernel of the region: Among them, k encoder Indicates the size of the convolution kernel when the convolution operation is used during upsampling.

8. A crop row centerline extraction method according to claim 7, characterized in that: For a given input feature map X∈R H×W×C , compressing the number of channels to the preset value C m , where H is the height of the input feature map, W is the width of the input feature map, C is the number of channels in the input feature map, and R is a real number set; for the compressed input feature map, use k encoder ×k encoder The convolution kernel's convolution layer is aligned to implement the convolution operation, and the number of channels of its output feature map is k up 2 ×σ 2 , where σ is the upsampling rate, k up represents the number of reorganization kernels, both of which are hyperparameters; Expand the feature map output by the convolution in the channel dimension and normalize the expanded elements to obtain a matrix containing the sampling kernel. According to the obtained sampling, the input feature map X∈R H×W×C Perform feature recombination: in, Represents feature reorganization, the matrix obtained by prediction The input feature map X l Neighborhood N(X l ,k up ) and the matrix of the corresponding position Do the dot product to complete the upsampling of this area.

9. A crop row centerline extraction method according to claim 1, characterized in that: The target detection evaluation module based on two-dimensional Gaussian distribution is configured as follows: The bounding box is modeled with a two-dimensional Gaussian distribution, and the similarity between the corresponding Gaussian distributions is calculated based on the normalized Wasserstein distance, and applied to the loss function of the model to obtain the loss function L based on the normalized Wasserstein distance. NWD ; L NWD =1-NWD(N a ,OF b ) Where NWD is the normalized Wasserstein distance, N a is the Gaussian distribution of the prediction box, N b is the Gaussian distribution of the true target box; Combined with the SIoU loss function, the target loss function is calculated: Among them, IoU_ratio is the proportion of SIoU loss function in the entire loss function.

10. A crop row centerline extraction method according to claim 1, characterized in that: The step of fitting the crop centerline according to the detection result output by the crop image detection model includes: After obtaining the center coordinate point of each crop through the crop image detection model, the identified center coordinate point is fitted into a straight line; a strip region of interest is set along the longitudinal direction of the image, and the center coordinate points of the crops in the same strip region of interest are extracted as a group to ensure that the center coordinate points of the crops in the same column are grouped together; the least squares method is used to fit the center points of the crops in the same group into a straight line, which is the center line of the crop row.

Citation Information

Patent Citations

  • Corn crop row identification method based on Faster-YOLOv8s network

    CN117036861A

  • Improved insulator spontaneous explosion detection method based on YOLOv8

    CN117456380A

  • Small target detection method and system based on multi-scale extended residual network

    CN117611963A