Real-time acquisition method of crop row direction for assisted navigation based on CNN

Abstract

This invention, applicable to the field of neural network technology, provides a CNN-based method for real-time acquisition of crop row direction for assisted navigation. The method comprises the following steps: acquiring a video file, rotating the image by a certain angle based on the angle α between the image center ridge and the horizontal baseline in the video file to obtain an image dataset containing 180 categories; filtering the image dataset containing 180 categories to obtain an image dataset containing 60 categories, and determining a training set, a validation set, and a test set; determining convolution control parameters, activation functions, an optimizer, and a CNN model structure to construct a CNN model; and verifying the accuracy and processing speed of the CNN model. This invention provides real-time, continuous, and acceptable directional guidance based on crop rows with high accuracy. It can be used stably and independently on unmanned intelligent farmland operation platforms, or in real-time auxiliary navigation mode when the primary navigation method unexpectedly fails.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network technology, and in particular to a method for real-time acquisition of crop row directions for auxiliary navigation based on CNN. Background Art

[0002] With the increasing application of intelligent unmanned equipment in various agricultural auxiliary fields, field navigation technology, including plant protection drones and field operation equipment, has gradually become an important research area. In recent years, various types of field navigation technologies have been proposed for agricultural purposes. Currently, autonomous navigation technologies for field equipment can be divided into: 1. RTK-GPS signal methods; 2. Visual imagery; 3. Ultralight radar; 4. Laser ranging; and 5. Wireless network positioning. Although these methods vary greatly in technical difficulty and implementation cost, they are specifically targeted at different applications and can fully utilize their advantages. Many of these methods are based on visual imagery and have received extensive attention and research. Because they are not restricted by the wireless signal reception environment, they do not require relatively expensive auxiliary equipment (such as GPS differential equipment) and can achieve satisfactory accuracy with relative ease.

[0003] The operating paths of intelligent unmanned equipment, such as crop protection drones, in field operations require regular coverage of target farmland areas. Considering issues such as navigation accuracy, flexibility, and application cost, as well as the unique visual characteristics of the work area, methods based on visual images offer advantages. Crop rows (ridges) are a common and distinct feature in field operations. Navigation strategies based on crop row (ridge) recognition are the optimal choice for intelligent operation and direction determination.

[0004] Crop rows are often used for navigation in agricultural machinery. Currently, a variety of solutions have been developed. Generally speaking, these can be categorized as follows: 1. Hough transform methods; 2. Color information; 3. Texture classification; 4. Linear regression; 5. Blob analysis; 6. Green pixel arrangement / accumulation; 7. Frequency analysis; 8. Stereo vision; 9. Horizontal strip detection; and 10. Vanishing point methods.

[0005] When drone-acquired crop row images are used to locate farmland equipment, the width of the crop rows in the images is not fixed, as the drone's altitude can be adjusted based on the scene, and drones can be used for farmland operations throughout the growing season. This requires navigation algorithms to be adaptable throughout the season. Adaptability to unexpected situations, such as hilly terrain in the planting area, irregularities in crop rows, and changes in plant cover during the growing season, should also be considered. Although the aforementioned algorithms have achieved good results, most are very specific. When they are separated from the target scene environment, and a field of planting often presents different appearances in different seasons, the expected accuracy will be greatly reduced. Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a real-time acquisition method of crop row direction for assisted navigation based on CNN to solve the problems existing in the above background technology.

[0007] The present invention is implemented as follows: a method for real-time acquisition of crop row direction for assisted navigation based on CNN, the method comprising the following steps:

[0008] The video file is collected and the image is rotated by a certain angle according to the angle α between the central ridge direction of the image and the horizontal baseline in the video file to obtain an image dataset containing 180 categories;

[0009] The image dataset containing 180 categories was screened to obtain an image dataset containing 60 categories, and the training set, validation set, and test set were determined;

[0010] Determine the convolution control parameters, activation function, loss function, optimizer and CNN model structure, and build the CNN model;

[0011] The accuracy of the CNN model is verified, and the processing speed of the CNN model is verified.

[0012] As a further solution of the present invention, the step of rotating the image by a certain angle according to the angle α between the image center ridge direction and the horizontal baseline in the video file to obtain an image dataset containing 180 categories specifically includes:

[0013] Extract an image from the video file every two frames, continuously cut out three overlapping image regions from the left, middle, and right regions of the extracted image, and save the overlapping image regions into the image set A folder as the initial image file;

[0014] Determine the angle α between the central ridge direction and the horizontal baseline of each image in the image set A folder, append the angle α to the image file name, capture a fixed area in the center of the image, and use the central ridge direction as the cropping line to obtain the image set B folder;

[0015] The image is rotated by a certain angle based on the angle α to generate an image with a cropping line ranging from 0° to 179° relative to the horizontal baseline. The central area of the image is cropped once every 1 degree of rotation.

[0016] Add the current rotation angle to name each new image file, and finally rotate to 180 angles, generating 180 folders. The resulting image dataset contains 180 categories.

[0017] As a further solution of the present invention: the steps of screening the image dataset containing 180 categories to obtain an image dataset containing 60 categories and determining the training set, validation set, and test set specifically include:

[0018] The image dataset containing 180 categories was screened to obtain an image dataset containing 60 categories. During the screening, every 3° was considered as a category.

[0019] The image file names in the image dataset are named according to the following rules: current rotation angle + cropping line angle + frame number + region sequence number + video name suffix. The region number is 0, 1, or 2, corresponding to the three overlapping image regions captured.

[0020] The image files of 60 categories are divided into training set, validation set and test set in the ratio of 7:2:1 and placed in the main folder.

[0021] As a further solution of the present invention: the step of determining the convolution control parameter specifically includes:

[0022] Set the step size of the image convolution operation to 1;

[0023] According to the boundary value of the conv2d convolution function in Keras, the edge length of the convolution kernel is reduced by 1;

[0024] The output of the convolutional layer is nonlinearly mapped, and the activation function uses a single threshold and a fast-converging ReLU (Rectified Linear Unit, f(x) = max(0, x)) function.

[0025] As a further solution of the present invention: ReLU is selected as the activation function of the hidden layer of the model, and the softmax classifier is selected as the activation function of the output layer of the model; the multi-classification cross entropy function is used as the loss function; and the Adam algorithm is selected as the CNN network optimizer.

[0026] As a further solution of the present invention: the CNN model structure is a CNN network structure with five hidden layers, which is used to identify crop row angles.

[0027] As a further solution of the present invention: the step of verifying the accuracy of the CNN model is specifically as follows: for the test video, each time a scene image is obtained, the CNN network processes the scene image, gives the deviation angle of the cropping line in the scene image and the vertical direction of the image center, and then corrects the image rotation according to the deviation angle and displays it; manually judges and marks the results of the image rotation correction, and there are three results: completely correct, slightly deviated, and severely deviated; obtains the final statistical results, including the total number of frames, the number of frames with corrected angles, the number of frames with slightly deviated angles, and the number of frames with severely deviated angles.

[0028] As a further solution of the present invention: when verifying the processing speed of the CNN model, the processing time of each frame is obtained and saved, and then the average processing time of each frame is calculated.

[0029] As a further solution of the present invention: when verifying the CNN model, a desktop environment is used for simulation verification, configured with an Intel core-i5-7300HQ CPU with 8GB of memory, a WIN10 operating system, an NVIDIA-GT1050 4G graphics card, the Python programming language, and the TensorFlow2.0 deep learning framework; an embedded platform is used to further evaluate the application performance, and the selected embedded platform environment is the Jetson Nano embedded deep learning board.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This convolutional neural network (CNN)-based system provides guidance or yaw correction for field drones or autonomous machines along crop rows. The final trained convolutional neural network model achieved an average accuracy of 98.74%, an average probability of deviation from the correct answer of 1.18%, and an average error rate of 0.07%. This system provides real-time, continuous, and acceptable directional guidance along crop rows with high accuracy. It can be used stably and independently in unmanned intelligent operation platforms for farmland, or in real-time auxiliary navigation mode when the primary navigation method fails unexpectedly. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The flowchart of a CNN-based method for real-time acquisition of crop row direction for assisted navigation is shown.

[0033] Figure 2 Schematic diagram of continuously intercepting three overlapping image regions from the left, center, and right regions of the extracted image.

[0034] Figure 3 Schematic diagram of the angle α between the central ridge direction and the horizontal baseline;

[0035] Figure 4This is a structural diagram of model C. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0037] The specific implementation of the present invention is described in detail below with reference to specific embodiments.

[0038] like Figure 1 、 Figure 2 and Figure 3 As shown, an embodiment of the present invention provides a method for real-time acquisition of crop row direction for assisted navigation based on CNN, the method comprising the following steps:

[0039] S100, a video file is acquired, and the image is rotated by a certain angle according to the angle α between the central ridge direction of the image and the horizontal baseline in the video file to obtain an image dataset containing 180 categories;

[0040] S200, screening an image dataset containing 180 categories to obtain an image dataset containing 60 categories, and determining a training set, a validation set, and a test set;

[0041] S300, determining convolution control parameters, activation function, loss function, optimizer and CNN model structure, and building a CNN model;

[0042] S400, verifies the accuracy of the CNN model and verifies the processing speed of the CNN model.

[0043] In this embodiment of the present invention, a complete set of training images is required to obtain ideal classification results from the CNN model. In this embodiment of the present invention, sufficient training, validation, and test images were obtained using drone-captured videos of field crop inspections. The video images were in the following state: the crops were growing, the ground in the ridges and furrows was covered, but the images could identify the orientation of the furrows.

[0044] In an embodiment of the present invention, the step of rotating the image by a certain angle according to the angle α between the central ridge direction of the image in the video file and the horizontal baseline to obtain an image data set containing 180 categories specifically includes: extracting an image from the video file once every two frames to reduce the similarity between training images collected at the same position, continuously intercepting three overlapping image areas (with a size of 512×512) from the left, middle and right areas of the extracted image, and saving the overlapping image areas in the image set A folder as the initial image file; all images in the initial image file are displayed in the processing interface, and the operator uses the mouse to start from the center position of the image (automatic positioning) and extend along the central ridge to the top area of the image, and determine the current image by the generated line segment The angle α between the central ridge direction in the image and the middle baseline of the image is calculated in real time. That is, the angle α between the central ridge direction and the horizontal baseline of each image in the image set A folder is determined, and the angle α is appended to the image file name. Then, a fixed area in the center of the image is captured, and the central ridge direction is used as the cropping line to obtain the image set B folder; based on the angle α, the image is rotated by a certain angle to generate an image with a cropping line in the range of 0° to 179° relative to the horizontal baseline. The central area of the image is intercepted every time it is rotated 1 degree, and the image size is 256×256; the current rotation angle is added to name each new image file, and finally 180 angles are rotated, resulting in 180 folders. The formed image dataset contains 180 categories.

[0045] It should be noted that the angle between the cropping line and the horizontal baseline in the original image is α, so to generate an image corresponding to the tilt angle θ, it should be rotated by an angle of (θ-α). When θ-α>0°, it should be rotated counterclockwise, otherwise it should be rotated clockwise to obtain the correct result image, that is, the angle between the cropping line and the horizontal baseline in the image is θ. In the above processing steps, the image set B is represented as: B = {img_1, img_2, ..., img_j, ..., img_N}, where N is the total number of images collected and preprocessed from the video. For the j-th image file img_j in B, the set of rotated images corresponding to all angles in the range of 0° to 179° generated based on the image is represented as: Simg j ={img j (0°), img j (1°),…,img j (k°),…,img j (179°)}, where img j (k°) is the rotation of the image img j The angle between the cropping line and the horizontal baseline in the image is k°. Therefore, the total set of images generated by all the rotation images in the range of 0° to 179° is: S = {Simg0, Simg1, ..., Simgj ,…,Simg N}.

[0046] In an embodiment of the present invention, the steps of filtering an image dataset containing 180 categories to obtain an image dataset containing 60 categories and determining a training set, validation set, and test set specifically include: filtering the image dataset containing 180 categories to obtain an image dataset containing 60 categories; manually specifying the angle of the cropped lines in each frame of a video image; and considering the errors caused by human angle perception and manual operation, the 180 categories are too detailed and likely contain errors. The accuracy provided by the training data may not be sufficient to accurately distinguish all 180 categories. Therefore, the embodiment of the present invention employs a category coarsening measure: reducing the number of categories while maintaining guaranteed navigation accuracy. Specifically, during the filtering process, every 3° is considered a category, resulting in a total of 60 categories. This means that the minimum difference between two training images from adjacent category folders is only 1°, which will reduce the convergence and accuracy of CNN classification (verified in actual experiments). Therefore, considering this trade-off, a new approach is adopted: retaining only the middle angle image of three original consecutive angle images in each category folder; and removing the left and right angle images (for example, for angles of 6°, 7°, and 8°, only the 7° training image is retained). In this way, although two-thirds of the training image data is lost due to the large number of original training images, the adequacy of the training data is not affected. In fact, it can improve the differences between categories and reduce their ambiguity, thereby improving the convergence speed and accuracy of CNN classification and achieving better actual classification results. So when CNN classifies an image whose true angle between the cropping line and the horizontal baseline is not the angle value of its category, it can be classified into any adjacent category. The selected angle categories are as follows: Class = {1°, 4°, 7°, ..., 178°}, for image img j (k°)∈Simg j , if k° mod 3° == 1, then classify the image into category set C (k-1) / 3 Otherwise, discard it. The image set of class t is: C t ={img1((3*t+1)°), img2((3*t+1)°),…,img j ((3*t+1)°),…,img N ((3*t+1)°)}.

[0047] Next, the image files in the image dataset are named according to the following naming convention: current rotation angle + crop line angle + frame number + region sequence number + video name suffix. Region numbers are 0, 1, or 2, corresponding to the three overlapping image regions captured. The image files of 60 categories are divided into a training set (1,000 images per category, totaling 60,000 images), a validation set (350 images per category, totaling 21,000 images), and a test set (155 images per category, totaling 9,300 images) in a 7:2:1 ratio and placed in the main folder. Unlike other CNN recognition and classification applications, the original training images in this application cannot be enhanced by stretching, scaling, or rotating, as this would destroy the original angular relationship of the target scene.

[0048] It should be noted that a CNN is a deep neural network with a convolutional architecture. By combining local receptive fields, shared weights, and spatial or temporal downsampling, it fully utilizes local features contained in image data, effectively reducing the number of deep network parameters and alleviating model overfitting. CNN classification models differ from traditional models in that they consider the 2D structure of images when processing them. They can directly use 2D images as input without complex preprocessing. This is equivalent to performing complex feature extraction, performing pattern classification and other tasks in a black box (i.e., a multilayer perceptron) to continuously optimize the desired network parameters. Ideal classification results are achieved in the network's output layer. The core elements of applying CNNs to problems are the network's structure and solution. Cellular neural networks offer advantages over traditional techniques: good fault tolerance, parallel processing capabilities, and self-learning capabilities. They can handle classification problems with complex environmental information, unclear background knowledge, or unclear inference rules. They are adaptable to a wide range of samples and have faster processing speeds. The construction and implementation of convolutional neural networks are typically based on deep learning neural network frameworks. Popular frameworks include Caffe, Theano, MXNet, TensorFlow 1, TensorFlow 2, Keras, Torch, etc. Due to its construction efficiency, flexibility, and ease of use, Python-based TensorFlow 2.0 was selected as the CNN development framework for the embodiments of the present invention.

[0049] Furthermore, it is necessary to determine the convolution control parameters, including: setting the step size of the image convolution operation to 1; for padding, in order to slide the sliding window to the boundary, the "(input_width+padding-kernel_width+step) / depth" value is an integer. In an embodiment of the present invention, the edge length of the convolution kernel is reduced by 1 according to the boundary value of the conv2d convolution function in Keras. The excitation layer makes useful features more obvious and weakens useless features. The method is to perform nonlinear mapping on the output results of the convolution layer, that is, "activation". The excitation function adopts the ReLU (Rectified Linear Unit, f(x) = max(0, x)) function with a single threshold, simple calculation and fast convergence, and has a wider excitation boundary and sparse activation.

[0050] Determine the activation function: The main function of the activation function in CNN is to provide nonlinear modeling, so that the network can approximate the nonlinear functions of many nonlinear models and can better solve more complex problems. In neural networks, ReLU, tanh or sigmoid functions are usually used as activation functions in hidden layers, while sigmoid or softmax classifiers are used as activation functions in the fully connected output layer. For the activation function used in the hidden layer of the network, the sigmoid function is defined as S(x) = 1 / (1+e -x ). Its main disadvantage is that its soft saturation can easily lead to the disappearance of its gradient. The tanh function is defined as tanh(x) = (1-e -2x ) / (1+e -2x ), which converges faster than the sigmoid function, but also has soft saturation and cannot solve the vanishing gradient problem. ReLU is defined as f(x) = max(x, 0) and is a very popular activation function. When x < 0, it is saturated. Since its derivative is 1 when x > 0, ReLU can keep the gradient unchanged to alleviate the vanishing gradient problem, and it can converge more efficiently and provide sparse expression capabilities for the network. Currently, most convolutional neural networks use ReLU as their activation function. For the activation function of the one-dimensional vector output from the fully connected output layer, the sigmoid function is generally suitable for multi-label binary classification, while the softmax classifier is generally used to solve multi-category classification problems. The Softmax classifier calculates the probability of each target class among all possible target classes and converts the fully connected layer into a probability distribution whose total is always 1. The output value helps to determine the target category of a given input. The embodiment of the present invention selects ReLU as the activation function of the hidden layer of the model and selects the softmax classifier as the activation function of the output layer of the model.

[0051] Determine the loss function: The loss function calculates and measures the difference between the predicted value and the actual value of each iteration of the network to guide the next training step in the right direction. The process of neural network training or optimization is the process of minimizing the loss function. The smaller the loss function, the closer the predicted output of the model is to the actual value, and the better the accuracy of the model. Common loss functions can be roughly divided into two categories: regression loss and classification loss. The most commonly used regression loss function is the mean squared error (MSE) loss function, which is defined as the sum of the squares of the distances between the target variable and the predicted value: The cross entropy loss function is a commonly used classification loss function that measures the difference between probability distributions. It usually includes binary cross entropy loss function and multi-classification cross entropy loss function. For N independent events, the binary cross entropy loss function of two corresponding categories is as follows: Where N is the number of independent events, y i is the label of sample i, y i ∈{0, 1} and p i Is the probability that sample i is predicted to be 1. Multi-class classification is an extension of two-class classification, where each class corresponds to a probability, and the multi-class cross entropy function (categorical cross entropy) of K classes is as follows: Where K is the number of categories, y i is a binary category (1 if the current category is the same as the category of sample i; 0 otherwise), p i is the i-th value of the input probability vector. The problem studied in this paper is a multi-classification problem. For the probability results given by the output layer activation function softmax, the multi-classification cross entropy function (classification cross entropy) is used as the loss function.

[0052] Determining the optimizer: To increase the training speed of CNNs, an optimizer is required. The choice of optimizer plays a crucial role in deep learning, as it determines whether training can converge quickly and achieve high accuracy and recall. Commonly used optimizers are mainly gradient descent and adaptive learning rate algorithms. The gradient descent optimizer is one of the fundamental optimizers in deep learning. In deep network training, the initial learning rate should be specified in advance and updated based on the number of training iterations. Typically, a high learning rate is used at the beginning of training and then decreased as the number of iterations increases. One class of algorithms that can automatically adjust the learning rate is called an adaptive learning rate optimizer. Examples include Adagrad, Adadeta, RMSprop (Root Mean Square Propagation), and Adam (Adaptive Moment Estimation) optimizers. Adam dynamically adjusts the learning rate of each parameter by using the exponentially weighted average of the gradients (first-order moment estimation) and the exponentially weighted average of the squares of the gradients (second-order moment estimation). The Adam optimizer offers many advantages over other optimizers: it is easy to implement, computationally efficient, and requires less memory; it is invariant to diagonal rescaling of gradients, making it suitable for problems with large amounts of data or large numbers of parameters; it is also suitable for solving nonstationary problems with high noise and sparse gradients; and its hyperparameters are intuitively interpretable and require only minor adjustments. Empirical results have also shown that the Adam algorithm performs comparable to other stochastic optimization methods in practice, and therefore, was chosen as the CNN optimizer for this paper.

[0053] Determining the CNN Model Architecture: For image recognition based on convolutional neural networks, the targets are texture, entity, behavioral pattern, and other recognition types. The target of this embodiment is crop line angle recognition, which differs somewhat from these recognition types. To determine an appropriate CNN network hierarchy, given the relatively low complexity of the task and the expected low number of features, three CNN network models with varying depth structures were tested and exploratory evaluation was performed on the available dataset. To obtain an average performance reference and avoid the misleading influence of unexpected extreme results during training and validation, and to account for the randomness of the initial CNN parameters, each test model was trained and validated 10 times, and the average was taken. The study found that increasing the number of hidden layers from 3 to 5 resulted in improved training and validation accuracy. However, when the number of hidden layers exceeded 5, overfitting (improved training accuracy, decreased validation accuracy) or failure (consistently low accuracy) occurred. A total of 21,000 training images from 60 orientation and angle categories were used during training and validation. Results showed that the average correct recognition rates for Models A (3 hidden layers), B (4 hidden layers), and C (5 hidden layers) were 98.19%, 98.50%, and 98.74%, respectively. However, in images with incorrectly identified crop row angles, some of the incorrectly identified angles were adjacent to the correct angle. For example, the correct angle was in category 10, while the incorrect angles were in categories 9 or 11. The average recognition rates for these minor errors were 1.61%, 1.40%, and 1.18%, respectively, for Models A, B, and C. The average percentages of severely incorrectly identified angles for the remaining non-adjacent angles were 0.2%, 0.1%, and 0.07%, respectively. These results are suitable for real-world low-speed field crop row navigation in crop areas captured in video. Model D's architecture is a general configuration with more than 5 hidden layers. Various specific configurations were attempted for these architectures, including varying and adjusting the insertion position and dropout rate, adjusting the number and position of maxpools, varying the number of filters in each layer, and varying the batch size, but none of them converged after 60 iterations. The application was also tested on a VGG16 network model, but it also failed to converge after 60 iterations. The target objects in this study, namely crop rows in fields, often represent textures in two-dimensional images. Texture features are less complex than those of other image objects, so background influences do not hinder classification. For further research and practical application, more crop row data from different seasons, locations, and crops will be collected for model training.

[0054] like Figure 4As shown in Figure 2, for more diverse and richer image data, a CNN network structure with five hidden layers, namely, model C, will be used to identify crop row angles because it can handle more complex situations based on deeper hidden layers.

[0055] In the embodiment of the present invention, since the final application verification of the trained CNN network model is to use a real farmland intelligent operation platform for crop navigation, this verification needs to include the influence of the platform's control ability and accuracy. If only the software algorithm is verified, it is necessary to eliminate the possible influence of the platform control system. The method used in the embodiment of the present invention is a video-based simulation. The specific method of verifying the actual path correction effect is to rotate the video frame image so that the ridge direction in the middle of the image is parallel to the y-axis direction of the screen. When verifying the CNN model, a desktop environment was used for simulation verification, configured with an Intel core-i5-7300HQ CPU with 8GB of memory, a WIN10 operating system, an NVIDIA-GT1050 4G graphics card, a Python programming language, and a TensorFlow2.0 deep framework; an embedded platform was used to further evaluate the application performance. The selected embedded platform environment was the Jetson Nano embedded deep learning board (4G-B01, ARM-A57@1.43 GHz, 128-core NVIDIA-Maxwell@921MHz).

[0056] In an embodiment of the present invention, the step of verifying the accuracy of the CNN model is specifically as follows: for a test video, each time a scene image is obtained, the CNN network processes the scene image, gives the deviation angle of the cropping line in the scene image from the vertical direction of the image center, and then corrects the image rotation according to the deviation angle and displays it; manually judges and marks the results of the image rotation correction, and the results have three situations: completely correct, slightly deviated, and severely deviated; obtains the final statistical results, including the total number of frames, the number of frames with corrected angles, the number of frames with slightly deviated angles, and the number of frames with severely deviated angles. In addition, when verifying the processing speed of the CNN model, the processing time of each frame is obtained and saved, and then the average processing time of each frame is calculated.

[0057] In terms of accuracy, since the preset angle division value is 3°, 0°~179° is divided into 60 categories, and each category is represented by its central angle, the actual deviation angle is a multiple of 3 during correction and alignment operations. This makes it easy for operators to distinguish angles and ensures that the error rate of manual recognition is less than 0.1%. Specifically, five operators tested the algorithm using the same operating procedure, and occasional recognition errors mainly occurred in correctness detection and adjacent deviations, but the error rate was very small. Verification experiments show that this method can be used as an acceptable navigation scheme with rough accuracy and no auxiliary signals, and can be used stably and independently for unmanned intelligent operation platforms in farmland, or in real-time auxiliary navigation mode when the main navigation method fails unexpectedly (such as navigation signal interruption).

[0058] The above is only a detailed description of the preferred embodiments of the present invention, which is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0059] It should be understood that, although the various steps in the flow chart of each embodiment of the present invention are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence according to the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0060] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0061] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the disclosure in the specification and examples. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered merely as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A real-time acquisition method of crop row direction for assisted navigation based on CNN, characterized in that: The method comprises the following steps: The video file is collected and the image is rotated by a certain angle according to the angle α between the central ridge direction of the image and the horizontal baseline in the video file to obtain an image dataset containing 180 categories; The image dataset containing 180 categories was screened to obtain an image dataset containing 60 categories, and the training set, validation set, and test set were determined; Determine the convolution control parameters, activation function, loss function, optimizer and CNN model structure, and build the CNN model; Verifying the accuracy of the CNN model and the processing speed of the CNN model, the step of rotating the image by a certain angle according to the angle α between the central ridge direction of the image and the horizontal baseline in the video file to obtain an image dataset containing 180 categories specifically includes: Extract an image from the video file every two frames, continuously cut out three overlapping image regions from the left, middle, and right regions of the extracted image, and save the overlapping image regions into the image set A folder as the initial image file; Determine the angle α between the central ridge direction and the horizontal baseline of each image in the image set A folder, append the angle α to the image file name, capture a fixed area in the center of the image, and use the central ridge direction as the cropping line to obtain the image set B folder; The image is rotated by a certain angle based on the angle α to generate an image with a cropping line ranging from 0° to 179° relative to the horizontal baseline. The central area of the image is cropped once every 1 degree of rotation. Add the current rotation angle to name each new image file, and finally rotate to 180 angles, generate 180 folders, and form an image dataset containing 180 categories. The image dataset containing 180 categories is filtered to obtain an image dataset containing 60 categories, and the steps of determining the training set, validation set, and test set specifically include: The image dataset containing 180 categories was screened to obtain an image dataset containing 60 categories. During the screening, every 3° was considered as a category. The image file names in the image dataset are named according to the following rules: current rotation angle + cropping line angle + frame number + region sequence number + video name suffix. The region number is 0, 1, or 2, corresponding to the three overlapping image regions captured. The image files of 60 categories are divided into a training set, a validation set, and a test set in a ratio of 7:2:1, and placed in a main folder. The step of determining the convolution control parameters specifically includes: Set the step size of the image convolution operation to 1; According to the boundary value of the conv2d convolution function in Keras, the edge length of the convolution kernel is reduced by 1; The output of the convolutional layer is nonlinearly mapped, and the activation function uses a single threshold and fast-converging ReLU (Rectified Linear Unit, )function.

2. The method for real-time acquisition of crop row direction for assisted navigation based on CNN according to claim 1, characterized in that: Select ReLU as the activation function of the model hidden layer, select softmax classifier as the activation function of the model output layer; use multi-class cross entropy function as the loss function; select Adam algorithm as the CNN network optimizer.

3. The method for real-time acquisition of crop row direction for assisted navigation based on CNN according to claim 1, characterized in that: The CNN model structure is a CNN network structure with five hidden layers, which is used to identify crop row angles.

4. The method for real-time acquisition of crop row direction for assisted navigation based on CNN according to claim 1, characterized in that: The steps for verifying the accuracy of the CNN model are specifically as follows: for the test video, each time a scene image is obtained, the CNN network processes the scene image, gives the deviation angle between the cropping line in the scene image and the vertical direction of the image center, and then corrects the image rotation according to the deviation angle and displays it; manually judges and marks the results of the image rotation correction, and there are three results: completely correct, slightly deviated, and severely deviated; and obtains the final statistical results, including the total number of frames, the number of frames with corrected angles, the number of frames with slightly deviated angles, and the number of frames with severely deviated angles.

5. The method for real-time acquisition of crop row direction for assisted navigation based on CNN according to claim 1, characterized in that: When verifying the processing speed of the CNN model, the processing time of each frame was obtained and saved, and then the average processing time of each frame was calculated.

6. The method for real-time acquisition of crop row direction for assisted navigation based on CNN according to claim 1, characterized in that: When verifying the CNN model, a desktop environment was used for simulation verification based on the Python programming language and the TensorFlow2.0 deep learning framework; an embedded platform was used to further evaluate the application performance. The selected embedded platform environment was the Jetson Nano embedded deep learning board.

Images