Agricultural navigation line extraction method of convolutional neural network based on anchor points

Abstract

The invention provides an agricultural navigation line extraction method of a convolutional neural network based on anchor points. According to the method, the position of a navigation line at each anchor point in a picture is directly found by using an anchor point-based convolutional neural network, and then the direction and the position of the navigation line are fitted by using a least square method so as to guide an agricultural machine to adjust the attitude and the direction. As the anchor point position is preset, the network is required to predict the position of the navigation line at the anchor point, so that the method is not influenced by shielding, crop row interruption and the like. Meanwhile, according to the method, crop types are not set, and the method depends on high similarity of agricultural crop planting modes, so that the method can directly cope with various different crops and environments.

Description

Technical Field

[0001] The present invention relates to the technical fields of image processing, deep neural network learning, and convolutional neural network, and in particular to an agricultural navigation line extraction method based on an anchor point convolutional neural network. Background Art

[0002] Image processing technology uses computers to process acquired images. It primarily includes image segmentation, image recognition, image enhancement, and restoration. Image processing technology can give machinery "eyes," acquiring crucial visual information, enabling more convenient and accurate control. Image processing technology is primarily used to extract valuable information from image data. Common image processing steps include image preprocessing, feature extraction, image segmentation, and edge detection. In agricultural automation, image processing is often used to obtain environmental information and assist automated equipment in making decisions. For example, image preprocessing techniques (such as denoising and grayscale conversion) can reduce interference in images, thereby improving the accuracy of subsequent analysis. In field environments, image processing technology is often applied to tasks such as crop identification, soil analysis, and inter-row guidance line extraction. Inter-row guidance line extraction is a key technology in precision agriculture, enabling agricultural machinery to automatically navigate between crop rows.

[0003] Deep learning is a machine learning method based on neural networks that processes complex data by simulating the workings of neurons in the human brain. Deep learning can automatically learn features from large amounts of data and has achieved remarkable results in fields such as image recognition and speech recognition. In agriculture, deep learning is widely used in precision agriculture, such as crop disease detection, fruit maturity assessment, and soil composition analysis. The advantage of deep learning technology lies in its ability to automatically extract complex features from images using large amounts of training data, avoiding the complexity of manual feature extraction. Therefore, in the task of extracting field navigation lines, deep learning can effectively address the challenges posed by factors such as varying lighting, weather, and soil types.

[0004] A convolutional neural network (CNN) is a neural network model specifically designed for image processing in deep learning. Through multi-layer convolution operations, CNNs can extract local features of an image, reduce computational complexity through pooling layers, and complete the final classification or regression task through fully connected layers.

[0005] A key feature of CNNs is their ability to automatically extract features from raw images without the need for manually designed feature extraction algorithms. This is particularly important in the complex environments of agriculture, where factors such as background, lighting, and crop morphology can vary. CNNs enable robust recognition of these variations. For the task of extracting field navigation lines, CNNs can be trained to learn the "inter-row" features in images and automatically identify navigation lines between crop rows in the field. Especially in complex field environments, CNNs can effectively identify irregular inter-row lines and potential interference objects (such as weeds and soil cracks). Summary of the Invention

[0006] This invention addresses the need to accurately locate the crop in the center of an image captured by agricultural machinery to help guide the machinery in adjusting its posture and direction when multiple ridges of crops are visible in the image ahead. However, in complex environments, such as those with abundant weeds, there is no effective strategy to simply and effectively eliminate the influence of weeds, accurately locate the crop in the center, and extract a navigation line based on this. Therefore, this invention proposes a method for extracting agricultural navigation lines using a convolutional neural network based on anchor points. This method enables a robot to autonomously fit a navigation line based on the position of the seedling line ahead in a field ridge, without requiring human intervention, thereby helping the machinery to navigate along the seedling line.

[0007] The present invention is achieved through the following technical solutions. The present invention proposes an agricultural navigation line extraction method based on a convolutional neural network of anchor points, and the method comprises the following steps:

[0008] Step 1: Use a camera to capture the area in front of the agricultural machinery and obtain crop location information in the field ahead;

[0009] Step 2: Use the anchor-based convolutional neural network to process the image and output the position of the navigation line corresponding to each anchor point;

[0010] Step 3. Finally, based on the positions of all anchor points, use the least squares method to extract the position, deflection angle, and offset of the navigation line to help the robot adjust its walking posture and direction in the field.

[0011] Furthermore, the anchor-based convolutional neural network is used to process images and obtain the coordinate position of the navigation line at each anchor point; the convolutional neural network includes three modules: a main framework module, a fully connected layer module, and a distributed focus module; the main framework module includes a convolution module, a residual module, a pyramid pooling module, and a residual block; the main framework module input is a 256×256×3 color image, which first undergoes a convolution operation with parameters k=7, s=2, p=3, and c=8; then undergoes a maximum pooling operation using the function nn.maxpool() with parameters k=3, s=2, p=1; then passes through a residual block, which is not used for downsampling, and its parameters are downsampling=Fasle, c=16; passes through a second residual block, which is used for downsampling, and its parameters are downsampling=True, c=24; passes through a third residual block, which is used for downsampling, and its parameters are downsampling=True, c=36; passes through a fourth residual block, which is used for downsampling, and its parameters are downsampling=True, c=48; the final output is a 48×8×8 tensor.

[0012] Furthermore, the fully connected layer module receives the output of the main frame module, first flattens the 48×8×8 tensor into 1×3072, passes it through the first fully connected layer, uses the function nn.Linear(), outputs the intermediate layer size of 1×4096, and then passes it through the second fully connected layer to output a one-dimensional vector of size 1×(N×256), where N represents the number of set anchor points.

[0013] Furthermore, the distribution focus module receives the output of the fully connected layer module, first transforms the 1×(N×256) vector into a two-dimensional vector of N×256, i.e., N rows and 256 columns, where each row represents an anchor point; then, the function nn.softmax() is used to perform an operation on each row to obtain the score of the 256 values ​​of each row; finally, formula (1) is used to perform a weighted average on each row to obtain the horizontal position of the navigation line corresponding to each anchor point in the image; where score(i) represents the score of the i-th value;

[0014]

[0015] Furthermore, the convolution module consists of a convolution operation Conv2d, a normalization operation BatchNorm2d, and an activation operation GELU; the convolution operation uses the function nn.Conv2d(), where k, s, p, and c represent the parameters in the function: convolution kernel size, step size, padding, and number of output channels, respectively; the normalization operation selects the function nn.BatchNorm2d(); and the activation operation selects the function nn.GELU().

[0016] Furthermore, the residual module includes two branches; the left branch first passes through a convolution module with parameters set to k=3, s=1, p=1, and then passes through a convolution module with parameters set to k=3, s=x, p=1; the right branch passes through a convolution module with parameters set to k=1, s=x; if the residual module is used for downsampling, that is, downsample=True, then the above x=2; otherwise, downsample=False, x=1; finally, the results of the two branches are added to obtain the output result of the residual module.

[0017] Furthermore, the pyramid pooling module is specifically as follows: first, through four branches, each branch performs a convolution operation, the parameters of the first branch are set to k=3, s=1, p=4, d=2; the parameters of the second branch are set to k=3, s=1, p=6, d=4; the parameters of the third branch are set to k=3, s=1, p=8, d=6; the parameters of the fourth branch are set to k=3, s=1, p=10, d=8; finally, the results of the four branches are spliced ​​using the function nn.concat(), and then undergo a convolution operation with its parameters set to k=1, s=1 to obtain the output of the pyramid pooling module.

[0018] Furthermore, the residual block consists of two residual modules and a pyramid pooling module; if the residual block is used for downsampling, the parameter downsample of the first residual module is True, otherwise it is False; then it passes through a pyramid pooling module; the parameter downsample of the second residual module is always False.

[0019] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the agricultural navigation line extraction method based on the anchor point convolutional neural network are implemented.

[0020] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the agricultural navigation line extraction method based on the anchor point convolutional neural network.

[0021] Beneficial effects of the present invention:

[0022] The method described in this paper uses an anchor-based convolutional neural network to directly locate the position of the navigation line at each anchor point in the image. The method then uses the least squares method to fit the direction and position of the navigation line, which is used to guide the agricultural machinery to adjust its posture and direction. Because the anchor point positions are preset, the network is required to predict the position of the navigation line at the anchor point. This method is not affected by occlusion, crop row interruptions, etc. Furthermore, the method does not specify the crop type, relying on the high similarity of agricultural crop planting patterns. Therefore, it can directly adapt to a variety of crops and environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0024] Figure 1 This is the structure diagram of the convolution module;

[0025] Figure 2 It is the structure diagram of the residual module;

[0026] Figure 3 This is the structure diagram of the pyramid pooling module;

[0027] Figure 4 is the residual block structure diagram;

[0028] Figure 5 This is the main frame module structure diagram;

[0029] Figure 6 This is the module structure diagram of the fully connected layer;

[0030] Figure 7 This is the structural diagram of the distribution focus module;

[0031] Figure 8 This is the structure diagram of the anchor-based convolutional neural network;

[0032] Figure 9 The network output effect diagram; among them, (a) is the navigation line position prediction result in various environments, and (b) is the navigation line position prediction result in the case of occlusion and crop row interruption. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0034] Combine Figures 1-9 The present invention proposes an agricultural navigation line extraction method based on a convolutional neural network of anchor points, and the method comprises the following steps:

[0035] Step 1: Use a camera to capture the area in front of the agricultural machinery and obtain crop location information in the field ahead;

[0036] Step 2: Use the anchor-based convolutional neural network to process the image and output the position of the navigation line corresponding to each anchor point;

[0037] Step 3. Finally, based on the positions of all anchor points, use the least squares method to extract the position, deflection angle, and offset of the navigation line to help the robot adjust its walking posture and direction in the field.

[0038] The anchor-based convolutional neural network is used to process images and obtain the coordinate position of the navigation line at each anchor point. The convolutional neural network includes three modules: the main framework module, the fully connected layer module, and the distribution focus module. The main framework module includes a convolution module (ConvModule), a residual module (ResModule), a pyramid pooling module (ASPPModule), and a residual block (Resbottle). The main framework module input is a 256×256×3 color image, such as Figure 5 As shown, first, it passes through a convolution operation with parameters k=7, s=2, p=3, and c=8; then it passes through a maximum pooling operation (Maxpooling), which uses the function nn.maxpool() with parameters k=3, s=2, and p=1; then it passes through a residual block, which is not used for downsampling and has parameters downsampling=Fasle and c=16; it passes through a second residual block, which is used for downsampling and has parameters downsampling=True and c=24; it passes through a third residual block, which is used for downsampling and has parameters downsampling=True and c=36; it passes through a fourth residual block, which is used for downsampling and has parameters downsampling=True and c=48; the final output is a 48×8×8 tensor.

[0039] like Figure 6As shown, the fully connected layer module receives the output of the main frame module, first flattens the 48×8×8 tensor into 1×3072, passes through the first fully connected layer, uses the function nn.Linear(), outputs the intermediate layer size of 1×4096, and then passes through the second fully connected layer to output a one-dimensional vector of size 1×(N×256), where N represents the number of set anchor points.

[0040] like Figure 7 As shown, the distribution focus module receives the output of the fully connected layer module, first transforms the 1×(N×256) vector into a two-dimensional vector of N×256, that is, N rows and 256 columns, where each row represents an anchor point; then uses the function nn.softmax() to perform an operation on each row to obtain the score of 256 values ​​of each row; finally, uses formula (1) to perform a weighted average on each row, and then obtains the horizontal position of the navigation line corresponding to each anchor point in the image; where score(i) represents the score of the i-th value;

[0041]

[0042] like Figure 1 As shown in the figure, the convolution module consists of a convolution operation Conv2d, a normalization operation BatchNorm2d, and an activation operation GELU; the convolution operation uses the function nn.Conv2d(), where k, s, p, and c represent the parameters in the function: convolution kernel size, step size, padding, and number of output channels; the normalization operation selects the function nn.BatchNorm2d(); the activation operation selects the function nn.GELU().

[0043] like Figure 2 As shown, the residual module includes two branches; the left branch first passes through a convolution module with parameters set to k=3, s=1, p=1, and then passes through a convolution module with parameters set to k=3, s=x, p=1; the right branch passes through a convolution module with parameters set to k=1, s=x; if the residual module is used for downsampling, that is, downsample=True, then the above x=2; otherwise, downsample=False, x=1; finally, the results of the two branches are added to obtain the output result of the residual module.

[0044] like Figure 3As shown, the pyramid pooling module is specifically as follows: first, through four branches, each branch performs a convolution operation, the parameters of the first branch are set to k=3, s=1, p=4, d=2; the parameters of the second branch are set to k=3, s=1, p=6, d=4; the parameters of the third branch are set to k=3, s=1, p=8, d=6; the parameters of the fourth branch are set to k=3, s=1, p=10, d=8; finally, the results of the four branches are spliced ​​using the function nn.concat(), and then undergo a convolution operation with its parameters set to k=1, s=1 to obtain the output of the pyramid pooling module.

[0045] like Figure 4 As shown, the residual block consists of two residual modules and a pyramid pooling module; if the residual block is used for downsampling, the parameter downsample of the first residual module is True, otherwise it is False; then it passes through a pyramid pooling module; the parameter downsample of the second residual module is always False.

[0046] The above is the anchor-based convolutional neural network, and its overall network framework structure is as follows: Figure 8 shown. Figure 9 The network output effect diagram; among them, (a) is the navigation line position prediction result in various environments, and (b) is the navigation line position prediction result in the case of occlusion and crop row interruption.

[0047] Example

[0048] The present invention proposes an agricultural navigation line extraction method based on an anchor point convolutional neural network. The method comprises two parts: an anchor point-based convolutional neural network and a straight line fitting algorithm.

[0049] The anchor-based convolutional neural network is as follows Figure 8 As shown, it is mainly used to directly find the location of the navigation line at each anchor point in the image. The convolutional neural network is mainly implemented by the following steps:

[0050] Step 1: Set the input image size to 256×256×3, which represent the length, width, and number of channels respectively. Set the number of anchor points to N and divide the image into N rows. Then the vertical position of the navigation line corresponding to the i-th anchor point in the image can be calculated using formula (2).

[0051]

[0052] Step 2: After the input image passes through the main frame, a 48×8×8 tensor output is obtained;

[0053] Step 3: The 48×8×8 tensor output obtained by the main framework is passed through the fully connected layer module to obtain a one-dimensional vector of 1×(N×256);

[0054] Step 4: The one-dimensional vector of 1×(N×256) obtained by the fully connected layer module is passed through the distributed focus module to obtain the horizontal positions of the navigation lines at the N anchor points in the image. These correspond one-to-one with the vertical positions mentioned above, and N point coordinates can be obtained, where the vertical position is the horizontal coordinate and the horizontal position is the vertical coordinate.

[0055] The linear fitting part uses the least square method. By using the above N coordinate points, the direction and position of the navigation line are calculated by fitting.

[0056] The present invention also proposes an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the agricultural navigation line extraction method based on the anchor point convolutional neural network are implemented.

[0057] The present invention also proposes a computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the steps of the agricultural navigation line extraction method based on the anchor point convolutional neural network.

[0058] The memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DRRAM). It should be noted that the memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0059] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disc (SSD)).

[0060] During implementation, each step of the above method can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0061] It should be noted that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by an integrated logic circuit of the hardware in the processor or by instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0062] The above is a detailed introduction to the agricultural navigation line extraction method based on anchor point convolutional neural network proposed in the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. An agricultural navigation line extraction method based on anchor point convolutional neural network, characterized in that: The method comprises the following steps: Step 1: Use a camera to capture the area in front of the agricultural machinery to obtain crop location information in the field in front; Step 2: Use the anchor-based convolutional neural network to process the image and output the position of the navigation line corresponding to each anchor point; Step 3. Finally, based on the positions of all anchor points, use the least squares method to extract the position, deflection angle and offset of the navigation line to help the robot adjust its walking posture and direction in the field.

2. The method according to claim 1, characterized in that The anchor-based convolutional neural network is used to process images and obtain the coordinate positions of navigation lines at each anchor point; the convolutional neural network includes three modules: a main framework module, a fully connected layer module, and a distributed focus module; the main framework module includes a convolution module, a residual module, a pyramid pooling module, and a residual block; the main framework module input is a 256×256×3 color image, which first undergoes a convolution operation with parameters k=7, s=2, p=3, and c=8; then undergoes a maximum pooling operation, which uses the function nn.maxpool() with parameters k=3, s= 2, p = 1; then it passes through a residual block, which is not used for downsampling, and its parameters are downsampling = Fasle, c = 16; it passes through a second residual block, which is used for downsampling, and its parameters are downsampling = True, c = 24; it passes through a third residual block, which is used for downsampling, and its parameters are downsampling = True, c = 36; it passes through a fourth residual block, which is used for downsampling, and its parameters are downsampling = True, c = 48; the final output is a 48×8×8 tensor.

3. The method according to claim 2, characterized in that The fully connected layer module receives the output of the main frame module, first flattens the 48×8×8 tensor into 1×3072, passes through the first fully connected layer, uses the function nn.Linear(), outputs the intermediate layer size of 1×4096, and then passes through the second fully connected layer to output a one-dimensional vector of size 1×(N×256), where N represents the number of set anchor points.

4. The method according to claim 2, characterized in that: The distribution focus module receives the output of the fully connected layer module, and first transforms the 1×(N×256) vector into a two-dimensional vector of N×256, i.e., N rows and 256 columns, where each row represents an anchor point; then, the function nn.softmax() is used to perform an operation on each row to obtain the score of 256 values ​​of each row; finally, formula (1) is used to perform a weighted average on each row to obtain the horizontal position of the navigation line corresponding to each anchor point in the image; wherein score(i) represents the score of the i-th value; 5. The method according to claim 2, characterized in that: The convolution module consists of a convolution operation Conv2d, a normalization operation BatchNorm2d and an activation operation GELU; the convolution operation uses the function nn.Conv2d(), where k, s, p, and c represent the parameters in the function: convolution kernel size, step size, padding, and number of output channels; the normalization operation selects the function nn.BatchNorm2d(); and the activation operation selects the function nn.GELU().

6. The method according to claim 2, characterized in that The residual module includes two branches; the left branch first passes through a convolution module, setting parameters to k=3, s=1, p=1, and then passes through a convolution module, setting parameters to k=3, s=x, p=1; the right branch passes through a convolution module, setting parameters to k=1, s=x; if the residual module is used for downsampling, that is, downsample=True, then the above x=2; Otherwise, downsample=False, x=1; finally, the two branch results are added to obtain the output result of the residual module.

7. The method according to claim 2, characterized in that The pyramid pooling module is specifically as follows: first, through four branches, each branch performs a convolution operation, the parameters of the first branch are set to k=3, s=1, p=4, d=2; the parameters of the second branch are set to k=3, s=1, p=6, d=4; the parameters of the third branch are set to k=3, s=1, p=8, d=6; the parameters of the fourth branch are set to k=3, s=1, p=10, d=8; finally, the results of the four branches are spliced ​​using the function nn.concat(), and then a convolution operation is performed, and its parameters are set to k=1, s=1, to obtain the output of the pyramid pooling module.

8. The method according to claim 2, characterized in that: The residual block consists of two residual modules and a pyramid pooling module; if the residual block is used for downsampling, the parameter downsample of the first residual module is True, otherwise it is False; then it passes through a pyramid pooling module; the parameter downsample of the second residual module is always False.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium for storing computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images