Crop planting row detection method, device, system and readable storage medium
Through convolutional neural network and bounding box confidence calculation, the problem of low detection accuracy of crop planting rows in the prior art is solved. Especially in high-density weed environment, higher detection accuracy and recognition efficiency are achieved, and suitable for complex crops. Such as the intelligent weeding equipment of rice.
Patent Information
- Application Number
- CN202210809926.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-11
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2042-07-11
AI Technical Summary
The existing crop planting detection methods have poor detection effects in high-density weed environments, especially when the straightness of the planting is poor, it is difficult to accurately identify, resulting in mechanical seedling injuries and other problems.
Convolutional neural network is used to perform downsampling and bounding box confidence calculation, and the confidence of the feature map is obtained through the estimated value and actual value of the horizontal offset of adjacent plants, weed interference is eliminated, and the bounding box with the greatest confidence is selected. Its location information is reverse learning and fitting the planting row.
It improves the accuracy of crop planting row detection, reduces the impact of inter-row and inter-plant weeds on detection anchor frames, is suitable for complex environments such as rice, and supports the research and development of intelligent weeding and harvesting equipment.
Smart Images

Figure CN115049932B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of agricultural automation technology, and in particular to a crop planting row detection method, equipment, system and readable storage medium. Background Art
[0002] For a long time, chemical weed control has been widely used due to its advantages such as low cost and convenient spraying. However, the long-term reliance on chemical agents for weed control has caused many disadvantages - increased weed resistance, increasing populations year by year, and excessive pesticide residues in crops, which have seriously affected crop yields and the ecological environment of the planting areas.
[0003] In contrast, intertillage weeding offers significant weed removal effectiveness and environmental friendliness. With the increasing mechanization of agriculture, weed control efficiency has significantly improved, making it a promising alternative to chemical weed control. However, existing weed control equipment relies heavily on manual labor for row-to-row weeding, and row-to-row accuracy and weed control effectiveness often depend on the operator's skill level, leading to frequent mechanical damage to seedlings. The few methods that can automatically detect crop rows also suffer from poor detection effectiveness in dense weed environments, and identification results are easily affected by the linearity of the row arrangement.
[0004] Currently, crop row detection methods based on image processing have the advantages of rich information elements, low-cost equipment, and a simple testing process. They have been successfully applied to the detection of crop rows in crops such as corn, sugar beets, and sunflowers. These methods mainly extract and identify features based on the color difference between the crop and the soil background and the relatively neat arrangement of the rows. Improved algorithms based on the super-green pixel method and the Hough transform have emerged. These algorithms are relatively effective only for detecting rows in environments with significant color differences and sparse weeds. Among crop row detection methods based on RGB or multispectral image data, researchers have extracted and analyzed the rows of crops such as corn, wheat, and soybeans. The filter analysis method used is not easily affected by inter-row weeds. By utilizing the color difference or height difference between the crop and the soil, this method can achieve a crop row detection accuracy of 90%. However, the filter analysis method must meet the basic assumption that the crop rows exhibit specific geometric features or patterns. When the distribution pattern of the rows is not obvious or the spacing between rows varies greatly, the detection anchor frame will show significant errors. In addition, by analyzing the connected areas of the image, pixel patches or the positions of the vanishing points on the image, the task of extracting planting rows in plots with sparse weeds can be completed to a certain extent. However, when faced with plots with continuous or high-density weeds, the relevant methods have large errors and are difficult to deal with situations where the planting rows have a certain curvature or poor straightness.
[0005] In summary, existing crop row detection methods mainly rely on significant trait information such as crop color and straight row arrangement. However, field crops such as rice often have complex field environments, poor row straightness, and strong interference from associated weeds with similar morphology. The row detection effect based on the above algorithm is poor. If applied to related agricultural machinery and equipment, it will seriously affect the algorithm's accurate judgment of the row boundary position, resulting in phenomena such as mechanical seedling crushing and seedling injury. Summary of the Invention
[0006] The purpose of the present invention is to provide a method for detecting crop planting rows, which solves the problems of low detection accuracy of existing crop planting row detection methods.
[0007] In order to achieve the above object, the present invention provides a method for detecting crop planting rows, comprising:
[0008] The camera travels at a constant speed on the ridge to capture the original image of the crop planting row;
[0009] Input the original image into a convolutional neural network for downsampling to obtain m×m feature maps of equal size, where m is equal to the number of pixels of the original image divided by the downsampling multiple;
[0010] Obtaining a confidence level of a bounding box of the feature map based on an estimated value and an actual value of horizontal offsets of adjacent plants within a planting row;
[0011] Retaining bounding boxes in the feature map whose confidence is greater than a set threshold, and selecting the bounding box with the highest confidence and its position information from the retained bounding boxes;
[0012] The position information of the bounding box with the highest confidence is used as a learning target, and input into the convolutional neural network for reverse learning to obtain the optimal recognition box of the plant and its position information;
[0013] The planting rows are fitted using the position information of the optimal recognition frame of the plants.
[0014] Optionally, before inputting the original image into the convolutional neural network for downsampling, the method further includes:
[0015] The original image is compressed using an image compression algorithm.
[0016] Optionally, the actual value d of the horizontal offset between adjacent plants in a planting row is calculated according to the following formula: l :
[0017] d l =s×tanα;
[0018] Wherein, s is the driving distance of the camera, and α is the heading angle of the camera.
[0019] Optionally, the confidence level P of the bounding box of the feature map is calculated according to the following formula: c :
[0020]
[0021]
[0022] Where d is the estimated value of the horizontal offset of adjacent plants in a planting row, w is the preset planting row deviation threshold, and P ob Represents the detection of whether there are plants in the anchor frame. When P ob =1, it means there are plants in the detection anchor frame. ob = 0 means there is no plant in the detection anchor frame, P cl represents the probability that the plant is a crop or a weed, P real represents the area of the intersection between the pre-calibrated ground-truth box and the bounding box, P box-area Represents the area of the bounding box, t is the weight parameter, according to d l The value of is determined.
[0023] Optionally, the position information of the bounding box with the highest confidence and the position information of the optimal identification box of the plant both include the width, height and center point position.
[0024] Optionally, the position information of the bounding box with the highest confidence is used as a learning target, input into the convolutional neural network for reverse learning, and several initial recognition frames are obtained. The initial recognition frames are screened using a non-maximum suppression function to obtain the optimal recognition frame of the plant and its position information.
[0025] Optionally, a piecewise linear regression function is used to fit the planting rows based on the position information of the optimal recognition frame of the plant.
[0026] The present invention also provides a crop planting row detection device, comprising:
[0027] A camera is used to ride on the ridge and travel at a constant speed to capture original images of the crop planting rows;
[0028] An input module is used to input the original image into a convolutional neural network for downsampling to obtain m×m feature maps of equal size, where m is equal to the number of pixels of the original image divided by the downsampling multiple;
[0029] a confidence calculation module, configured to obtain the confidence of the bounding box of the feature map based on an estimated value and an actual value of horizontal offsets of adjacent plants in a planting row;
[0030] a bounding box screening module, configured to retain bounding boxes in the feature map whose confidence level is greater than a set threshold, and select the bounding box with the highest confidence level and its position information from the retained bounding boxes;
[0031] a reverse learning module, configured to take the position information of the bounding box with the highest confidence as a learning target, input it into the convolutional neural network for reverse learning, and obtain the optimal recognition box of the plant and its position information;
[0032] The fitting module is used to fit the planting rows using the position information of the optimal recognition frame of the plants.
[0033] The present invention also provides a crop planting row detection system, comprising a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the crop planting row detection method is implemented.
[0034] The present invention also provides a non-transitory computer-readable storage medium having instructions stored thereon. When the instructions are executed, the crop planting row detection method is implemented.
[0035] In the crop planting row detection method provided by the present invention, the estimated value and actual value of the horizontal offset of adjacent plants in the planting row are used to calculate the confidence of the bounding box of the feature map obtained after downsampling, thereby eliminating the interference of plants such as weeds in the detection anchor frame, improving the detection accuracy of the crop planting row, and reducing the impact of weeds between rows and between plants on the detection anchor frame. It is particularly suitable for extracting crop planting rows of rice and other crops with similar ecological characteristics and difficult to distinguish. It can serve as a reference for the "weeding and seedling avoidance" operation path, thereby providing support for the research and development of intelligent weeding and harvesting equipment for crops such as rice.
[0036] In addition, when dw≤d l When ≤d+w, P c =P ob ×P cl ×t, which is equivalent to classifying objects that meet the horizontal offset of adjacent plants in the planting row as crops and assigning high weights, which reduces the computational burden and improves the recognition efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 A flowchart of a crop row detection method provided by an embodiment of the present invention;
[0038] Figure 2 A schematic diagram of a rice planting row with a certain curvature and a symbiotic environment of rice and weeds provided by an embodiment of the present invention;
[0039] Figure 3 Schematic diagram of the optimal recognition frame of the plant identified by an embodiment of the present invention
[0040] Figure 4 A schematic diagram of a fitted planting row provided in an embodiment of the present invention;
[0041] Figure 5 A structural block diagram of the crop planting row detection device provided in this embodiment;
[0042] Wherein, the accompanying drawings are marked as follows:
[0043] 10-camera; 20-input module; 30-confidence calculation module; 40-bounding box screening module; 50-reverse learning module; 60-fitting module. DETAILED DESCRIPTION
[0044] The following is a more detailed description of the specific embodiments of the present invention with reference to schematic diagrams. The advantages and features of the present invention will become more apparent from the following description. It should be noted that the drawings are greatly simplified and not to exact scale, and are only used for the purpose of conveniently and clearly illustrating the embodiments of the present invention.
[0045] Figure 1 Flowchart of the crop planting row detection method provided in this embodiment. Figure 1 As shown, the crop row detection method provided in this embodiment includes steps S100, S200, S300, S400, S500 and S600.
[0046] Execute step S100: the camera travels at a constant speed on the ridge to capture the original image of the planting row of the crop.
[0047] Specifically, the camera is installed on a mobile device, and the lens of the camera can be set at an angle of 15° to 45° downward in the horizontal direction. The vertical height from the planting row can be about 1m. The mobile device travels at a constant speed on the ridge, and the camera starts to shoot the field crops to obtain the original image of the crop planting row.
[0048] Next, the original image may be compressed using an image compression algorithm (such as a JPEG image compression algorithm). In this embodiment, the original image is compressed to 480 dpi×480 dpi, but the present invention is not limited thereto.
[0049] Execute step S200: input the original image into a convolutional neural network for downsampling to obtain m×m feature maps of equal size, where m is equal to the number of pixels of the original image divided by the downsampling multiple.
[0050] Specifically, after the original image is input into the convolutional neural network, the convolutional neural network automatically downsamples the original image to obtain a feature map consisting of m×m equally sized grids, where m is equal to the number of pixels in the original image divided by the downsampling factor. In this embodiment, m=5, but this should not be limiting.
[0051] It is understandable that it is common knowledge to use the convolutional neural network to automatically downsample the original image to obtain the feature map, and no further details will be given here.
[0052] Executing step S300: obtaining the confidence of the bounding box of the feature map based on the estimated value and the actual value of the horizontal offset of adjacent plants in the planting row.
[0053] Specifically, the estimated value of the horizontal offset of adjacent plants in a planting row can be an empirical value. For example, in the rice transplanting process, the uneven hardness and thickness of the paddy mud bottom layer lead to a serious resistance difference between the driving wheels on both sides of the transplanter. Regardless of whether manual steering or automatic driving is used, the transplanting operation will always produce a heading deviation, causing the rice planting row to present a certain curvature. The schematic diagram of the rice and weed symbiotic environment is shown in the figure below. Figure 2 As shown, based on the curvature information of the planting row, the estimated value d=1.02m of the horizontal offset of adjacent plants in the rice planting row can be introduced, but it should not be limited to this. For other crops, the estimated value of the horizontal offset of adjacent plants in the planting row can also be other empirical values.
[0054] Furthermore, the actual value of the horizontal offset of adjacent plants in a planting row can be obtained by calculation. In this embodiment, the actual value d of the horizontal offset of adjacent plants in a planting row is calculated according to the following formula: l :
[0055] d l =s×tanα;
[0056] Wherein, s is the driving distance of the camera, and α is the heading angle of the camera.
[0057] In this embodiment, s = 1m, angle α is 45°, so d l =1.
[0058] Then, the confidence P of the bounding box of the feature map is calculated according to the following formula: c :
[0059]
[0060]
[0061] Where d is the estimated value of the horizontal offset of adjacent plants in a planting row, w is the preset planting row deviation threshold, and P ob Represents the detection of whether there are plants in the anchor frame. When P ob =1, it means there are plants in the detection anchor frame. ob = 0 means there is no plant in the detection anchor frame, P cl represents the probability that the plant is a crop or a weed (i.e., the coverage ratio between the bounding box and the pre-calibrated true box), P real represents the area of the intersection between the pre-calibrated ground-truth box and the bounding box, P box-area Represents the area of the bounding box, t is the weight parameter, according to d l The value of is determined, in this embodiment, t is 0.70.
[0062] Execute step S400: retain the bounding boxes in the feature map whose confidence is greater than a set threshold, and select the bounding box with the highest confidence and its position information from the retained bounding boxes.
[0063] Specifically, in this embodiment, the threshold Q is set to 0.60, that is, only P is retained. c The bounding box corresponding to ≥0.60 is used as the target of reverse learning.
[0064] Then, from P c ≥Q, find the bounding box with the largest confidence (its confidence is P c,m ), and then obtain the position parameter (x min ,y min ,x max ,y max ), where x min ,y min ,x max ,y max are respectively the minimum value of the horizontal coordinate, the minimum value of the vertical coordinate, the maximum value of the horizontal coordinate, and the maximum value of the vertical coordinate of the bounding box with the highest confidence.
[0065] Next, based on the position parameter (x min ,y min ,x max ,y max ) Obtain the position information of the bounding box with the maximum confidence according to the following formula:
[0066]
[0067]
[0068] Where w is the width of the bounding box with the highest confidence, h is the height of the bounding box with the highest confidence, and C x 、C y are the horizontal coordinates and vertical coordinates of the center point of the bounding box with the maximum confidence.
[0069] Execute step S500: take the position information of the bounding box with the highest confidence as a learning target, input it into the convolutional neural network for reverse learning, and obtain the optimal recognition box of the plant and its position information.
[0070] Specifically, the position information of the bounding box with the highest confidence is used as the learning target, and is input into the convolutional neural network for reverse learning to obtain several initial recognition frames. The initial recognition frames are screened using a non-maximum suppression function to obtain the optimal recognition frame of the plant and its position information.
[0071] It can be understood that the position information of the optimal identification frame also includes the width, height, and horizontal and vertical coordinates of the center point of the optimal identification frame. Figure 3 This is a schematic diagram of the optimal identification frame of the identified plant provided in this embodiment, as shown in FIG. Figure 3 As shown, the optimal recognition frame ( Figure 3 The dotted box in the figure is the identified crop plant. After determining the position information of the optimal identification frame, the position of each crop plant in the planting row has been determined.
[0072] Execute step S600: fit the planting row using the position information of the optimal recognition frame of the plant.
[0073] Specifically, the planting rows can be fitted using a piecewise linear regression function based on the position information of the optimal recognition frame of the plant. Figure 4 This is a schematic diagram of the planting row fitted in this embodiment. Figure 4 As shown, when the piecewise linear regression function is used to fit the planting row, the center points of the optimal identification frames located in the same row can be connected by a curve to form the planting row.
[0074] This embodiment also provides a crop row detection device. Figure 5 The structural block diagram of the crop planting row detection device provided in this embodiment is as follows: Figure 5 As shown, the crop planting row detection equipment includes:
[0075] The camera 10 is used to ride on the ridge and travel at a constant speed to capture the original image of the crop planting row;
[0076] An input module 20 is configured to input the original image into a convolutional neural network for downsampling to obtain m×m feature maps of equal size, where m is equal to the number of pixels of the original image divided by the downsampling multiple;
[0077] a confidence calculation module 30 for obtaining the confidence of the bounding box of the feature map based on the estimated value and the actual value of the horizontal offset of adjacent plants in the planting row;
[0078] a bounding box screening module 40, configured to retain bounding boxes in the feature map whose confidence levels are greater than a set threshold, and select the bounding box with the highest confidence level and its position information from the retained bounding boxes;
[0079] A reverse learning module 50 is configured to use the position information of the bounding box with the highest confidence as a learning target, input it into the convolutional neural network for reverse learning, and obtain the optimal recognition box of the plant and its position information;
[0080] The fitting module 60 is used to fit the planting rows using the position information of the optimal recognition frame of the plants.
[0081] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the steps of the above-mentioned crop planting row detection method are implemented.
[0082] Among them, the processor can perform various actions and processes according to the instructions stored in the memory. Specifically, the processor can be an integrated circuit chip with signal processing capabilities. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc., which can be an X86 architecture or an ARM architecture, etc.
[0083] The memory stores executable instructions that are executed by the processor to perform the above-described crop planting row detection method. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the method described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0084] According to another aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium is proposed, on which instructions are stored. When the instructions are executed, the steps in the crop planting row detection method described above can be implemented.
[0085] Similarly, the non-transitory computer-readable storage medium in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. It should be noted that the computer-readable storage medium described herein is intended to include, but is not limited to, these and any other suitable types of memory.
[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of the present invention can be implemented with the help of software and necessary general-purpose hardware. Of course, it can also be implemented with hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the existing technology, can be embodied in the form of a software product, which can be stored in a non-transitory computer-readable storage medium.
[0087] In summary, in the crop planting row detection method provided in this embodiment, the confidence of the bounding box of the feature map obtained after downsampling is calculated using the estimated value and actual value of the horizontal offset of adjacent plants in the planting row, thereby eliminating the interference of plants such as weeds in the detection anchor frame, improving the detection accuracy of crop planting rows, and reducing the impact of weeds between rows and between plants on the detection anchor frame. It is particularly suitable for the extraction of crop planting rows with similar ecological characteristics to weeds and crops such as rice, which are difficult to distinguish. It can be used as a reference for the "weeding and seedling avoidance" operation path, thereby providing support for the research and development of intelligent weeding and harvesting equipment for crops such as rice. In addition, when dw≤d l When ≤d+w, P c =P ob ×P cl ×t, which is equivalent to classifying objects that meet the horizontal offset of adjacent plants in the planting row as crops and assigning high weights, which reduces the computational burden and improves the recognition efficiency.
[0088] The above description is merely a preferred embodiment of the present invention and does not limit the present invention in any way. Any person skilled in the art who, without departing from the scope of the present invention, makes any equivalent substitution, modification, or other changes to the technical solution and technical content disclosed in the present invention shall be deemed to be within the scope of the present invention and still fall within the scope of protection of the present invention.
Claims
1. A method for detecting crop planting rows, It is characterized in that include: The camera travels at a constant speed on the ridge to capture the original image of the crop planting row; Input the original image into a convolutional neural network for downsampling to obtain m×m feature maps of equal size, where m is equal to the number of pixels of the original image divided by the downsampling multiple; Obtaining the confidence of the bounding box of the feature map based on the estimated value and the actual value of the horizontal offset of adjacent plants in the planting row; Retaining the bounding boxes in the feature map whose confidence is greater than a set threshold, and selecting the bounding box with the largest confidence and its position information from the retained bounding boxes; The position information of the bounding box with the highest confidence is used as a learning target, and input into the convolutional neural network for reverse learning to obtain the optimal recognition box of the plant and its position information; The planting row is fitted using the position information of the optimal recognition frame of the plant.
2. The method for detecting crop planting rows according to claim 1, It is characterized in that Before inputting the original image into the convolutional neural network for downsampling, the method further includes: The original image is compressed using an image compression algorithm.
3. The method for detecting crop planting rows according to claim 1, It is characterized in that The actual value d of the horizontal offset of adjacent plants in a planting row is calculated according to the following formula: l : d l =s×tanα; Wherein, s is the driving distance of the camera, and α is the heading angle of the camera.
4. The method for detecting crop planting rows according to claim 3, It is characterized in that The confidence P of the bounding box of the feature map is calculated according to the following formula: c : Where d is the estimated value of the horizontal offset of adjacent plants in a planting row, w is the preset planting row deviation threshold, and P ob Represents the detection of whether there are plants in the anchor frame. When P ob =1 indicates that there are plants in the detection anchor frame. ob = 0 means there is no plant in the detection anchor frame, P cl represents the probability that the plant is a crop or a weed, P real represents the area of the intersection between the pre-calibrated real box and the bounding box, P box-area represents the area of the bounding box, t is the weight parameter, and according to d l The value of is determined.
5. The method for detecting crop planting rows according to claim 1, It is characterized in that The position information of the boundary box with the highest confidence and the position information of the optimal identification box of the plant both include the width, height and the position of the center point.
6. The method for detecting crop planting rows according to claim 1, It is characterized in that The position information of the bounding box with the highest confidence is used as a learning target, input into the convolutional neural network for reverse learning to obtain a number of initial recognition frames, and the initial recognition frames are screened using a non-maximum suppression function to obtain the optimal recognition frame of the plant and its position information.
7. The method for detecting crop planting rows according to claim 1, It is characterized in that The planting rows are fitted using a piecewise linear regression function based on the position information of the optimal recognition frame of the plant.
8. A crop row detection device, It is characterized in that include: A camera, used for riding on the ridge and traveling at a constant speed to capture original images of the planting rows of the crops; An input module, used for inputting the original image into a convolutional neural network for downsampling to obtain m×m feature maps of equal size, where m is equal to the number of pixels of the original image divided by the downsampling multiple; A confidence calculation module, used for obtaining the confidence of the bounding box of the feature map based on the estimated value and the actual value of the horizontal offset of adjacent plants in the planting row; A bounding box screening module is used to retain the bounding boxes in the feature map whose confidence is greater than a set threshold, and select the bounding box with the largest confidence and its position information from the retained bounding boxes; A reverse learning module, used to take the position information of the bounding box with the highest confidence as a learning target, input it into the convolutional neural network for reverse learning, and obtain the optimal recognition box of the plant and its position information; The fitting module is used to fit the planting row using the position information of the optimal recognition frame of the plant.
9. A crop row detection system, It is characterized in that The method comprises a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the method for detecting crop planting rows as claimed in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium, It is characterized in that The non-transitory computer-readable storage medium stores instructions, and when the instructions are executed, the crop planting row detection method according to any one of claims 1 to 7 is implemented.
