Method, device, agricultural machinery and medium for identifying farmland soil surface

By combining image binarization and Hough transform with a deep learning model, the system automatically identifies farmland soil surfaces, solving the problem of poor robustness caused by reliance on manual annotation in existing technologies. This achieves fast and accurate farmland soil surface identification and adaptability to complex scenarios.

CN113807131BActive Publication Date: 2024-11-12GUANGZHOU XAIRCRAFT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010537234.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-06-12
Publication Date
2024-11-12
Estimated Expiration
2040-06-12

AI Technical Summary

Technical Problem

Existing methods for identifying farmland soil surfaces rely on manually labeled data, resulting in poor robustness and low identification efficiency.

Method used

A method combining image binarization, Hough transform, and deep learning models is used to automatically identify farmland soil surfaces. A deep learning model is generated by training a semantic segmentation convolutional neural network to quickly generate farmland soil surface labels.

Benefits of technology

It enables rapid and accurate identification of farmland soil surfaces, has the ability to identify complex scenarios, and supports unmanned vehicle navigation and crop yield prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113807131B_ABST
    Figure CN113807131B_ABST
Patent Text Reader

Abstract

The disclosed embodiment provides a method for identifying farmland soil surface, including: acquiring an image of the farmland; obtaining annotations of the farmland soil surface according to the image; pairing the image and the annotations to form a paired data set, and dividing the data set into a test data set and multiple training data sets; using the multiple training data sets and the test data sets to train and optimize parameters of a semantic segmentation convolutional neural network, respectively, and finally obtaining a deep learning model for identifying the farmland soil surface. Through the above technical solution, the semantic segmentation convolutional neural network is trained and tested by using the farmland image and its associated farmland soil surface annotations as paired data sets to generate a deep learning model, and the deep learning model is used to identify the farmland soil surface based on the farmland image. In this way, farmland soil surface annotations can be quickly generated, and a recognition model can be obtained through training, thereby having the ability to recognize farmland soil surface recognition in more complex scenes (such as non-mechanized planting).
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to a method, a device, an agricultural machine and a storage medium for identifying a farmland soil surface. Background Art

[0002] Farmland soil surface recognition can be used for the navigation of agricultural machinery (such as farmland robots) based on visual navigation. At present, farmland soil surface recognition methods segment crops in farmland through image analysis (such as SVM, clustering, etc.). This method is heavily dependent on manually labeled data, which requires a large amount of labeling work and results in poor robustness of the recognition method. Summary of the invention

[0003] The purpose of the embodiments of the present disclosure is to provide a method, device, agricultural machinery and storage medium that can quickly and accurately identify the soil surface of farmland.

[0004] In order to achieve the above object, according to a first aspect of the present disclosure, a method for determining a label of a farmland soil surface is provided, comprising:

[0005] Binarize the farmland image to extract the area where the vegetation is located in the image and generate a binary map of the image;

[0006] Determine the main orientation of the planting row in the image;

[0007] Taking the main direction and the direction perpendicular to the main direction as the coordinate system, the number of vegetation pixels in the main direction is accumulated to obtain an accumulation curve;

[0008] Determine the peak point and width of the cumulative curve;

[0009] Generate planting row rectangular strips row by row according to the peak vertices and widths, obtain a planting row area binary map, and obtain a non-crop area mask map based on the planting row area binary map; and

[0010] The soil surface area is extracted based on the non-crop area mask map and image to obtain the soil surface annotation.

[0011] In an embodiment of the present disclosure, binarizing the image of the farmland includes binarizing the image using at least one of the following methods:

[0012] Color space differentiation method;

[0013] Color index differentiation method;

[0014] Vegetation index differentiation method.

[0015] In an embodiment of the present disclosure, determining the main direction of a planting row in an image includes:

[0016] Convert the binary image to the polar coordinate system Hough space;

[0017] Returns a data list through linear Hough transform, which includes peak value, angle and distance;

[0018] The data lists were statistically analyzed to obtain the main directions of the planting rows.

[0019] In an embodiment of the present disclosure, performing statistical analysis on the data list to obtain the main direction of the planting row includes using at least one of the following methods:

[0020] The angle that appears most frequently in the data list is selected as the main direction of the planting row;

[0021] When the returned result is lower than the set threshold, the angle ranked first in the data list is selected as the main direction of the planting row.

[0022] In an embodiment of the present disclosure, the method further includes: before determining the peak apex and width of the cumulative curve, smoothing the cumulative curve.

[0023] In an embodiment of the present disclosure, accumulating the number of vegetation pixels in the main direction to obtain an accumulation curve includes:

[0024] Rotate the binary image until the main direction is vertical;

[0025] Expand the rotated binary image to a square, where the side length of the square is equal to the diagonal length of the binary image;

[0026] Fill the area outside the binary image in the square with zero values;

[0027] The number of non-zero pixels in the main direction is counted column by column to generate a cumulative curve.

[0028] In an embodiment of the present disclosure, obtaining a non-crop area mask image according to a planting row area binary image includes:

[0029] Performing an inversion operation on the pixels of the binary image of the planting row area;

[0030] The planting row area binary image is reversely rotated to restore the main direction to the direction before rotation;

[0031] The binary image of the planting row area is restored to its size before expansion to obtain a mask image of the non-crop area.

[0032] In an embodiment of the present disclosure, extracting the soil surface area according to the non-crop area mask map and the image to obtain the soil surface annotation includes:

[0033] In the case where the non-crop area is soil surface, the non-crop area mask map is used as the annotation;

[0034] In the case that the non-crop area is not entirely soil surface, the soil surface area is extracted, and the portion of the non-crop area mask image corresponding to the extracted soil surface area is used as the soil surface label.

[0035] In an embodiment of the present disclosure, the method further comprises:

[0036] Before accumulating the number of vegetation pixels in the main direction, determining whether a constraint condition is required;

[0037] Under the condition of requiring constraints, performing the step of accumulating the number of vegetation pixels in the main direction;

[0038] In the absence of constraints, a non-crop area mask map is obtained based on the binarized image.

[0039] In an embodiment of the present disclosure, the constraint includes a narrowest soil strip.

[0040] According to a second aspect of the present disclosure, a method for identifying a farmland soil surface is provided, comprising:

[0041] Acquire images of farmland;

[0042] Obtaining a label of the farmland soil surface in the image according to the image;

[0043] Pairing images and annotations associated with the images to form paired datasets, and dividing the datasets into a test dataset and a plurality of training datasets;

[0044] Iteratively training a semantic segmentation convolutional neural network using multiple sets of training data sets to obtain a deep learning model; data for each iteration is obtained by randomly extracting a number of image data from the multiple sets of training data sets, so as to update the parameters of the deep learning model through multiple iterations;

[0045] The test data set is input into the deep learning model, and the parameters of the deep learning model are optimized. The optimized deep learning model is used to identify the farmland soil surface.

[0046] In an embodiment of the present disclosure, the method further comprises:

[0047] Before using multiple sets of training data sets to train the convolutional neural network, the convolutional neural network is pre-trained using known data sets.

[0048] In an embodiment of the present disclosure, obtaining a label of a farmland soil surface in an image according to the image includes:

[0049] The annotations are obtained using the above-mentioned method for determining the annotations of the farmland soil surface.

[0050] According to a third aspect of the present disclosure, a method for planning an operation path for an unmanned vehicle is provided, comprising:

[0051] Determining the farmland soil surface using the above-mentioned method for identifying the farmland soil surface;

[0052] Obtaining the ridge positions of the farmland according to the determined farmland soil surface; and

[0053] The operating path of the unmanned vehicle is determined according to the position between ridges.

[0054] In an embodiment of the present disclosure, determining the operation path of the unmanned vehicle according to the position between ridges includes:

[0055] Determine the spacing between soil surface rows based on the position between ridges;

[0056] Adjust the wheelbase of the unmanned vehicle to an integer multiple of the spacing.

[0057] According to a fourth aspect of the present disclosure, there is provided a method for predicting crop yield, comprising:

[0058] Determining the farmland soil surface using the above-mentioned method for identifying the farmland soil surface;

[0059] The ridge spacing is obtained according to the determined farmland soil surface;

[0060] Compare the ridge spacing with the preset ridge spacing;

[0061] The crop yield is predicted based on the comparison results.

[0062] According to a fifth aspect of the present disclosure, there is provided an apparatus, comprising:

[0063] Processor; and

[0064] The memory is configured to store instructions, which are configured to enable the processor to perform at least one of the following when executed by the processor:

[0065] The above method for determining the marking of the farmland soil surface;

[0066] The above method for identifying the soil surface of a farmland;

[0067] The above method for planning an operation path for an unmanned vehicle;

[0068] The above method is used to predict crop yield.

[0069] According to a sixth aspect of the present disclosure, there is provided an agricultural machine comprising the above-mentioned device.

[0070] In an embodiment of the present disclosure, the agricultural machine includes at least one of the following:

[0071] Drones, unmanned vehicles, rice transplanters, and seed drills.

[0072] According to a seventh aspect of the present disclosure, a computer-readable storage medium is provided, on which instructions are stored, and when the instructions are executed by a processor, the processor can execute at least one of the following:

[0073] The above method for determining the marking of the farmland soil surface;

[0074] The above method for identifying the soil surface of a farmland;

[0075] The above method for planning an operation path for an unmanned vehicle;

[0076] The above method is used to predict crop yield.

[0077] Through the above technical solution, the semantic segmentation convolutional neural network is trained and tested by using the images of farmland and its associated farmland soil surface annotations as paired data sets to generate a deep learning model, which is used to identify the farmland soil surface based on the images of farmland. In this way, farmland soil surface annotations can be quickly generated, and a recognition model can be obtained through training, thereby having the ability to identify farmland soil surface recognition in more complex scenes (such as non-mechanized planting).

[0078] Other features and advantages of the embodiments of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the embodiments of the present disclosure but do not constitute a limitation on the embodiments of the present disclosure. In the accompanying drawings:

[0080] Figure 1A is a flowchart schematically illustrating an example of a method for identifying a farmland soil surface according to an embodiment of the present disclosure;

[0081] Figure 1B Schematically illustrates an example network structure of a deep learning network used in a method for identifying a farmland soil surface according to an embodiment of the present disclosure;

[0082] Figure 2 The diagram schematically shows a display method of soil surface annotation according to an embodiment of the present disclosure;

[0083] Figure 3 is a flowchart schematically illustrating an example of a method for determining a farmland soil surface label according to an embodiment of the present disclosure;

[0084] Figure 4A The figure schematically shows a farmland image acquired according to an embodiment of the present disclosure;

[0085] Figure 4B A schematic diagram schematically shows an expanded square and a rotated binarized image according to an embodiment of the present disclosure;

[0086] Figure 5 Schematically shows a smoothed cumulative curve according to an embodiment of the present disclosure;

[0087] Fig. 6A A binary map of a planting row area according to an embodiment of the present disclosure is schematically shown;

[0088] Figure 6B The binary image obtained after the inversion operation is performed on the planting row area binary image according to the embodiment of the present disclosure is schematically shown;

[0089] Figure 6C The schematic diagram shows Figure 6B The area of ​​the original binary image (i.e., the binary image before rotation and expansion) in the figure shown;

[0090] Fig.6D The non-crop area mask image obtained after de-rotation and de-expansion according to an embodiment of the present disclosure is schematically shown;

[0091] Figure 7 is a flowchart schematically illustrating an example of a method for determining a farmland soil surface label according to another embodiment of the present disclosure;

[0092] Figure 8 A binary map of a farmland soil surface without requiring constraints according to an embodiment of the present disclosure is shown;

[0093] Fig. 9 is a flowchart schematically illustrating an example of a method for planning an operation path for an unmanned vehicle according to another embodiment of the present disclosure;

[0094] Fig.10 is a flowchart schematically illustrating an example of a method for predicting crop yield according to another embodiment of the present disclosure; and

[0095] Fig.11 It is a structural block diagram schematically showing an example of an apparatus for implementing at least one of the methods according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0096] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0097] It should be noted that if the embodiments of the present disclosure involve directional indications (such as up, down, left, right, front, back, etc.), such directional indications are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0098] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present disclosure, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present disclosure.

[0099] Figure 1A FIG. 1 is a flow chart schematically showing an example of a method for identifying a farmland soil surface according to an embodiment of the present disclosure. Figure 1A As shown, a method for identifying farmland soil surface is provided, which may include the following steps.

[0100] In step S11, an image of the farmland is obtained. In the embodiment of the present disclosure, the image of the farmland can be obtained by at least one method. For example, in one example, a camera mounted on a drone can be used to take a bird's-eye view of the farmland to obtain an image. In another example, a camera located at a fixed point (e.g., a bird's-eye view point located at or near the farmland area, such as a pole, observation tower, etc.) can be used to take a bird's-eye view of the farmland to obtain an image. In yet another example, a balloon (e.g., a hot air balloon) can be used to take a bird's-eye view of the farmland to obtain an image.

[0101] In step S12, a label (label) of the farmland soil surface in the image is obtained according to the acquired image. The purpose of labeling is to classify the soil surface marks in the image, such as soil surface, background, etc. In one embodiment of the present disclosure, the image can be manually labeled to obtain the soil surface label. For example, an image semantic segmentation and labeling tool (for example, Labelme) can be used to mark the soil surface area in the farmland image with a polygonal box. In another embodiment of the present disclosure, the labeling can be generated by an appropriate algorithm, which will be described in detail below. In another embodiment of the present disclosure, the labeling can be generated by an appropriate algorithm, which will be described in detail below. The obtained soil surface labeling (or label image) can be a grayscale image, such as Figure 2 shown.

[0102] In step S13, the image and the associated annotation are paired to form a paired data set, and the data set is divided into a test data set and a plurality of training data sets. Specifically, multiple images of farmland can be obtained, and the annotation associated with each image is obtained, and the image and the associated annotation are paired to form a paired data set. The data set is divided into a plurality of training data sets and a test data set. In one example, the amount of data in the training data set can be more than the amount of data in the test data set. For example, the training data set can account for 90% of the data set, and the test data set can account for 10% of the data set.

[0103] In step S14, the semantic segmentation convolutional neural network is iteratively trained using multiple sets of training data sets to obtain a deep learning model. The data for each iteration is obtained by randomly extracting a number of image data from the multiple sets of training data sets, so as to update the parameters of the deep learning model through multiple iterations.

[0104] Specifically, when training a semantic segmentation convolutional neural network, each iteration can randomly extract a number of image data from multiple sets of training data sets to form a batch to update the parameters of the convolutional neural network and calculate the loss function (examples of loss functions may include but are not limited to BCEWithLogitsLoss, Focal Loss or cross entropy). The loss function can be used to estimate the degree of inconsistency between the predicted value and the true value of the model. It is a non-negative real-valued function. The smaller the loss function, the better the robustness of the model. The loss function is calculated using the predicted value and the true value, and the parameters of the convolutional neural network are adjusted based on the loss function. The parameters of the deep learning model can be updated through multiple iterations in this way.

[0105] In the embodiments of the present disclosure, gradient descent and back propagation algorithms can be used to optimize the training of the convolutional neural network. In one example, the Learning Rate Range Test algorithm can be used to find a suitable learning rate range, and then the cyclical learning rate method can be used to train the convolutional neural network.

[0106] In an embodiment of the present disclosure, the acquired farmland image can be preprocessed. For example, in one example, the farmland image can be enhanced and used as a detection image. In another example, each farmland image can be cropped into a plurality of sub-images of K*N pixels in size (where K and N represent natural numbers, and K can be the same as N or different from N). The pixel size K*N of the sub-image can be determined based on the processing frequency and existing of the graphics card used to process the image. In one example, the sub-images can be unified into a three-channel RGB image of size 512*512.

[0107] In step S15, the test data set is input into the deep learning model, and the parameters of the deep learning model are optimized.

[0108] After the semantic segmentation convolutional neural network is trained using the training data set to obtain a deep learning model, the test data set can be input into the deep learning model to optimize the parameters of the deep learning model. The optimized deep learning model can be used to identify farmland soil surface.

[0109] Specifically, after the convolutional neural network is trained using the training data set to obtain a deep learning model, the test data set is used to verify the model, i.e., target detection. Intersection-over-Union (IOU) can be used in target detection. IOU accuracy is an evaluation criterion for semantic segmentation accuracy, which can be defined as the ratio of the intersection and union of the actual value set and the predicted value set. The test data set is input into the deep learning model, and the deep learning model outputs a predicted image according to the input test data set, and calculates the IOU according to the predicted value of the predicted image and the true value of the annotation associated with the farmland image corresponding to the predicted image. The method for calculating IOU belongs to the method known to technicians in the relevant field, and will not be repeated here. The objective function is determined according to the calculated IOU. For example, the IOU loss can be calculated, the IOU loss can be incorporated into the objective function of the deep learning model, and the parameters of the deep learning model can be adjusted (or optimized) based on the objective function until the IOU accuracy reaches the expected value. For example, multiple iterations can be performed until the IOU accuracy no longer improves, for example, the degree of improvement in IOU accuracy tends to zero or is lower than a threshold.

[0110] In an optional embodiment of the present disclosure, before inputting a training data set to train the semantic segmentation convolutional neural network, the convolutional neural network can be pre-trained. For example, the convolutional neural network can be pre-trained with a known data set. The known data set may include, but is not limited to, ImageNet.

[0111] In an embodiment of the present disclosure, the semantic segmentation convolutional neural network may include, but is not limited to, for example, a fully convolutional network (FCN), Unet, and LinkNet.

[0112] Figure 1B An example network structure of a deep learning network used in a method for identifying farmland soil surface according to an embodiment of the present disclosure is schematically shown. Figure 1B The example network structure of the deep learning network shown can be used to implement the reference Figure 1A A method for identifying a soil surface of a farmland according to an embodiment of the invention is described. Figure 1BThe network structure of the deep learning network shown may include an encoder module 110 and a decoder module 120. The encoder module 110 may be used to gradually reduce feature maps and obtain higher semantic information. The decoder module 120 may be used to gradually restore spatial information.

[0113] Samples (such as images) in the training data set or the test data set are input into the deep learning network. The encoder module 110 can extract the feature information of each image in the data set, and the decoder module 120 upsamples the feature information and outputs a predicted image. The predicted image is the result of semantic segmentation, and the pixel values ​​of the predicted image correspond to a specific object type (i.e., a preset type). The output predicted image can be a grayscale image of the soil surface area, which is the result of semantic segmentation. In one example, the size of the grayscale image can be the same as the size of the collected farmland image. In the grayscale image, the value of any pixel can represent the semantic category (i.e., the preset type) to which the deep learning model predicts the object at that location. For example, 0 represents the background and 1 represents the soil surface.

[0114] The encoder module 110 may include a deep convolutional neural network (DCCN) and atrous spatial pyramid pooling (ASPP). DCCN may be used to extract features of farmland images, and the features may include color, shape, texture features, etc. ASPP may be used to increase the receptive field of the feature map, that is, the size of the convolution kernel seen on the image. In this embodiment, examples of the backbone network selected by DCCN may include, but are not limited to, ResNet, VGG, SENet, Xception, Mobilenet. Figure 1B As shown, in the encoder, DCNN uses multi-scale atrous convolution (Atrous Conv), including 3x3 Conv atrous convolution (layer), and the expansion rate (rate) can be 6, 12, and 18 respectively (that is, 3x3 Conv rate 6, 3x3 Conv rate 12, and 3x3 Conv rate 18 shown in the figure), which can perceive the semantic information on the input farmland image in a larger range to facilitate accurate segmentation. The input image (such as a farmland image) is input into the encoder module 110 and point-by-point atrous convolution (1x1 Conv shown in the figure), 3x3 Conv rate 6, 3x3 Conv rate 12, and 3x3 Conv rate 18 are performed in parallel. After pooling (such as Figure 1B After the Image Pooling operation shown in FIG. 1 , the feature information (feature map) is output after 1×1 Conv. The pooling operation in the encoder module 110 may adopt the maximum pooling (Maxpooling).

[0115] At the decoder module 120, the feature information output by the encoder module 110 may first be upsampled, for example Figure 1B As shown, it is first bilinearly upsampled (e.g., using an upsampling factor of 4), and then combined (Concatenation, Concat shown in the figure) with low-level features (Low-Level Features) with the same spatial resolution from the backbone network of the encoder module 110. In one example, before combining, the low-level features can be first 1x1 convolved (1x1 Conv) to reduce the number of channels. After combining, multiple convolution operations (e.g., 3x3 convolutions) can be applied, followed by bilinear upsampling (e.g., using an upsampling factor of 4), and finally outputting the predicted image.

[0116] Although Figure 1B FIG. 4 shows an example of a network structure of a deep learning network that can be applied to the method for identifying farmland soil surface according to an embodiment of the present disclosure, but a person skilled in the art can understand that the network structure can be implemented in the same manner as described above. Figure 1B Other types of deep learning networks that have the same or similar functions and achieve the same or similar effects or performances as the illustrated embodiments are also feasible, so the scope of the present disclosure is not limited to the specific network structure described in the specific embodiments.

[0117] In an optional embodiment of the present disclosure, before inputting a training data set to train the convolutional neural network, the convolutional neural network may be pre-trained. For example, the convolutional neural network may be pre-trained with a known data set. The known data set may include, but is not limited to, ImageNet.

[0118] In an embodiment of the present disclosure, the semantic segmentation convolutional neural network may include, but is not limited to, for example, a fully convolutional network (FCN), Unet, and LinkNet.

[0119] According to the method for identifying farmland soil surface provided by the embodiment of the present disclosure, the semantic segmentation convolutional neural network can be trained and tested by using the image of the farmland and its associated soil surface annotation as a paired data set to generate a deep learning model, and the deep learning model is used to identify the farmland soil surface based on the farmland image. In this way, a farmland soil surface label can be quickly generated, and a recognition model can be obtained through training, thereby having the ability to identify farmland soil surface in more complex scenes (such as non-mechanized planting).

[0120] After identifying the soil surface, the soil surface moisture content can be analyzed based on the identification results, and can be reused for regional segmentation of multispectral data to perform multispectral analysis of the soil surface. In addition, the soil surface distribution can be used for field robot navigation, etc.

[0121] As described above, the annotations may be generated by an appropriate algorithm. In an embodiment of the present disclosure, a method for determining annotations of a farmland soil surface is provided. The method may be applied to the above step S12. Figure 3 FIG. 1 is a flow chart schematically showing an example of a method for determining a farmland soil surface label according to an embodiment of the present disclosure. Figure 3 As shown, the method may include the following steps.

[0122] In step S31, the image of the farmland is binarized to extract the area where the vegetation is located in the image and generate a binarized image of the image.

[0123] Specifically, the purpose of extracting the area where vegetation (e.g., crops) is located is to segment the area in the image where vegetation is located. The area in the binarized image is either a vegetation area or a non-vegetation area. The binarization method may include, but is not limited to, color space (e.g., RGB to HSV), color index, vegetation index, and other distinction methods. In an example, taking the color index as an example, the excess green index (ExcessGreen), there is the following formula:

[0124] ExG=2*Green–Red–Blue

[0125] In this formula, ExG represents the super green index, Green represents the pixel value of the green channel, Red represents the pixel value of the red channel, and Blue represents the pixel value of the blue channel. This formula can be applied to images. In practical applications, the above formula can be subjected to mathematical processing such as normalization. After calculating the super green index, the vegetation area can be separated by a threshold value. The super green index greater than the threshold value is a non-zero value, and the value less than the threshold value is set to zero, thereby generating a binary image. The threshold can be set manually or obtained by the Ostu method. For example, the vegetation area can be a non-zero pixel, and the vegetation area can be a zero pixel.

[0126] Although the binarization process is described above using the super green index as an example, those skilled in the art may understand that other binarization methods are also feasible.

[0127] In step S32, the main direction of the planting row in the image is determined.

[0128] For example, the main direction of the planting row can be determined by the Hough line detection (linear Hough transform) method. More specifically, in the embodiment of the present disclosure, the binary image is converted to the polar coordinate system Hough space, and a data list is returned from the linear Hough transform (for example, using the function: skimage.transform.hough_line_peaks), and the data list includes peak values, angles, and distances. Here, the angle can refer to the polar coordinate angle, that is, the angle between the line connecting the pixel point and the origin in the original image (i.e., the binary image) and the x-axis (rectangular coordinate system), and the distance can refer to the distance from the pixel point to the origin. The returned peak value can be understood as the (Hough space) cumulative value, that is, the number of pixels passing through a hypothetical straight line in the original rectangular coordinate system. The more the number, the higher the credibility that this hypothetical straight line is a true straight line. The Hough transform assumes all possible straight lines in the image with a certain accuracy, and then accumulates the number of existing pixels on each hypothetical straight line, thereby obtaining the cumulative value on each hypothetical straight line. For example, assuming that there is a set of parallel straight lines (n) in the original image, then for this set of parallel straight lines, the angle is the same, and the number of times the angle appears is the same as the number of straight lines, which is n. Perform statistical analysis on the data list to obtain the main direction of the planting row. Examples of statistical analysis methods may include but are not limited to: selecting the angle that appears most frequently in the data list as the main direction of the planting row; when fewer results are returned (for example, below a set threshold), selecting the angle ranked first in the data list as the main direction of the planting row.

[0129] In step S33, the main direction and the direction perpendicular to the main direction are used as coordinate systems, and the number of vegetation pixels in the main direction is accumulated along the direction perpendicular to the main direction to obtain an accumulation curve.

[0130] In the embodiment of the present disclosure, in order to facilitate image processing, the binarized image can be rotated to a main direction in a vertical direction, and the binarized image can be expanded to a square. Specifically, the binarized image can be rotated according to the main direction angle so that the main direction is vertical. The diagonal length of the binarized image is calculated according to the height and width of the binarized image, and the binarized image is expanded to a square, and the side length of the square is equal to the diagonal length. The area outside the binarized image in the square can be filled with zero values. Figure 4A The acquired farmland image is schematically shown. Figure 4B The schematic diagram shows the expanded square and the rotated binarized image.

[0131] The main direction and the direction perpendicular to the main direction may be used as a coordinate system, such as an xy coordinate system, where the x-axis may be a direction perpendicular to the main direction and the y-axis may be the main direction. The number of non-zero pixels in the y-axis direction is counted column by column along the x-axis to generate a cumulative curve.

[0132] In step S34, the peak apex and width of the cumulative curve are determined.

[0133] In an optional embodiment of the present disclosure, before determining the peak vertex and width of the cumulative curve, the cumulative curve may be smoothed. The methods suitable for smoothing may include, but are not limited to, denoising after moving average, denoising after Univariate Spline fitting, and denoising after Savitzky_Golay Filter smoothing. The denoising situation may include, but is not limited to, correcting the value of negative numbers after Savitzky_Golay Filter smoothing, etc. Figure 5 The smoothed cumulative curve (curve II) is schematically shown. Figure 5 In the figure, curve I is the original peak, and the dots are the peaks found by the algorithm. It can be seen that if it is not smoothed, many erroneous peaks will appear, which is not conducive to line finding.

[0134] In the embodiments of the present disclosure, scipy.signal.find_peaks and other similar methods can be used to calculate the peak vertices and widths of the smoothed curve. The purpose of smoothing is mainly to reduce errors through parameter constraints. For example, a minimum value requirement can be set for the peak height parameter. If the peak is lower than the minimum value, it will be discarded. Figure 5 The leftmost peak can be discarded because it does not reach the minimum value (the x mark is the selected peak point). Similarly, in addition to the peak protrusion height, parameter constraints can also be set for the peak width.

[0135] In step S35, planting row rectangular strips are generated row by row according to the peak vertices and widths to obtain a planting row area binary map, and a non-crop area mask map is obtained based on the planting row area binary map.

[0136] Each row in the binary image of the planting row area can correspond to a planting row rectangle bar, the center line of the rectangle is the column where the peak fixed point is located, the width of the rectangle is the peak width, and the height of the rectangle is the width of the original image. The planting row rectangle bar means the planting row area under an ideal state. The area inside the rectangle bar can be a non-zero value, and the area between the rectangle bars can be a zero value. The peak width can be a value that can be obtained by setting parameters, and the peak width corresponding to different peak heights can be obtained. The pixels of the binary image of the planting row area are inverted, that is, non-zero values ​​become zero values, and zero values ​​become non-zero values, so that a mask image of the non-crop area can be obtained.

[0137] If the main direction is rotated as described above in step S33, the planting row area binary image is reversely rotated and reversely expanded to obtain a non-crop area mask image corresponding to the original image (binarized image). The reverse rotation and reverse expansion can be the reverse processing of the rotation and expansion in the above embodiment. That is, the main direction is rotated back to the original direction, and the planting row area binary image is restored to the size before expansion. Fig. 6A The binary image of the planting row area is schematically shown. Fig. 6A As shown, the pixels of the white rectangular strips may have non-zero values, and the pixels of the black areas between the rectangular strips may have zero values. Figure 6B The figure schematically shows the binary image obtained after the inversion operation is performed on the binary image of the planting row area. Figure 6C The schematic diagram shows Figure 6B The area of ​​the original binarized image (ie, the binarized image before rotation and expansion) in the figure shown. Fig.6D The non-crop area mask image obtained after de-rotation and de-expansion is schematically shown.

[0138] In step S36, the soil surface area is extracted according to the non-crop area mask map and the image to obtain a soil surface label (label map).

[0139] In the disclosed embodiment, if the non-crop area is the soil surface, the non-crop area mask can be directly used as a label. If the non-crop area is not entirely the soil surface, the soil surface area can be extracted. Specifically, the non-crop area mask can be screened for land conditions, such as land color screening, and the land in this scene can be set to be yellow or gray. Of course, those skilled in the art will appreciate that other extraction methods are also applicable. The portion of the non-crop area mask corresponding to the extracted soil surface area can be used as a label for the soil surface.

[0140] Figure 7 is a flowchart schematically illustrating an example of a method for determining farmland soil surface annotation according to another embodiment of the present disclosure. Figure 7 The embodiment shown is Figure 3 The illustrated embodiments are mostly the same, except that, before executing step S32, the method may further include step S71: determining whether it is strict, and if so, executing step S32. Specifically, whether it is strict may refer to whether a constraint condition is required, such as the narrowest soil strip. In the case where a constraint condition is required, it is easy to know the width of the land gap, which is beneficial to the navigation of agricultural machinery (such as a field robot). If no constraint condition is required, it can be understood that any area that is not vegetation (crops) may be land. Figure 8 A binary map of the farmland soil surface without required constraints is shown.

[0141] Figure 7The method of the illustrated embodiment may further include step S72: if the judgment is not strict, that is, no constraint condition is required, the non-crop area mask map may be directly obtained based on the binarized image. For example, the binarized image may be inverted (for example, if the binary value is 0 and 1, 0 becomes 1, and 1 becomes 0) to obtain the non-crop area mask map.

[0142] According to the method for determining the annotation of the farmland soil surface in accordance with the embodiment of the present disclosure, visual processing technology and algorithms are used to automatically process images of the farmland, quickly extract the soil surface area, and obtain a training image-annotation set for convolutional neural network training. The obtained neural network has the ability to cope with more complex situations, and can even cope with scenes where the planting rows are not very straight.

[0143] Farmland soil surface recognition can be applied to a variety of scenarios. In one embodiment of the present disclosure, one application of farmland soil surface recognition can be to plan the operation path of an unmanned vehicle. Fig. 9 FIG. 1 is a flow chart schematically showing an example of a method for planning an operation path for an unmanned vehicle according to another embodiment of the present disclosure. Fig. 9 As shown, in an embodiment of the present disclosure, a method for planning an operating path for an unmanned vehicle may include the following steps.

[0144] In step S91, the farmland soil surface is determined. For example, the farmland soil surface can be determined using the method for identifying the farmland soil surface in the above embodiment.

[0145] In step S92, the position between ridges of the farmland is obtained according to the determined farmland soil surface. Specifically, after the farmland soil surface is determined, the position between ridges of the farmland can be obtained (identified) from the farmland soil surface.

[0146] In step S93, the operation path of the unmanned vehicle is determined according to the position between the ridges. Specifically, after the position between the ridges is determined, the operation path of the unmanned vehicle can be planned along the position between the ridges, so that the unmanned vehicle operates along the position between the ridges.

[0147] In the disclosed embodiment, the wheelbase of the unmanned vehicle is adjustable. When performing operation path planning, it is expected that the driving trajectory of the wheels of the unmanned vehicle always follows the soil surface row, that is, it crosses one or more ridges without pressing the ridges during driving. Therefore, in the method of the disclosed embodiment, the spacing between the soil surface rows can be determined according to the position between the ridges, and the wheelbase of the unmanned vehicle can be adjusted according to the spacing. For example, the wheelbase of the unmanned vehicle can be adjusted to an integer multiple of the spacing, so that the wheels of the unmanned vehicle can travel along the trajectory of the soil surface row as much as possible.

[0148] In the embodiments of the present disclosure, another application of farmland soil surface identification may be predicting crop yields. Fig.10FIG. 1 is a flow chart schematically showing an example of a method for predicting crop yield according to another embodiment of the present disclosure. Fig.10 As shown, in an embodiment of the present disclosure, a method for predicting crop yield may include the following steps.

[0149] In step S101, the farmland soil surface is determined. For example, the farmland soil surface can be determined using the method for identifying the farmland soil surface in the above embodiment.

[0150] In step S102, the ridge spacing is obtained according to the determined farmland soil surface. Specifically, after the farmland soil surface is determined, the ridge spacing of the farmland, that is, the distance between two adjacent ridges, can be obtained (identified) from the farmland soil surface.

[0151] In step S103, the ridge spacing is compared with a preset ridge spacing.

[0152] In step S104, the crop yield is predicted according to the comparison result.

[0153] Specifically, each crop plant has a specific ridge spacing corresponding to the maximum yield (e.g., ideal maximum yield), and the specific ridge spacing can be used as a preset ridge spacing. Too large or too small ridge spacing of the plant will affect its yield. Therefore, in the embodiment of the present disclosure, the crop yield can be predicted by comparing the ridge spacing (actual ridge spacing) obtained from the farmland soil table with the preset ridge spacing. In one example, the yield prediction can involve a preliminary prediction, that is, predicting whether the yield can reach the maximum yield based on the comparison result. In another example, the relationship between the difference between the ridge spacing and the preset ridge spacing and the yield impact value (the difference with the maximum yield) can be predetermined for a specific crop, such as a lookup table. The lookup table can be determined based on a limited number of experiments or based on experience. After determining the actual ridge spacing, the corresponding yield impact value is found from the lookup table based on the difference between the actual ridge spacing and the preset ridge spacing, and the crop yield is predicted based on the maximum yield and the yield impact value.

[0154] Fig.11 The structure block diagram of an example of a device for identifying a farmland soil surface according to an embodiment of the present disclosure is schematically shown. Fig.11 As shown, in an embodiment of the present disclosure, the device may include a processor 1110 and a memory 1120. The memory 1120 may store instructions, which, when executed by the processor 1110, may enable the processor 1110 to execute the method for identifying a farmland soil surface or the method for determining a label of a farmland soil surface described in the previous embodiment.

[0155] Examples of processor 1110 may include, but are not limited to, a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) circuit, any other type of integrated circuit (IC), and a state machine, etc. The processor may perform signal encoding, data processing, power control, input / output processing.

[0156] Examples of memory 1120 may include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk-read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a processor.

[0157] In an embodiment of the present disclosure, an agricultural machine is also provided, which may include the device for identifying the farmland soil surface described in the above embodiment. Examples of agricultural machines may include, but are not limited to, drones, unmanned vehicles, rice transplanters, seed drills, and the like.

[0158] In an embodiment of the present disclosure, a computer-readable storage medium is also provided, on which instructions are stored. When the instructions are executed by a processor, the processor can execute the method for identifying the farmland soil surface or the method for determining the marking of the farmland soil surface described in the previous embodiment.

[0159] For the convenience of description, the above device is described in terms of functions and is divided into various units and described separately. Of course, when implementing the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0160] It will be appreciated by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may be provided in the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may be provided in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0161] The present disclosure is described with reference to the flowchart and / or block diagram of the method, device (system), and computer program product according to the embodiment of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the function specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0162] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0163] These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0164] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0165] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0166] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0167] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system implementation, since it is basically similar to the method implementation, the description is relatively simple, and the relevant parts can be referred to the partial description of the method implementation.

[0168] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A method for determining the marking of a farmland soil surface, characterized in that: include: Binarizing the image of the farmland to extract the area where the vegetation is located in the image and generate a binary map of the image; determining a main direction of a planting row in the image; Taking the main direction and the direction perpendicular to the main direction as the coordinate system, the number of vegetation pixels in the main direction is accumulated to obtain an accumulation curve; Determining the peak apex and width of the cumulative curve; Generating planting row rectangular strips row by row according to the wave peak vertices and the width to obtain a planting row area binary map, and obtaining a non-crop area mask map according to the planting row area binary map; and Extracting a soil surface area according to the non-crop area mask map and the image to obtain a soil surface annotation; The step of accumulating the number of vegetation pixels in the main direction to obtain an accumulation curve includes: Rotating the binarized image until the main direction is in a vertical direction; Expanding the rotated binary image to a square, wherein the side length of the square is equal to the diagonal length of the binary image; Fill the area outside the binary image in the square with zero values; The number of non-zero pixels in the main direction is counted column by column to generate a cumulative curve.

2. The method according to claim 1, characterized in that The binarization of the image of the farmland comprises binarization of the image using at least one of the following methods: Color space differentiation method; Color index differentiation method; Vegetation index differentiation method.

3. The method according to claim 1, characterized in that Determining the main direction of the planting row in the image comprises: Convert the binary image to the polar coordinate system Hough space; Return a data list by linear Hough transform, wherein the data list includes peak value, angle and distance; The data list is statistically analyzed to obtain the main direction of the planting row.

4. The method according to claim 3, characterized in that The statistical analysis of the data list to obtain the main direction of the planting row includes using at least one of the following methods: Select the angle that appears most frequently in the data list as the main direction of the planting row; When the returned result is lower than the set threshold, the angle ranked first in the data list is selected as the main direction of the planting row.

5. The method according to claim 1, characterized in that Also includes: Before determining the peak apex and width of the cumulative curve, the cumulative curve is smoothed.

6. The method according to claim 1, characterized in that The mask map of the non-crop area obtained according to the binary map of the planting row area includes: Performing an inversion operation on the pixels of the binary image of the planting row area; Reversely rotating the planting row area binary image to restore the main direction to the direction before the rotation; The planting row area binary image is restored to the size before expansion to obtain the non-crop area mask image.

7. The method according to claim 1, characterized in that The step of extracting the soil surface area according to the non-crop area mask map and the image to obtain soil surface annotation includes: In the case where the non-crop area is a soil surface, using the non-crop area mask image as the annotation; In the case that the non-crop area is not entirely soil surface, the soil surface area is extracted, and the portion of the non-crop area mask image corresponding to the extracted soil surface area is used as a soil surface label.

8. The method according to claim 1, characterized in that Also includes: Before accumulating the number of vegetation pixels in the main direction, determining whether a constraint condition is required; In the case where a constraint condition is required, performing the step of accumulating the number of vegetation pixels in the main direction; In the case where no constraint conditions are required, a non-crop area mask image is obtained according to the binarized image.

9. The method according to claim 8, characterized in that The constraints include the narrowest soil strip.

10. A method for identifying farmland soil surface, characterized in that: include: Acquire images of farmland; Obtaining the annotation of the farmland soil surface in the image according to the image and the method for determining the annotation of the farmland soil surface according to any one of claims 1 to 9; Pairing the image and the annotation associated with the image to form a paired dataset, and dividing the dataset into a test dataset and a plurality of training datasets; Iteratively training the semantic segmentation convolutional neural network using the multiple sets of training data sets to obtain a deep learning model; the data of each iteration is obtained by randomly extracting a number of image data from the multiple sets of training data sets, so as to update the parameters of the deep learning model through multiple iterations; The test data set is input into the deep learning model, parameters of the deep learning model are optimized, and the optimized deep learning model is used to identify farmland soil surface.

11. The method according to claim 10, characterized in that Also includes: Before using the multiple sets of training data sets to train the semantic segmentation convolutional neural network, the semantic segmentation convolutional neural network is pre-trained using a known data set.

12. The method according to claim 10, characterized in that The step of obtaining a label of a farmland soil surface in the image according to the image includes: The annotation is obtained using the method for determining the annotation of the farmland soil surface according to any one of claims 1 to 9.

13. A method for planning an operation path for an unmanned vehicle, characterized in that: include: Determining the farmland soil surface using the method for identifying the farmland soil surface according to any one of claims 10 to 12; Obtaining the ridge positions of the farmland according to the determined farmland soil surface; and The operation path of the unmanned vehicle is determined according to the position between the ridges.

14. The method according to claim 13, characterized in that Determining the operation path of the unmanned vehicle according to the position between ridges includes: Determine the spacing between soil surface rows according to the ridge positions; The wheelbase of the unmanned vehicle is adjusted to an integral multiple of the spacing.

15. A method for predicting crop yield, characterized in that: include: Determining the farmland soil surface using the method for identifying the farmland soil surface according to any one of claims 10 to 12; The ridge spacing is obtained according to the determined farmland soil surface; comparing the ridge spacing with a preset ridge spacing; The crop yield is predicted based on the comparison results.

16. A device for processing farmland soil surface, characterized in that: include: processor; as well as A memory configured to store instructions, wherein the instructions are configured to enable the processor to perform at least one of the following when executed by the processor: A method for determining a mark on a farmland soil surface as claimed in any one of claims 1 to 9; The method for identifying the soil surface of a farmland as claimed in any one of claims 10 to 12; The method for planning an operating path for an unmanned vehicle according to claim 13 or 14; The method for predicting crop yield as claimed in claim 15.

17. An agricultural machine, characterized in that: Comprising the apparatus according to claim 16.

18. A computer-readable storage medium, characterized in that: Instructions are stored on the computer-readable storage medium, and when the instructions are executed by a processor, the processor can perform at least one of the following: A method for determining a mark on a farmland soil surface as claimed in any one of claims 1 to 9; The method for identifying the soil surface of a farmland as claimed in any one of claims 10 to 12; The method for planning an operating path for an unmanned vehicle according to claim 13 or 14; The method for predicting crop yield as claimed in claim 15.