Method for measuring inter-row path spacing of closed-row crops

Abstract

The invention discloses a method for measuring the path spacing between rows of closed-row crops. The method comprises the following steps that crop image information of different visual angles is collected through a first depth camera arranged at the top of agricultural equipment and two second depth cameras arranged at two front wheels, and the visual fields of the two second depth cameras are overlapped with the first depth camera; respectively extracting boundaries of soil and crops in the two second images of the second depth camera, and further determining an advancing route of the corresponding front wheel; according to the position relation between the second depth camera and the first depth camera, respectively mapping the starting points of the advancing routes in the two second images into the first image of the first depth camera; the distance between the two starting points mapped in the first depth camera is calculated. The method is high in measurement accuracy and high in stability, can be used for path calculation and navigation guidance of inter-row operation in the middle and later periods of crop growth, facilitates agricultural equipment to timely adjust the wheel track according to the measured distance, and realizes flexible and efficient autonomous operation.

Description

Technical Field

[0001] The present application belongs to the field of smart agricultural technology, and specifically relates to a method for measuring the path spacing between rows of closed-row crops. Background Art

[0002] The application of intelligent agricultural equipment is one of the key technologies to ensure food security. During precision field operations, agricultural equipment needs to be able to accurately navigate and move along paths between crop rows, but these paths are usually determined by crop planting and growth patterns, and their spacing may vary depending on factors such as terrain, planting errors, and crop types.

[0003] Currently, agricultural equipment is often guided between rows by satellite fixed tracks or manually controlled navigation. This method has the problems of low efficiency, high cost and inability to adapt to complex field environments. The development of technologies such as computer vision and artificial intelligence has made it possible to detect the path between rows. This method can not only improve the accuracy and efficiency of navigation, but also provide reliable technical support for the measurement of path spacing when agricultural equipment is traveling between crop rows.

[0004] However, in the middle and late stages of crop growth, the branches and leaves of two adjacent rows of crops will intertwine with each other, covering the ground gaps between the crop rows, forming a continuous, closed green area, which is called a closed row. In this case, the traditional visual navigation method that uses the natural line characteristics of crop rows in the field will be disturbed and unable to obtain clear inter-row path information, which in turn affects the measurement of the inter-row path spacing of crops. If the wheelbase of the agricultural intelligent equipment does not match the inter-row path spacing of the closed-row crops, it will make it difficult for the agricultural equipment to effectively guide the rows, which will affect the efficiency and accuracy of field operations and may even damage the normal growth of crops. Summary of the invention

[0005] In order to solve the above problems, a method for measuring the path spacing between rows of closed-row crops is invented, and the method comprises the following steps:

[0006] A first depth camera is arranged on the top of the agricultural equipment and two corresponding second depth cameras are arranged at the two front wheels to respectively collect crop image information from different perspectives, wherein the second depth camera and the first depth camera have overlapping fields of view; the image collected by the first depth camera is recorded as the first image, and the images collected by the two second depth cameras are recorded as the second images;

[0007] respectively extracting the boundary lines between the soil and the crop in the two second images of the second depth camera, and then determining the travel routes of the corresponding front wheels;

[0008] According to the positional relationship between the second depth camera and the first depth camera, the starting points of the travel routes in the two second images are respectively mapped into the first image of the first depth camera;

[0009] The distance between the two starting points mapped in the first depth camera is calculated as the spacing of the paths between the closed-row crops.

[0010] Preferably, the step of respectively extracting the boundary lines between soil and crops in the two second images of the second depth camera and then determining the travel routes of the corresponding front wheels specifically includes:

[0011] Do the same for each second image:

[0012] Binarize the second image to obtain two binary images;

[0013] Drawing a plurality of lines running through the binary image on the binary image at preset intervals, wherein the direction of the lines is perpendicular to the arrangement direction of the crops;

[0014] Read each pixel value on all lines in turn. If the pixel value of this pixel is different from the pixel value of the next pixel, select this pixel as a feature point.

[0015] Calculate the center points corresponding to the feature points at both ends of the soil area on all lines, and fit all the center points to obtain the travel route of the front wheel corresponding to the second depth camera to which the second image belongs. Preferably, the step of mapping the starting points of the travel routes in the two second depth camera images to the image of the first depth camera respectively according to the positional relationship between the second depth camera and the first depth camera specifically includes:

[0016] According to the respective intrinsic parameter matrices of the three depth cameras and the respective rotation matrices and translation matrices from the two second depth cameras to the first depth camera, the starting points of the travel routes in the two second images are respectively mapped into the first image of the first depth camera through coordinate transformation.

[0017] Preferably, the step of mapping the starting points of the travel routes in the two second images into the first image of the first depth camera respectively through coordinate transformation specifically includes:

[0018] The pixel coordinates p of the starting point of the travel route in one of the second depth camera images are A Convert to the corresponding camera coordinate system P A ,

[0019]

[0020] in, is the inverse of the intrinsic parameter matrix of a second depth camera, d A is a depth value corresponding to a starting point in a second depth camera;

[0021] Point P in the camera coordinate system of the second depth cameraA P converted to the camera coordinate system of the first depth camera A ′,

[0022] P A ′=R A P A +T A

[0023] Among them, R A is the rotation matrix from the first depth camera to the second depth camera, T A is the translation matrix from the first depth camera to the second depth camera;

[0024] The first depth camera coordinate P A ′ is converted to point p′ in the corresponding pixel coordinate system A ,

[0025] p′ A =KP A ′ / d

[0026] Wherein, K is the intrinsic parameter matrix of the first depth camera, and d is the depth value corresponding to a starting point in the first depth camera;

[0027] The pixel coordinate p of the starting point of the travel route in another second depth camera image is B Mapped to p′ in the pixel coordinate system of the first depth camera in the same way as above B .

[0028] Preferably, the step of calculating the distance between two starting points mapped in the first depth camera further comprises:

[0029] p′ A The corresponding point P in the camera coordinate system A ′ is (X A , Y A , Z A ), p′ B The corresponding point P in the camera coordinate system B ′(X B , Y B , Z B ), calculate P A ′ and P B ′ as the distance between the two starting points,

[0030]

[0031] Preferably, it is characterized in that the second image is an image including the vicinity of the roots of crops, and the first image is a global image including the front of the agricultural equipment.

[0032] This method has high measurement accuracy and strong stability, and can be used for path calculation and navigation guidance of inter-row operations in the middle and late stages of crop growth. It is convenient for agricultural equipment to adjust the wheelbase in time according to the measured distance, thus achieving flexible and efficient autonomous operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 A flow chart of a method for measuring path spacing between rows of field crops provided in an embodiment of the present application;

[0036] Figure 2 Example diagrams of different road condition data sets and annotation results provided in the embodiments of the present application;

[0037] Figure 3 Extracting sample graphs for travel routes;

[0038] Figure 4 Define a graph for the camera coordinate system transformation relationship;

[0039] Figure 5 This is an example diagram of the travel routes in the first depth camera and the second depth camera in this embodiment;

[0040] Figure 6 This is an example diagram of the distance measurement of the travel route in this embodiment;

[0041] Figure 7 This is an example diagram of route spacing measurement in this embodiment. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0043] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0044] This embodiment proposes a field path route distance measurement method, combined with Figure 1 , the method comprises the following steps:

[0045] Crop image information from different perspectives is collected through a first depth camera set on the top of the agricultural equipment and two second depth cameras set at the two front wheels, wherein the fields of view of the two second depth cameras overlap with the first depth camera.

[0046] In this embodiment, the two second depth cameras are depth cameras located at the two front wheels of the agricultural equipment, respectively located on a separate mounting frame, and shooting toward the front side of the wheel. Since the top leaves of crops grow staggered after the rows are closed, but the roots of the crops still show a linear distribution characteristic, it is more appropriate to install the two second depth cameras at the front wheels, and their installation positions can be appropriately adjusted according to the growth conditions of the crops to ensure that the characteristic information of the path between the rows can be obtained, while avoiding being blocked by the leaves of the crops.

[0047] The first depth camera is a depth camera installed on the top of the agricultural equipment. It is located at the front of the agricultural equipment and its task is to capture the image information of the farmland in front of the vehicle. At the same time, in order to facilitate the subsequent coordinate conversion operation, the shooting angle needs to be adjusted to ensure that the camera has an overlapping field of view with the two second depth cameras at the wheels.

[0048] The boundary lines between the soil and the crop in the two second images of the second depth camera are respectively extracted, and then the travel route of the corresponding front wheel is determined.

[0049] In this embodiment, the second images captured by the two second depth cameras are input into the pre-trained deep neural network in the form of video frames. In the sample annotation stage before training, we use the line connecting the roots of the crop rows as the boundary to distinguish soil and crops. Since the shape of crop branches and leaves is extremely irregular, all possible weeds, some branches and leaves, and stones on the road surface are identified as soil parts, which is conducive to the rapid extraction of subsequent road feature points. Some of the data sets and their annotation results are shown in Figure 1. Figure 2 As shown, Figure 2 a, 2b, and 2c are the road conditions at three different times. Figure 2 d, 2e, and 2f are the labels of the corresponding images.

[0050] The deep neural network adopts a semantic segmentation model, such as U-Net, SegNet, PSPNet and DeepLab, etc. The specific selection is not limited in the present invention. The general semantic segmentation architecture can be considered as an encoder-decoder network.

[0051] In order to exclude the possible holes in the image, first find all the connected areas included in the output image, and select the area with the largest area as the largest connected area. Set the pixel values ​​of all pixels in the connected area with the largest area to 1, and set the pixel values ​​of other positions to 0, and then you will get an image with only the path area as the target foreground. The boundary lines of soil and crops in the binary image obtained after binarization will be relatively complete, where the white area within the boundary represents the soil, that is, the wheel drivable area, and the black area outside the boundary represents the crop, that is, the non-drivable area. Making full use of these features can improve the robustness of the field path extraction results.

[0052] Draw a number of horizontal lines running through the binary image every 10 lines, wherein the lines are perpendicular to the arrangement direction of the crops; read each pixel value on the lines in turn, and if the pixel value of the pixel point is different from the pixel values ​​of the pixel points on the left and right sides of the line where the pixel point is located, it means that the pixel point is on the boundary, and the pixel point is selected as the feature point.

[0053] After extracting all feature points on the left and right boundaries using the above process, the two feature points on the same line are averaged and weighted to generate the midpoint of the path in the drivable area. At this time, the weights on both sides are 0.5. The midpoint of the path is then fitted by the least squares fitting method to obtain the route of the drivable area. The result is as follows: Figure 3 shown.

[0054] According to the positional relationship between the second depth camera and the first depth camera, the starting points of the travel routes in the two second images are respectively mapped into the first image of the first depth camera;

[0055] In this embodiment, in order to achieve more comprehensive environmental perception, the travel routes extracted by the second depth cameras at the two wheels are mapped to the first depth camera on the top of the agricultural equipment. Since the positions of the two second depth cameras at the wheels and the first depth camera on the top of the agricultural equipment are relatively fixed and there is an overlap in the field of view, the mapping between feature points on different camera images can be achieved through coordinate transformation technology, and the above process requires the internal parameter matrix and external parameter matrix of the second depth camera and the first depth camera to be obtained in advance.

[0056] Each camera has its own intrinsic matrix, which is used to transform pixels from the pixel coordinate system to the camera coordinate system.

[0057] The specific parameters in the intrinsic matrix are obtained using the camera calibration toolbox of MATLAB. First, a set of calibration images should be prepared. These images should have different angles and positions to cover the entire field of view of the camera. Then, after opening the camera calibration application in MATLAB, the program will automatically detect the corner points and calculate the intrinsic parameter information such as the focal length f. x 、f y , principal point position c x 、c y , thereby completing the extraction of the intrinsic parameter matrix of any camera. The intrinsic parameter matrix is ​​usually represented as K, and its form is shown below.

[0058]

[0059] The external parameter matrix consists of two parts: the rotation matrix and the translation matrix, which are used to transform points from one camera coordinate system to another camera coordinate system, such as Figure 4 As shown, the coordinate origin O of the two camera coordinate systems O-XYZ is the optical center of the depth camera, the Z axis is the optical axis of the depth camera, and its positive direction points to the shooting scene. The X axis is parallel to the camera mirror. When observed along the shooting direction of the camera, its direction is horizontal to the right. The X axis, Y axis, and Z axis satisfy the right-hand rule.

[0060] The rotation matrix describes the relative orientation relationship between the two camera coordinate systems. We define the installation pitch angle θ of camera 1 and camera 2. 1 and θ 2 , the two cameras rotate only on the x-axis. At this time, according to the difference in the depression angle of camera 2 relative to camera 1 in the x-direction, θ = θ 2 -θ 1 , we can get the rotation matrix R in the form shown below.

[0061]

[0062] The translation matrix describes the relative position relationship between the two camera coordinate systems. We define the offset Δx of camera 2 relative to camera 1 in the x direction, the offset Δy of camera 2 relative to camera 1 in the y direction, and the offset Δz of camera 2 relative to camera 1 in the height (z direction). At this time, the translation matrix T can be expressed as a three-dimensional vector.

[0063]

[0064] The pixel coordinates p of the starting point of the travel route in one of the second depth camera images are A Convert to the corresponding camera coordinate system P A ,

[0065]

[0066] in, is the inverse of the intrinsic parameter matrix of a second depth camera, d A is a depth value corresponding to a starting point in a second depth camera;

[0067] Point P in the camera coordinate system of the second depth camera A P converted to the camera coordinate system of the first depth camera A ′,

[0068] P A ′=R A P A +T A

[0069] Among them, R A is the rotation matrix from the first depth camera to the second depth camera, T A is the translation matrix from the first depth camera to the second depth camera;

[0070] The first depth camera coordinate P A ′ is converted to point p′ in the corresponding pixel coordinate system A ,

[0071] p′ A =KP A ′ / d

[0072] Wherein, K is the intrinsic parameter matrix of the first depth camera, and d is the depth value corresponding to a starting point in the first depth camera;

[0073] The pixel coordinate p of the starting point of the travel route in another second depth camera image is B Mapped to p′ in the pixel coordinate system of the first depth camera in the same way as above B .

[0074] like Figure 5 As shown, Figure 5 a is the first image taken by the first depth camera, Figure 5 b and Figure 5 c are the second images taken by two second depth cameras respectively.

[0075] Calculate the distance between two starting points mapped in the first depth camera.

[0076] In this embodiment, the spatial distance between two starting points in the first image in the camera coordinate system is calculated using the Euclidean distance formula, that is, the actual distance between the two points.

[0077] like Figure 6 As shown in the figure, the red and green points represent the starting points of the two navigation baselines, and the black line between the two points represents the distance to be measured. After calculation, the camera coordinates corresponding to the left and right starting points are [-1328.55, 2.21, 1512.0] and [-298.95, 2.18, 1488.0] respectively. Substitute these two points into the following Euclidean distance formula to calculate the distance between the two starting points:

[0078]

[0079] The spatial distance between the two points is 1030, which is in millimeters (mm). The unit is then converted to meters (m), and the converted value 1.03m is displayed in the image.

[0080] In order to further verify the effectiveness of the above algorithm, an area with basically consistent row spacing was selected, and the path spacing of the four crop rows in the center of 250 frames of images was continuously measured. The measurement results showed that the spacing values ​​were stable at about 1m, which was consistent with the row spacing in actual planting. Figure 7 As shown in the figure, the rationality and stability of the algorithm are verified.

[0081] This method is based on visual information perception and coordinate mapping technology. Through the collaborative work between the first depth camera and two second depth cameras with overlapping fields of view, the travel routes of the two front wheels are extracted, and then the starting points of their respective routes are accurately mapped in the first depth camera image and the distance is measured. This method of obtaining information using multi-view cameras can more comprehensively reflect the actual scene and avoid the limitations of a single perspective. It is particularly effective in the closed row scene in the middle and late stages of crop growth. Through real-time measurement of path spacing, agricultural equipment with variable wheelbase can flexibly adjust the wheelbase size in a dynamic environment, always ensuring accurate operating paths and stable navigation performance. For equipment that does not have a wheelbase adjustment function, when a significant change in path spacing is detected, effective adjustment measures can be taken to avoid crushing crops or damaging equipment, thereby achieving efficient autonomous operation management.

[0082] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for measuring the path spacing between rows of closed-row crops, characterized in that The method comprises the following steps: The crop image information of different perspectives is collected by a first depth camera arranged on the top of the agricultural equipment and two second depth cameras arranged at the two front wheels, wherein the field of view of the two second depth cameras overlaps with that of the first depth camera; the image collected by the first depth camera is recorded as the first image, and the images collected by the two second depth cameras are recorded as the second image; respectively extracting the boundary lines between the soil and the crop in the two second images of the second depth camera, and then determining the travel routes of the corresponding front wheels; According to the positional relationship between the second depth camera and the first depth camera, the starting points of the travel routes in the two second images are respectively mapped into the first image of the first depth camera; The distance between the two starting points mapped in the first depth camera is calculated as the spacing of the paths between the closed-row crops.

2. The method for measuring the path spacing between rows of closed-row crops according to claim 1, characterized in that: The step of respectively extracting the boundary lines between soil and crops in the two second images of the second depth camera and then determining the travel path of the corresponding front wheel specifically includes: Do the same for each second image: Binarize the second image to obtain two binary images; Drawing a plurality of lines running through the binary image on the binary image at preset intervals, wherein the direction of the lines is perpendicular to the arrangement direction of the crops; Read each pixel value on all lines in turn. If the pixel value of this pixel is different from the pixel value of the next pixel, select this pixel as a feature point. The center points corresponding to the feature points at both ends of the soil area on all the lines are calculated, and the travel route of the front wheel corresponding to the second depth camera to which the second image belongs is obtained by fitting all the center points.

3. The method for measuring the path spacing between rows of closed-row crops according to claim 1, characterized in that: The step of mapping the starting points of the travel routes in the two second depth camera images to the image of the first depth camera respectively according to the positional relationship between the second depth camera and the first depth camera specifically includes: According to the respective intrinsic parameter matrices of the three depth cameras and the respective rotation matrices and translation matrices from the two second depth cameras to the first depth camera, the starting points of the travel routes in the two second images are respectively mapped into the first image of the first depth camera through coordinate transformation.

4. The method for measuring the path spacing between rows of closed-row crops according to claim 3, characterized in that: The step of mapping the starting points of the travel routes in the two second images to the first image of the first depth camera respectively through coordinate conversion specifically includes: The pixel coordinates p of the starting point of the travel route in one of the second depth camera images are A Convert to the corresponding camera coordinate system P A , in, is the inverse of the intrinsic parameter matrix of a second depth camera, d A is a depth value corresponding to a starting point in a second depth camera; Point P in the camera coordinate system of the second depth camera A P converted to the camera coordinate system of the first depth camera A ′, P A ′=R A P A +T A Among them, R A is the rotation matrix from the first depth camera to the second depth camera, T A is the translation matrix from the first depth camera to the second depth camera; The first depth camera coordinate P A ′ is converted to point p′ in the corresponding pixel coordinate system A , p′ A =KP A ′ / d Wherein, K is the intrinsic parameter matrix of the first depth camera, and d is the depth value corresponding to a starting point in the first depth camera; The pixel coordinate p of the starting point of the travel route in another second depth camera image is B Mapped to p′ in the pixel coordinate system of the first depth camera in the same way as above B .

5. The method for measuring the path spacing between rows of closed-row crops according to claim 1, characterized in that: The step of calculating the distance between two starting points mapped in the first depth camera further comprises: p′ A The corresponding point P in the camera coordinate system A ′ is (X A , Y A , Z A ), p′ B The corresponding point P in the camera coordinate system B ′(X B , Y B , Z B ), calculate P A ′ and P B ′ as the distance between the two starting points, 6. The method for measuring the path spacing between rows of closed-row crops according to any one of claims 1 to 5, characterized in that: The second image is an image including the vicinity of crop roots, and the first image is a global image including the front of the agricultural equipment.

Images