Method for measuring the path distance between rows of a row crop

Abstract

This application discloses a method for measuring the path spacing between rows of closed-row crops. The method includes the following steps: acquiring crop image information from different perspectives using a first depth camera mounted on the top of the agricultural equipment and two second depth cameras mounted at the two front wheels, wherein the fields of view of the two second depth cameras overlap with those of the first depth camera; extracting the boundary lines between the soil and crops from the two second images from the second depth cameras, thereby determining the corresponding travel path of the front wheels; mapping the starting points of the travel paths in the two second images onto the first image of the first depth camera based on the positional relationship between the second depth cameras and the first depth camera; and calculating the distance between the two starting points mapped to the first depth camera. This method has high measurement accuracy and strong stability, and can be used for path calculation and navigation guidance in inter-row operations during the middle and late stages of crop growth, facilitating timely adjustment of wheel spacing by agricultural equipment based on the measured distance, and enabling flexible and efficient autonomous operation.

Description

Technical Field

[0001] This application belongs to the field of smart agriculture technology, specifically relating to a method for measuring the path spacing between rows of closed-row crops. Background Technology

[0002] Food security is a crucial foundation of national security, and the application of intelligent agricultural equipment is one of the key technologies for achieving sustainable agricultural production and ensuring food security. In precision field operations, agricultural equipment needs to be able to accurately navigate and move along the paths between crop rows. However, these paths are usually determined by the crop planting and growth patterns, and their spacing may vary due to factors such as terrain, planting errors, and crop type.

[0003] Currently, agricultural equipment often relies on fixed satellite trajectories or manual navigation for row guidance. This approach suffers from low efficiency, high cost, and inability to adapt to complex field environments. Advances in technologies such as computer vision and artificial intelligence have made it possible to detect work paths between rows. This method not only improves the accuracy and efficiency of navigation but also provides reliable technical support for measuring the path spacing of agricultural equipment as it moves between crop rows.

[0004] However, in the later stages of crop growth, the branches and leaves of adjacent rows of crops intertwine, covering the gaps in the ground between the rows and forming a continuous, closed green area known as row closure. In this situation, the traditional visual navigation method that utilizes the natural line characteristics of crop rows in the field becomes interfered with, making it impossible to obtain clear inter-row path information, thus affecting the measurement of the spacing between crop rows. If the wheel track of agricultural intelligent equipment does not match the spacing between the inter-row paths of closed-row crops, it will make it difficult for the agricultural equipment to provide effective inter-row guidance, thereby affecting the efficiency and accuracy of field operations, and may even damage the normal growth of crops. Summary of the Invention

[0005] To address the aforementioned problems, a method for measuring the path spacing between rows of closed-row crops has been invented. The method includes the following steps:

[0006] A first depth camera is installed on the top of the agricultural equipment, and two second depth cameras are installed at the two front wheels respectively to collect crop image information from different perspectives. The second depth cameras have overlapping fields of view with the first depth camera. The image collected by the first depth camera is recorded as the first image, and the images collected by the two second depth cameras are recorded as the second images.

[0007] The boundary between soil and crops in the two second images from the second depth camera are extracted respectively, and then the corresponding front wheel travel path is determined.

[0008] Based on the positional relationship between the second depth camera and the first depth camera, the starting points of the travel routes in the two second images are respectively mapped onto the first image of the first depth camera;

[0009] The distance between the two starting points mapped in the first depth camera is the spacing of the inter-row path of the closed-row crop.

[0010] Preferably, the step of extracting the boundary line between soil and crops from the two second images of the second depth camera and then determining the corresponding front wheel travel path specifically includes:

[0011] Perform the same operation as follows for each second image:

[0012] The second image is binarized to obtain two binarized images;

[0013] Several lines are drawn across the binarized image at preset intervals, wherein the direction of the lines is perpendicular to the direction of the crop arrangement.

[0014] Read the pixel value of each line sequentially. If the pixel value of a pixel is different from the pixel value of the next pixel, select that pixel as a feature point.

[0015] Calculate the center points corresponding to the feature points at both ends of the soil region on all lines, and fit all center points to obtain the travel path of the front wheel corresponding to the second depth camera in the second image. Preferably, the step of mapping the starting point of the travel path in the two second depth camera images to the first depth camera image according to the positional relationship between the second depth camera and the first depth camera specifically includes:

[0016] Based on the intrinsic parameter matrices of the three depth cameras and the rotation and translation matrices of the two second depth cameras to the first depth camera, the starting points of the travel routes in the two second images are mapped to the first image of the first depth camera through coordinate transformation.

[0017] Preferably, the step of mapping the starting points of the travel routes in the two second images to the first image of the first depth camera through coordinate transformation specifically includes:

[0018] Find the pixel coordinates p of the starting point of the travel path in one of the second depth camera images. A Transform to the corresponding camera coordinate system P A ,

[0019]

[0020] in, The inverse of the intrinsic parameter matrix of a second depth camera, d AThis is the depth value corresponding to the starting point in a second depth camera;

[0021] Point P in the camera coordinate system of the second depth camera A Transform P to the camera coordinate system of the first depth camera A ′,

[0022] P A ′=R A P A +T A

[0023] Among them, R A It is the rotation matrix from the first depth camera to the second depth camera, T A It is the translation matrix from the first depth camera to the second depth camera;

[0024] P in the first depth camera coordinate system A Convert ′ to point p′ in the corresponding pixel coordinate system A ,

[0025] p′ A =KP A ′ / d

[0026] Where K is the intrinsic parameter matrix of the first depth camera, and d is the depth value corresponding to a starting point in the first depth camera;

[0027] The pixel coordinates p of the starting point of the travel path in another second depth camera image. B Map p′ to the pixel coordinate system of the first depth camera using the same method described above. B .

[0028] Preferably, the step of calculating the distance between two starting points mapped in the first depth camera further includes:

[0029] p′ A Point P in the corresponding camera coordinate system A ′ for (X A Y A Z A ), p′ B Point P in the corresponding camera coordinate system B ′(X B Y B Z B ), calculate P A ′ and P B The Euclidean distance D between the two starting points is taken as the distance between them.

[0030]

[0031] Preferably, the second image is an image including the area near the roots of the crop, and the first image is a global image including the area in front of the agricultural equipment.

[0032] This method has high measurement accuracy and strong stability. It can be used for path calculation and navigation guidance in the mid-to-late stages of crop growth, and facilitates timely adjustment of wheel spacing for agricultural equipment based on the measured distance, enabling flexible and efficient autonomous operation. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0034] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 A flowchart illustrating a method for measuring the spacing between rows of field crops provided in an embodiment of this application;

[0036] Figure 2 Examples of different road condition datasets and annotation results provided in embodiments of this application;

[0037] Figure 3 Extract example images for the travel route;

[0038] Figure 4 Define a diagram for the camera coordinate system transformation relationship;

[0039] Figure 5 This is an example diagram of the travel routes of the first depth camera and the second depth camera in this embodiment;

[0040] Figure 6 This is an example diagram of the measurement of the distance between travel routes in this embodiment;

[0041] Figure 7 This is an example diagram of the measurement of the distance between flight lines in this embodiment. Detailed Implementation

[0042] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0043] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0044] This embodiment proposes a method for measuring the spacing between field paths and flight routes, combined with... Figure 1 The method includes the following steps:

[0045] The agricultural equipment uses a first depth camera mounted on top and two second depth cameras mounted on the two front wheels to collect crop images from different perspectives. The fields of view of the two second depth cameras overlap with those of the first depth camera.

[0046] In this embodiment, the two second depth cameras are depth cameras located at the two front wheels of the agricultural equipment, each mounted on a separate bracket and facing the front of the wheel. Since the top leaves of crops grow in an alternating pattern after the rows close, but the roots still exhibit a linear distribution, mounting the two second depth cameras at the front wheels is suitable. Their installation positions can be adjusted appropriately according to the crop's growth status to ensure the acquisition of characteristic information about the inter-row paths, while avoiding obstruction by crop leaves.

[0047] The first depth camera is mounted on top of the agricultural equipment, at the front of the equipment, and its task is to capture images of the farmland in front of the vehicle. To facilitate subsequent coordinate transformation, the shooting angle needs to be adjusted to ensure that the field of view of this camera overlaps with that of the two second depth cameras located at the wheels.

[0048] The boundary between soil and crops in the two second images from the second depth camera are extracted respectively, and then the corresponding front wheel travel path is determined.

[0049] In this embodiment, the second images captured by the two second depth cameras are input into a pre-trained deep neural network in the form of video frames. During the sample annotation stage before training, we distinguish between soil and crops using the line connecting the roots of crop rows as the boundary. Because the shapes of crop branches and leaves are extremely irregular, we consider all possible weeds, some branches and leaves, and stones on the road surface as soil, which facilitates the rapid extraction of subsequent road feature points. Part of the dataset and its annotation results are shown below. Figure 2 As shown, where Figure 2 a, 2b, and 2c represent road conditions at three different times. Figure 2 d, 2e, and 2f are the labels of the corresponding images.

[0050] The deep neural network described employs a semantic segmentation model, such as U-Net, SegNet, PSPNet, and DeepLab, etc., and the specific choice is not limited in this invention. A general semantic segmentation architecture can be considered as an encoder-decoder network.

[0051] To eliminate potential hole regions in the image, we first identify all connected regions in the output image and select the region with the largest area as the maximum connected region. We then set the pixel values ​​of all pixels within this maximum connected region to 1 and the pixel values ​​of all other locations to 0, resulting in an image where only the path region is the target foreground. After binarization, the boundaries of soil and crops in the resulting binary image will be more complete. The white areas within the boundaries represent soil, i.e., the area where vehicles can drive, while the black areas outside the boundaries represent crops, i.e., the area where vehicles cannot drive. Utilizing these features can improve the robustness of field path extraction results.

[0052] Draw several horizontal lines that run through the binary image every 10 rows, with the lines perpendicular to the direction in which the crops are arranged. Read the pixel value of each line in sequence. If the pixel value of a point is different from the pixel values ​​of the pixels on the left and right sides of the line where the point is located, it means that the pixel is on the boundary, and the pixel is selected as a feature point.

[0053] After extracting all feature points on the left and right boundaries using the above process, the two feature points on the same line are averaged and weighted to generate the midpoint of the path in the drivable area. At this point, the weights of the left and right sides are both 0.5. The path midpoint is then fitted using the least squares fitting method to obtain the travel route in the drivable area. The result is as follows. Figure 3 As shown.

[0054] Based on the positional relationship between the second depth camera and the first depth camera, the starting points of the travel routes in the two second images are respectively mapped onto the first image of the first depth camera;

[0055] In this embodiment, to achieve more comprehensive environmental perception, the travel routes extracted by the second depth cameras at the two wheels are mapped to the first depth camera on top of the agricultural equipment. Since the positions of the two second depth cameras at the wheels and the position of the first depth camera on top of the agricultural equipment are relatively fixed and have overlapping fields of view, coordinate transformation technology can be used to map feature points between images from different cameras. However, the above process requires the intrinsic and extrinsic parameter matrices of the second and first depth cameras to be obtained in advance.

[0056] Each camera has its own intrinsic parameter matrix, which is used to transform pixels from the pixel coordinate system to the camera coordinate system.

[0057] The specific parameters in the intrinsic parameter matrix are obtained using MATLAB's camera calibration toolbox. First, a set of calibration images needs to be prepared. These images should have different angles and positions to cover the entire field of view of the camera. Then, after opening the camera calibration application in MATLAB, the program will automatically detect corner points and calculate intrinsic parameter information such as focal length f. x f y Principal point position c x c y This allows for the extraction of the intrinsic parameter matrix of any camera. The intrinsic parameter matrix is ​​usually represented as K, and its form is shown below.

[0058] The extrinsic matrix includes the rotation matrix. The translation matrix and the translation matrix are used to transform a point from one camera coordinate system to another, such as... Figure 4 As shown, the origin O of the two camera coordinate systems O-XYZ is the optical center of the depth camera, the Z-axis is the optical axis of the depth camera, its positive direction points to the shooting scene, the X-axis is parallel to the camera mirror, and its direction is horizontal to the right when viewed along the shooting direction of the camera. The X-axis, Y-axis and Z-axis satisfy the right-hand rule.

[0059] The rotation matrix describes the relative orientation relationship between two camera coordinate systems. We define the mounting tilt angles θ1 and θ2 of camera 1 and camera 2. The two cameras rotate only on the x-axis. At this time, based on the difference in tilt angle between camera 2 and camera 1 in the x-direction, θ = θ2 - θ1, we can obtain the rotation matrix R, as shown below.

[0060]

[0061] The translation matrix describes the relative positional relationship between two camera coordinate systems. We define the offset Δx of camera 1 relative to camera 2 in the x-direction, the offset Δy of camera 1 relative to camera 2 in the y-direction, and the offset Δz of camera 1 relative to camera 2 in the height (z-direction). The translation matrix T can then be represented as a three-dimensional vector.

[0062]

[0063] Find the pixel coordinates p of the starting point of the travel path in one of the second depth camera images. A Transform to the corresponding camera coordinate system P A ,

[0064]

[0065] in, The inverse of the intrinsic parameter matrix of a second depth camera, d A This is the depth value corresponding to the starting point in a second depth camera;

[0066] Point P in the camera coordinate system of the second depth camera A Transform P to the camera coordinate system of the first depth camera A ′,

[0067] P A ′=R A P A +T A

[0068] Among them, R A It is the rotation matrix from the first depth camera to the second depth camera, T A It is the translation matrix from the first depth camera to the second depth camera;

[0069] P in the first depth camera coordinate system A Convert ′ to point p′ in the corresponding pixel coordinate system A ,

[0070] p′ A =KP A ′ / d

[0071] Where K is the intrinsic parameter matrix of the first depth camera, and d is the depth value corresponding to a starting point in the first depth camera;

[0072] The pixel coordinates p of the starting point of the travel path in another second depth camera image. B Map p′ to the pixel coordinate system of the first depth camera using the same method described above. B .

[0073] like Figure 5 As shown, Figure 5 a is the first image captured by the first depth camera. Figure 5 b and Figure 5 c represents the second image captured by the two second depth cameras.

[0074] Calculate the distance between the two starting points mapped in the first depth camera.

[0075] In this embodiment, the spatial distance between the two starting points in the first image in the camera coordinate system is calculated using the Euclidean distance formula, which is the actual distance between the two points.

[0076] like Figure 6 As shown, the red and green dots represent the starting points of two navigation baselines, and the black line connecting the two points represents the distance to be measured. Substituting these two points into the following Euclidean distance formula, the distance between the two starting points is calculated.

[0077]

[0078] The spatial distance between these two points is 1030, with the unit being millimeters (mm). This unit is then converted to meters (m), and the converted value of 1.03m is displayed in the image.

[0079] To further verify the effectiveness of the algorithm, a region with relatively uniform row spacing was selected, and the path spacing of the four central crop rows was continuously measured in 250 frames of images. The measurement results show that the spacing values ​​are consistently around 1 meter, which is consistent with the actual row spacing in planting. Figure 7 As shown, this verifies the rationality and stability of the algorithm.

[0080] This method, based on visual information perception and coordinate mapping technology, extracts the travel paths of the two front wheels through the collaborative work of a first depth camera and two second depth cameras with overlapping fields of view. The starting points of each path are then accurately mapped into the first depth camera image, and distance is measured. This method, utilizing multi-view cameras, can more comprehensively reflect the actual scene, avoiding the limitations of a single perspective, and is particularly effective in scenarios where crops are closing off during the later stages of growth. Real-time calculation of path spacing allows agricultural equipment with variable wheel spacing to flexibly adjust its wheel spacing in dynamic environments, ensuring accurate operating paths and stable navigation performance. For equipment without wheel spacing adjustment capabilities, effective adjustment measures can be taken when a significant change in path spacing is detected to avoid crushing crops or damaging equipment, thereby achieving efficient autonomous operation management.

[0081] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for measuring the path spacing between rows of closed-row crops, characterized in that... The method includes the following steps: The agricultural equipment uses a first depth camera mounted on top and two second depth cameras mounted on the two front wheels to collect crop images from different perspectives. The fields of view of the two second depth cameras overlap with those of the first depth camera. The image captured by the first depth camera is recorded as the first image, and the images captured by the two second depth cameras are recorded as the second images. The boundary between soil and crops in the two second images from the second depth camera are extracted respectively, and then the corresponding front wheel travel path is determined. Based on the positional relationship between the second depth camera and the first depth camera, the starting points of the travel routes in the two second images are respectively mapped onto the first image of the first depth camera; Specifically, this includes: based on the intrinsic parameter matrices of the three depth cameras and the rotation and translation matrices of the two second depth cameras to the first depth camera, mapping the starting points of the travel routes in the two second images to the first image of the first depth camera through coordinate transformation; The step of mapping the starting points of the travel routes in the two second images to the first image of the first depth camera through coordinate transformation specifically includes: Find the pixel coordinates p of the starting point of the travel path in one of the second depth camera images. A Transform to the corresponding camera coordinate system P A , in, The inverse of the intrinsic parameter matrix of a second depth camera, d A This is the depth value corresponding to the starting point in a second depth camera; Point P in the camera coordinate system of the second depth camera A Transform to the camera coordinate system of the first depth camera , Among them, R A It is the rotation matrix from the first depth camera to the second depth camera, T A It is the translation matrix from the first depth camera to the second depth camera; The coordinates of the first depth camera Convert to points in the corresponding pixel coordinate system , Where K is the intrinsic parameter matrix of the first depth camera, and d is the depth value corresponding to a starting point in the first depth camera; The pixel coordinates p of the starting point of the travel path in another second depth camera image. B Mapped to the pixel coordinate system of the first depth camera using the same method described above. The distance between the two starting points mapped in the first depth camera is the spacing of the inter-row path of the closed-row crop.

2. The method for measuring the path spacing between rows of closed-row crops according to claim 1, characterized in that, The step of extracting the boundary line between soil and crops from the two second images of the second depth camera, and then determining the corresponding front wheel travel path, specifically includes: Perform the same operation as follows for each second image: The second image is binarized to obtain two binarized images; Several lines are drawn across the binarized image at preset intervals, wherein the direction of the lines is perpendicular to the direction of the crop arrangement. Read the pixel value of each line sequentially. If the pixel value of a pixel is different from the pixel value of the next pixel, select that pixel as a feature point. Calculate the center points corresponding to the feature points at both ends of the soil area on all lines, and fit all center points to obtain the travel path of the front wheel corresponding to the second depth camera to which the second image belongs.

3. The method for measuring the spacing between rows of closed-row crops according to claim 1, characterized in that, The step of calculating the distance between the two starting points mapped in the first depth camera further includes: Points in the corresponding camera coordinate system For (X) A Y A Z A ), Points in the corresponding camera coordinate system (X B Y B Z B ),calculate and The Euclidean distance D is taken as the distance between the two starting points. 。 4. The method for measuring the path spacing between rows of closed-row crops according to any one of claims 1 to 3, characterized in that, The second image is an image that includes the area near the roots of the crop, and the first image is a global image that includes the area in front of the agricultural equipment.

Images