Robot autonomous targeting system and method based on odometer-crop canopy feature fusion

Abstract

The present invention discloses an odometer-crop canopy feature fusion robot autonomous targeting system and method, and relates to the field of agricultural robots. The robot canopy autonomous targeting system includes an autonomous navigation chassis system and a canopy visual capture system. Through the canopy multi-organ feature solidification model and the odometer navigation primary coarse positioning method, the robot is guided to the primary positioning movement of the target canopy position and automatically generates random targeting sampling and plant-by-plant targeting sampling operation mode decisions; then, the distance between the robot and the target canopy center is calculated by the segmented canopy point cloud center capture positioning or the continuous canopy three-dimensional point cloud density comparison positioning method; finally, the robot is guided to move to the target position by the target point position correction method. The present invention improves the accuracy of the robot's autonomous targeting of the plant canopy.

Description

Technical Field

[0001] The present invention relates to the field of agricultural robots, and in particular to an odometer-crop canopy feature fusion robot autonomous targeting system and method. Background Art

[0002] In recent years, agricultural robots in my country have developed rapidly, and agricultural robots such as inspection, picking and plant protection are widely used in field operations. Autonomous and precise targeting of crop canopies is a prerequisite for ensuring high-quality and efficient operation of agricultural robots. Traditional infrared detection technology is greatly affected by environmental factors, has a short measurement distance and weak anti-interference ability, and has large errors in canopy positioning; laser radar detection technology has limited perception of plant objects, which can easily lead to misjudgment and missed judgments; and image segmentation based on vision is easily affected by natural light, background interference and other information to affect accuracy and stability. It is difficult to ensure the accuracy of canopy positioning of targets by visual detection alone. Summary of the invention

[0003] In view of the shortcomings in the prior art, the present invention provides a robot autonomous targeting system and method that integrates odometer and crop canopy characteristics, which effectively improves the robot's accuracy in capturing the target plant canopy.

[0004] The present invention achieves the above technical objectives through the following technical means.

[0005] A robot autonomous targeting method based on odometer-crop canopy feature fusion, specifically:

[0006] The single-frame RGB image of the plant canopy information is input into the canopy multi-organ feature solidification model, and the multi-level features of leaves, flowers and fruits are compared layer by layer in turn; if only a single organ of leaf is detected in the image, it is a piecewise canopy; if multiple organs of leaf-flower, leaf-fruit, flower-fruit or leaf-flower-fruit are detected, it is a continuous canopy;

[0007] The crop behavior canopy is segmented, the crop row-by-row targeted sampling operation mode is activated, and the target mileage X of subsequent sampling points is generated D :X D =X D0 +nR, n=1,2,3,4..., and X D ≤S; R is the current crop row spacing, X D0 is the current mileage of the depth camera, S is the length of the crop row,

[0008] The crop behavior is a continuous canopy, the crop row random target sampling operation mode is activated, and the target mileage X of the subsequent sampling points of the current crop row is randomly generated D : Among them, X2 is the mileage parameter of servo hub motor I, X3 is the mileage parameter of servo hub motor II, and L is the length of the vehicle body;

[0009] By the target mileage X D The current distance of the depth camera is X D0 The horizontal distance difference generates the number of movement circles N of the servo hub motor I and the servo hub motor II;

[0010] If the robot has not completed N circles, and X D0 =X D , then stop moving; if the robot completes N circles and X D0 =X D , then stop moving; if the robot completes N circles and X D0 ≠X D , then stop the movement;

[0011] Capture and locate the center of the canopy point cloud: determine whether the robot is at the target position based on the center of the canopy and the camera field of view. If the center of the canopy is offset from the camera field of view, the control system will drive the robot to continue walking and correct the target position, otherwise it will perform the target operation.

[0012] Comparison and positioning of point cloud density of continuous canopy: The control system performs online comparison between the three-dimensional depth point cloud cluster A1 containing only plant organs and the three-dimensional depth point cloud cluster A0 of the entire canopy: If P<P0, the control system will drive the robot to continue walking and correct the target position, otherwise it will perform the target operation, where P0 is a preset value.

[0013] A further technical solution is that the canopy multi-organ feature solidification model is as follows: the leaves, flowers and fruits of the entire crop growth cycle are manually marked with morphological and color multi-features and made into a training set, the leaves in the training set are classified as primary features, flowers as secondary features, and fruits as tertiary features, which are implanted into the semantic segmentation model in the order of one-two-three features for successive training, and the leaves, flowers and fruits obtained through layer-by-layer training are weightedly fused through feature maps to obtain the canopy multi-organ feature solidification model.

[0014] In a further technical solution, the current mileage of the depth camera is X D0 satisfy:

[0015] In a further technical solution, the current crop row planting spacing R satisfies: Where X D1 is the mileage parameter of the initial position, and P is the preset number of plants.

[0016] In a further technical solution, the number of movement circles N satisfies: Where C is the circumference of the servo hub motor.

[0017] A further technical solution is that the target position correction is specifically as follows:

[0018] The center of the depth camera's field of view and the position of the target are transformed from the camera to the world coordinate system, and the distance difference ΔX between the center of the depth camera's field of view and the position of the target on the X-axis of the depth camera coordinate system is calculated. The robot is driven to continue moving forward until the servo hub motor I and the servo hub motor II complete the number of k movements and stop, where k is: Where C is the circumference of the servo hub motor.

[0019] According to a further technical solution, the process of determining the center position of the slice canopy is as follows:

[0020] By aligning the color-depth coordinates of the depth camera, a three-dimensional depth point cloud cluster containing only plant organs is obtained. The control system clusters the three-dimensional depth point cloud to extract the horizontal-vertical center lines. The intersection of the extracted horizontal-vertical center lines is the center position of the segmented canopy.

[0021] A robot autonomous targeting system integrating odometer and crop canopy characteristics, comprising:

[0022] The servo hub motor I and the servo hub motor II are respectively installed at the bottom of the longitudinal support column at the front of the vehicle body;

[0023] The depth camera is fixed in the middle of the horizontal support column under the vehicle body, and is L / 2 away from the front and rear ends of the vehicle body;

[0024] Control system, communicates with servo hub motor I, servo hub motor II and depth camera

[0025] The beneficial effects of the present invention are as follows: in view of the outstanding problem of poor accuracy of agricultural robots in irregular canopies under complex field environments, the present invention solves the problem of the robot's judgment of the segmented or continuous morphology of the current crop row canopy and the real-time update of the crop spacing of the segmented canopy through a canopy multi-organ feature solidification model and an odometer navigation first-level coarse positioning method, guides the robot to the first-level positioning movement of the target canopy position and automatically generates random target sampling and plant-by-plant target sampling operation mode decisions; then, by capturing and positioning the center of the segmented canopy point cloud or by comparing and positioning the density of the continuous canopy three-dimensional point cloud, it is verified whether the robot is located at the center of the target canopy; and finally, by the target point position correction method, the robot is accurately moved and positioned to the center of the target canopy. The present invention solves the problems of poor positioning accuracy and low efficiency of the canopy position when the robot is operating, and improves the accuracy of the robot's autonomous targeting of the plant canopy. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a schematic diagram of the structure of the robot autonomous targeting system integrating odometer and crop canopy characteristics of the present invention;

[0027] Figure 2 This is a schematic diagram of the robot walking between rows according to the present invention;

[0028] Figure 3 This is a schematic diagram of the sliced ​​canopy of the present invention;

[0029] Figure 4 This is a schematic diagram of the continuous canopy layer of the present invention;

[0030] Figure 5 This is a schematic diagram of target position correction according to the present invention;

[0031] In the figure, 1. vehicle body, 2. driven wheel I, 3. driven wheel II, 4. servo hub motor I, 5. servo hub motor II, 6. camera bracket, 7. depth camera, 8. laser radar, 9. plant. DETAILED DESCRIPTION

[0032] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, but the protection scope of the present invention is not limited thereto.

[0033] The present invention is suitable for inspection, picking and targeting operations of plant protection robots, and is described below using disease inspection of ridge-planted strawberries as an example.

[0034] like Figure 1 As shown, the robot autonomous targeting system with odometer-crop canopy feature fusion of the present invention includes an autonomous navigation chassis system and a canopy visual capture system. The autonomous navigation chassis system includes a vehicle body 1, driven wheels Ⅰ2, driven wheels Ⅱ3, servo hub motor Ⅰ4, servo hub motor Ⅱ5 and laser radar 8. The driven wheels Ⅰ2 and Ⅱ3 are respectively installed at the bottom of the longitudinal support column at the rear of the vehicle body 1, the servo hub motor Ⅰ4 and servo hub motor Ⅱ5 are respectively installed at the bottom of the longitudinal support column at the front of the vehicle body 1, and the laser radar 8 is installed in the middle of the front lateral support of the vehicle body 1. The laser radar 8 is used to ensure that the robot walks in a straight line. The canopy visual capture system includes a camera bracket 6 and a depth camera 7. The two ends of the camera bracket 6 are respectively fixed in the middle of the lateral support column below the vehicle body 1, and the camera bracket 6 is L / 2 away from the front and rear ends of the vehicle body 1. The depth camera 7 is fixed in the middle of the camera bracket 6. The servo hub motor I4, the servo hub motor II5, the laser radar 8 and the depth camera 7 are all connected to the control system of the vehicle body 1 to coordinate and control the robot's autonomous target operation.

[0035] The robot autonomous targeting method of the invention integrating odometer and crop canopy features comprises a continuous / fragmented canopy self-judgment method, an odometer navigation primary coarse positioning method and a canopy visual capture secondary precise positioning method.

[0036] like Figure 3 , 4As shown in the figure, the method for self-judgment of continuous / segmented canopy is as follows:

[0037] Step 1: Manually mark the leaves, flowers and fruits of the crop growth cycle with morphology and color features and prepare a training set. The leaves in the training set are classified as primary features, flowers as secondary features, and fruits as tertiary features, see Table 1; the features are sequentially implanted into the DeepLab V3+ semantic segmentation model in the order of one-two-three features for successive training. The leaves (primary features), flowers (secondary features) and fruits (tertiary features) obtained through layer-by-layer training are weightedly fused through feature maps to obtain a canopy multi-organ feature solidification model, which includes the morphological and color features of leaves, flowers and fruits;

[0038] Table 1

[0039]

[0040] Step 2. When the robot is initially operating, the control system controls the depth camera 7 to scan a single-frame RGB image of the plant canopy information, and inputs it into the canopy multi-organ feature solidification model, and performs layer-by-layer comparison of multi-level features of leaves, flowers and fruits in turn; if the image only detects a single organ, leaf, the control system determines it to be a segmented canopy; if the image detects multiple organs, such as leaf-flower, leaf-fruit, flower-fruit or leaf-flower-fruit, the control system determines it to be a continuous canopy.

[0041] like Figure 2 , 3 As shown in 4, the first-level rough positioning method of odometer navigation has the following operation process:

[0042] Step 1: When the vehicle 1 is moving forward, the control system records the mileage parameters fed back by the servo wheel hub motor Ⅰ4 and the servo wheel hub motor Ⅱ5 in real time, and calculates the current mileage X of the depth camera 7 in real time through the spatial horizontal position relationship between the depth camera 7 and the vehicle 1. D0 , where X D0 for:

[0043]

[0044] In formula (1), X2 is the mileage parameter of the servo hub motor Ⅰ4, X3 is the mileage parameter of the servo hub motor Ⅱ5, and L is the length of the vehicle body 1;

[0045] The control system determines that the crop behavior is a segmented canopy and executes step two; the control system determines that the crop behavior is a continuous canopy and executes step three.

[0046] Step 2: The crop behavior is segmented into canopies. The control system drives the vehicle 1 to move and controls the depth camera 7 to scan the canopy of the plants 9. When the number of scanned plants 9 meets the preset value, the vehicle 1 stops moving and calculates the current crop row spacing R, where R is:

[0047]

[0048] In formula (2), X D1 is the mileage parameter of the initial position, and P is the preset number of plants;

[0049] According to the depth camera 7 current mileage X D0 The control system activates the crop row-by-row targeted sampling operation mode (this operation mode is the existing technology) and generates the target mileage X of the subsequent sampling points. D , X D =X D0 +nR, n=1, 2, 3, 4...; and X D ≤S, where S is the length of the crop row.

[0050] Step 3: Crop behavior canopy, according to the current mileage of the depth camera 7 X D0 , the control system activates the random target sampling operation mode of the crop row and randomly generates the target mileage X of the subsequent sampling points of the current crop row D ,

[0051] The crop row random target sampling operation mode is: the robot is based on the distance between the current position and the end of the crop row, that is, The control system randomly generates the position coordinates of the next sampling point.

[0052] Step 4: The control system calculates the target range X D Current mileage with depth camera 7X D0 The horizontal distance difference is calculated and the number of movement circles N of the servo hub motor Ⅰ4 and the servo hub motor Ⅱ5 is generated, where N is:

[0053]

[0054] In formula (3), C is the circumference of the servo hub motor.

[0055] Step 5: The control system drives the servo wheel hub motor I4 and the servo wheel hub motor II5 to move toward the target point (target mileage end point) at the set speed V, and calculates the current mileage X of the depth camera 7 in real time during the walking process. D0 and the target mileage X D Perform online comparison; if the robot has not completed N circles, and X D0 =X D , then stop moving; if the robot completes N circles and X D0 =X D , then stop moving; if the robot completes N circles and X D0 ≠X D, then stop moving. In this embodiment, L = 1m, S = 100m, C = 50cm, V = 1m / s.

[0056] The canopy visual capture secondary precise positioning method includes the segmented canopy point cloud center capture positioning sub-method, the continuous canopy three-dimensional point cloud density comparison positioning sub-method and the target chassis correction sub-method. Specifically:

[0057] The method for capturing and positioning the center of the sliced ​​canopy point cloud is as follows: the vehicle body 1 is at the first-level rough positioning of the odometer (near the target point), and the control system controls the depth camera 7 to scan the canopy information to obtain a single-frame two-dimensional RGB image. And the 3D depth point cloud cluster of the canopy as a whole ε is the horizontal field of view of the depth camera, is the vertical field of view of the depth camera; The image is input into the canopy multi-organ feature solidification model, and by comparing the image features of leaves, flowers, and fruits, similar pixels are retained to obtain a two-dimensional RGB image containing only the nine organs of the plant. By aligning the color (C1)-depth (A0) coordinates of the depth camera 7, a 3D depth point cloud cluster containing only the organs of the plant 9 is obtained. The control system clusters the lateral-longitudinal centerline extraction of the three-dimensional depth point cloud (the extraction method is the existing technology), and the intersection of the extracted lateral-longitudinal centerlines is the center position of the segmented canopy; if the center position of the segmented canopy is offset from the center of the camera's field of view, the control system will drive the robot to continue walking and correct the target position; if the center of the canopy coincides with the center of the camera's field of view, the robot is at the target position and performs the target operation.

[0058] The method of comparing the density of the continuous canopy point cloud and positioning the sub-method: the vehicle body 1 is at the first-level rough positioning of the odometer (near the target point), and the same method of capturing and positioning the center of the segmented canopy point cloud is used to obtain a three-dimensional depth point cloud cluster containing only the 9 organs of the plant. 3D depth point cloud cluster with canopy as a whole The control system performs an online comparison of A1 and A0: If P<P0 (preset value), the control system will drive the robot to continue walking and correct the target position; if P≥P0 (preset value), the robot is located at the target position and performs the target operation.

[0059] like Figure 5 As shown, for the target position correction method, the control system converts the center of the field of view of the depth camera 7 and the target position into the camera-world coordinate system, calculates the real distance difference ΔX of the X-axis of the depth camera coordinate system, and then drives the robot to continue moving forward at the set speed V until the servo hub motor I4 and the servo hub motor II5 complete the number of movement circles k and stop, where k is:

[0060]

[0061] The embodiments are preferred implementations of the present invention, but the present invention is not limited to the above-mentioned implementations. Any obvious improvements, substitutions or modifications that can be made by those skilled in the art without departing from the essential content of the present invention belong to the protection scope of the present invention.

Claims

1. A robot autonomous targeting method based on odometer-crop canopy feature fusion, characterized by: The single-frame RGB image of the plant canopy information is input into the canopy multi-organ feature solidification model, and the multi-level features of leaves, flowers and fruits are compared layer by layer in turn; if only a single organ of leaf is detected in the image, it is a piecewise canopy; if multiple organs of leaf-flower, leaf-fruit, flower-fruit or leaf-flower-fruit are detected, it is a continuous canopy; The crop behavior canopy is segmented, the crop row-by-row targeted sampling operation mode is activated, and the target mileage X of subsequent sampling points is generated D :X D =X D0 +nR, n=1,2,3,4..., and X D ≤S; R is the current crop row spacing, X D0 is the current mileage of the depth camera, S is the length of the crop row, The crop behavior is a continuous canopy, the crop row random target sampling operation mode is activated, and the target mileage X of the subsequent sampling points of the current crop row is randomly generated D : Among them, X2 is the mileage parameter of servo hub motor I, X3 is the mileage parameter of servo hub motor II, and L is the length of the vehicle body; By the target mileage X D The current distance of the depth camera is X D0 The horizontal distance difference generates the number of movement circles N of the servo hub motor I and the servo hub motor II; If the robot has not completed N circles, and X D0 =X D , then stop moving; if the robot completes N circles and X D0 =X D , then stop moving; if the robot completes N circles and X D0 ≠X D , then stop the movement; Capture and locate the center of the canopy point cloud: determine whether the robot is at the target position based on the center of the canopy and the camera field of view. If the center of the canopy is offset from the camera field of view, the control system will drive the robot to continue walking and correct the target position, otherwise it will perform the target operation. Comparison and positioning of point cloud density of continuous canopy: The control system performs online comparison between the three-dimensional depth point cloud cluster A1 containing only plant organs and the three-dimensional depth point cloud cluster A0 of the entire canopy: If P<P0, the control system will drive the robot to continue walking and correct the target position, otherwise it will perform the target operation, where P0 is a preset value.

2. The robot autonomous targeting method according to claim 1, characterized in that: The canopy multi-organ feature solidification model is as follows: leaves, flowers and fruits of the whole growth cycle of crops are manually marked with morphological and color multi-features and prepared into a training set, the leaves in the training set are classified as primary features, flowers as secondary features, and fruits as tertiary features, and the features are sequentially implanted into the semantic segmentation model in the order of one-two-three features for successive training, and the leaves, flowers and fruits obtained through layer-by-layer training are weightedly fused through feature maps to obtain the canopy multi-organ feature solidification model.

3. The robot autonomous targeting method according to claim 1, characterized in that: The current mileage of the depth camera is X D0 satisfy:

4. The robot autonomous targeting method according to claim 1, characterized in that: The current crop row planting spacing R satisfies: Where X D1 is the mileage parameter of the initial position, and P is the preset number of plants.

5. The robot autonomous targeting method according to claim 1, characterized in that: The number of movement circles N satisfies: Where C is the circumference of the servo hub motor.

6. The robot autonomous targeting method according to claim 1, characterized in that: The target position correction is specifically as follows: The center of the depth camera's field of view and the position of the target are transformed from the camera to the world coordinate system, and the distance difference ΔX between the center of the depth camera's field of view and the position of the target on the X-axis of the depth camera coordinate system is calculated. The robot is driven to continue moving forward until the servo hub motor I and the servo hub motor II complete the number of k movements and stop, where k is: Where C is the circumference of the servo hub motor.

7. The robot autonomous targeting method according to claim 1, characterized in that: The process of determining the center position of the slice canopy is as follows: By aligning the color-depth coordinates of the depth camera, a three-dimensional depth point cloud cluster containing only plant organs is obtained. The control system clusters the three-dimensional depth point cloud to extract the horizontal-vertical center lines. The intersection of the extracted horizontal-vertical center lines is the center position of the segmented canopy.

8. A system for implementing the robot autonomous targeting method according to any one of claims 1 to 7, characterized in that: include: The servo hub motor I and the servo hub motor II are respectively installed at the bottom of the longitudinal support column at the front of the vehicle body; The depth camera is fixed in the middle of the horizontal support column under the vehicle body, and is L / 2 away from the front and rear ends of the vehicle body; The control system communicates with the servo hub motor I, servo hub motor II and the depth camera.

Images