A precision agricultural robot operation method

Abstract

The present invention relates to the field of agricultural operation technology, specifically a precision operation agricultural robot operation method, comprising the following steps: image acquisition of target crops; identification of target features of target crops in images; extraction of depth information of identified target features, calculation of the position of target features; fusion calculation of the agricultural robot's own position and the target feature position; calculation and planning of the agricultural robot's motion planning and path planning, production of instruction sequences; control of the agricultural robot to move along a specific trajectory, and completion of specified actions. The present invention can realize efficient and simple logic pathways of software architecture and hardware architecture, and achieve more efficient and convenient effects in a case closer to real operation scenes; it can adapt to a variety of different scenes. In precision operation scenes, this solution can accurately identify the target features of crops and drive special operation arms to achieve specific operation actions.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural operations, and in particular to an operation method of a precision agricultural robot. Background Art

[0002] Existing agricultural operation equipment mainly operates mechanically and cannot meet the needs of precise operation. Existing agricultural operation equipment, such as rice transplanters, harvesters, sprayers, etc., all use large-scale mechanical methods to perform imprecise operations. Their operation vehicles can be located and guided by satellite signals such as RTK, but their operation equipment cannot perform precise operations based on the characteristics of the crops themselves. For example, fixed-point weeding, fixed-point spraying, directional picking, precise maintenance and other operation processes cannot be achieved. Summary of the invention

[0003] In order to overcome the defects in the prior art, the purpose of the present invention is to provide a precision agricultural robot operation method.

[0004] The object of the present invention is achieved through the following technical solutions:

[0005] A precision agricultural robot operation method, applied to an agricultural robot, comprises the following steps:

[0006] S1, collect images of target crops through visual sensors;

[0007] S2, using target recognition and detection algorithms to identify target features of target crops in the image;

[0008] S3, calculating or collecting global or local depth information, extracting depth information for the identified target feature, and calculating the position of the target feature;

[0009] S4, through the motion / position sensor, the agricultural robot's own position and the target feature position are integrated and calculated;

[0010] S5. Calculate and plan the action planning and path planning of the agricultural robot and produce instruction sequences according to the real-time updated data of the target feature position and the agricultural robot's own position;

[0011] S6. According to the motion planning and path planning of the agricultural robot, the agricultural robot moves along a specific trajectory and completes the specified action.

[0012] As a preferred technical solution, step S2 specifically includes the following steps:

[0013] S2-1, using a target detection algorithm to identify the location of the target feature;

[0014] S2-2, calculate the location of target features through target crop features and topology;

[0015] S2-3. Integrate the calculation results of multiple models, make comprehensive judgments, and calculate the maximum probability position of the target feature.

[0016] As a preferred technical solution, the target crop is a stem-leaf plant, the target feature is the top of the stem and leaf of the target crop, and the step S2-2 specifically includes the following steps:

[0017] S2-2-1, performing Gaussian filtering to reduce noise on the image, performing background threshold segmentation on the image, and extracting a relatively complete target crop image;

[0018] S2-2-2, performing dimensionality reduction processing on the acquired color target crop image, converting the image into a grayscale image, retaining the shape distribution information of the trunk and stems and leaves of the target crop in the original image, performing Canny operator edge detection, and obtaining a binary image of the edge of the target crop and the edge of the trunk of the stems and leaves;

[0019] S2-2-3, process the acquired binary image: perform straight line detection on the binary image, where the curved image is the leaf edge and the remaining image is the stem and leaf trunk, extract the straight line image as C1, and the remaining image as C2, and the pixel values ​​in images C1 and C2 are the edge information of the binary image;

[0020] S2-2-4, let C1(x,y) and C2(x,y) be the pixel points at the coordinate positions in the straight line image C1 and the residual image C2, judge C1(x,y) and C2(x,y), if C1(x,y)=C2(x,y), it means (x,y) is the leaf tip coordinate, if C1(x,y)≠C2(x,y), it means (x,y) is not the leaf tip coordinate, save all leaf tip coordinates {(x,y)};

[0021] S2-2-5. Convert the straight line image C1 into a vector, with the starting point being the leaf tip coordinate {(x,y)}, obtain the intersection coordinates {(x',y')} of all vectors, draw a circle with i pixels as the radius, and use each pixel in C1 as the center of the circle for judgment. If there are intersection coordinates (x',y') in the circle and there are more than n intersections, the intersections are considered to be clustered, and the average coordinates of all intersections in the circle are calculated (Σx / n,Σy / n). The coordinates are the center coordinates of the top of the stems and leaves of the target crop, that is, the position coordinates of the target feature; where i and n are both natural numbers greater than 0.

[0022] As a preferred technical solution, in step S4, the agricultural robot comprehensively calculates the three-dimensional coordinates of the agricultural robot and its operating actuators by integrating GPS RTK real-time differential positioning, IMU information, and wheel speed meter information, and calculates its position change in the world coordinate system.

[0023] As a preferred technical solution, in the steps S1-S6, the delay in the entire algorithm logic is eliminated through time synchronization technology.

[0024] The precision agricultural robot operation method provided by the present invention mainly has the following technical effects: 1. The present invention provides an efficient and simple software architecture and logic path, uses a visual sensor as the main feature recognition module, and uses a multi-sensor fusion method to perform position calculation. Through a variety of algorithms deployed on the computing unit, including a visual processing algorithm, a deep calculation algorithm, and a machine learning related model, the coordinate position of the target feature (including two-dimensional coordinates and three-dimensional coordinates) is extracted, and the motion planning of the terminal specialized robotic arm is realized through path planning and motion planning algorithms, which is closer to the real operation scene and achieves a more efficient and convenient effect; it can adapt to a variety of different scenarios. In the precision operation scenario, this solution can accurately identify the target characteristics of crops and drive the special operation arm to achieve specific operation actions. 2. The position of the target feature is inferred through the target crop characteristics and topology. This is based on the growth characteristics of the stem and leaf target crop (such as cotton). By identifying the crop trunk and stem and leaf distribution, the top growth position of the target crop is inferred. This logic identifies the crop trunk and stem and leaves in the image and converts them into vector graphs. By connecting the vector graphs, the starting point is obtained, which is the top of the crop. Compared with the traditional direct recognition of target features, this logic is less affected by image quality and takes into account the structural characteristics of the plant itself, effectively improving the recognition rate. 3. Time synchronization technology can be used to eliminate delays in the entire algorithm logic.

[0025] The concept, specific structure and effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of the operation method of the precision agricultural robot of the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not used to limit the present invention.

[0028] Example 1: Figure 1As shown, taking cotton topping operation as an example, the precision operation agricultural robot operation method provided by this patent includes the following steps:

[0029] S1. Image collection:

[0030] Use a binocular camera or a depth camera (visual sensor) to capture images of the cotton in the working area from a downward perspective.

[0031] S2. Cotton top identification:

[0032] S2-1, identify the cotton top position through target detection algorithm;

[0033] S2-2, infer the cotton top position through cotton leaf characteristics and topology;

[0034] S2-2-1, performing Gaussian filtering on the image to remove noise interference, improve image quality, perform background threshold segmentation on the cotton image, and extract a relatively complete cotton plant image;

[0035] S2-2-2, performing dimensionality reduction processing on the acquired color cotton image, converting the image into a grayscale image, retaining the main trunk and stem and leaf shape distribution information of the cotton in the original image, performing edge detection using the Canny operator, obtaining the cotton edge and stem and leaf main trunk image, achieving effective separation of the cotton and background image, and retaining only the cotton edge and stem and leaf main trunk information;

[0036] S2-2-3, process the obtained cotton edge binary image. According to the growth characteristics of cotton, the edge of cotton leaves is relatively curved, so the curved image should be the leaf edge, and the remaining image is the stem and leaf trunk. Perform straight line detection on the cotton edge binary image, extract the straight line image as C1, and the remaining image as C2. The pixel values ​​in images C1 and C2 are the edge information of the cotton edge binary image;

[0037] S2-2-4, let C1(x,y), C2(x,y) be the pixel points at the coordinate positions in the straight line image C1 and the residual image C2, judge C1(x,y) and C2(x,y), if C1(x,y)=C2(x,y), it means (x,y) is the leaf tip coordinate, if C1(x,y)≠C2(x,y), it means (x,y) is not the leaf tip coordinate, save all leaf tip coordinates {(x,y)};

[0038] S2-2-5. Convert the straight line image C1 into a vector, with the starting point being the leaf tip coordinate {(x,y)}, obtain the intersection coordinates {(x',y')} of all vectors, draw a circle with 5 pixels as the radius, and use each pixel in C1 as the center of the circle for judgment. If there are intersection coordinates (x',y') in the circle and there are more than n intersections, the intersections are considered to be clustered, and the average coordinates of all intersections in the circle are calculated (Σx / n,Σy / n), which are the center coordinates of the cotton top; n is a natural number greater than 0, and this parameter can be set.

[0039] S2-3. Integrate the results of multiple models, make comprehensive judgments, and calculate the maximum probability position of the cotton top.

[0040] S3. Calculation of cotton top position:

[0041] S3-1, calculate the depth map through the binocular camera or the depth camera;

[0042] S3-2, extracting the depth of the cotton top position identified in the previous step;

[0043] S3-3. Convert the cotton top position into three-dimensional coordinates.

[0044] S4, integrates IMU, RTK and wheel speed meter information, comprehensively calculates the cotton top position and the fuselage position, and calculates their position changes in the world coordinate system.

[0045] S5. Optimize the target sequence using one or more of the following optimization targets as criteria:

[0046] a. Shortest time;

[0047] b. Shortest path;

[0048] c. No mutual interference;

[0049] d. The best matching speed;

[0050] Finally, multiple movement paths of the robotic arms used for cotton topping are generated, and each robotic arm can operate in the most efficient way.

[0051] S6. Drive each robotic arm to complete the topping action.

[0052] Example 2: Taking weeding as an example, the precision agricultural robot operation method provided by this patent includes the following steps:

[0053] S1. Image collection:

[0054] Use a monocular, binocular camera or a depth camera to collect the characteristics of weeds from a downward perspective.

[0055] S2. Weed target feature recognition:

[0056] S2-1, identify the characteristic position of weed targets through target detection algorithm;

[0057] S2-2, infer the target feature position of weeds through weed features and topology;

[0058] S2-3. Integrate the results of multiple models, make comprehensive judgments, and calculate the maximum probability position of the weed target feature.

[0059] S3. Calculation of weed target feature position:

[0060] S3-1. Calculate the ground height;

[0061] S3-2, calculating the characteristic position of weed targets;

[0062] S3-3. Convert the characteristic position of the weed target into three-dimensional coordinates.

[0063] S4, integrates IMU, RTK and wheel speed meter information, comprehensively calculates the characteristic position of the weed target and the position of the aircraft, and calculates its position change in the world coordinate system.

[0064] S5. Optimize the target sequence using one or more of the following optimization targets as criteria:

[0065] a. Shortest time;

[0066] b. Shortest path;

[0067] c. No mutual interference;

[0068] d. The best matching speed;

[0069] The result is multiple movement paths for the robotic arms used for weeding, allowing each robotic arm to operate in the most efficient manner.

[0070] S6, driving the robotic arm to complete the weeding action.

[0071] The precision agricultural robot operation method provided by the present invention has the following technical effects:

[0072] 1. The present invention provides an efficient and simple software architecture and logic path, using visual sensors as the main feature recognition module, using multi-sensor fusion to calculate the position, extracting the coordinate position (including two-dimensional coordinates and three-dimensional coordinates) of the target feature through multiple algorithms deployed on the computing unit, including visual processing algorithms, deep calculation algorithms, and machine learning related models, and realizing the action planning of the terminal specialized mechanical arm through path planning and action planning algorithms, achieving a more efficient and convenient effect under the condition of being closer to the real operation scene; it can adapt to a variety of different scenes, and in the precise operation scene, this scheme can accurately identify the target features of crops and drive the special operation arm to achieve specific operation actions. 2. The position of the target feature is inferred through the target crop features and topology, which is based on the growth characteristics of the stem and leaf target crops (such as cotton) themselves. By identifying the crop trunk and stem and leaf distribution, the growth position of the top of the target crop is inferred. The logic identifies the crop trunk and stem and leaves in the image and converts them into vector graphs. By connecting the vector graphs, the starting point is obtained, which is the top of the crop. Compared with the traditional direct recognition of target features, this logic is less affected by image quality, and takes into account the structural characteristics of the plant itself, effectively improving the recognition rate. 3. Time synchronization technology can be used to eliminate delays in the entire algorithm logic.

[0073] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other.

Claims

1. A precision agricultural robot operation method, applied to an agricultural robot, characterized in that: The steps include: S1, collect images of target crops through visual sensors; S2, using target recognition and detection algorithms to identify target features of target crops in the image; S3, calculating or collecting global or local depth information, extracting depth information for the identified target feature, and calculating the position of the target feature; S4, through the motion / position sensor, the agricultural robot's own position and the target feature position are integrated and calculated; S5. Calculate and plan the action planning and path planning of the agricultural robot and produce instruction sequences according to the real-time updated data of the target feature position and the agricultural robot's own position; S6. According to the action planning and path planning of the agricultural robot, the agricultural robot moves along a specific trajectory and completes the specified action; The step S2 specifically includes the following steps: S2-1, using a target detection algorithm to identify the location of the target feature; S2-2, calculate the location of target features through target crop features and topology; S2-3, integrate the calculation results of multiple models, make comprehensive judgments, and calculate the maximum probability position of the target feature; The target crop is a plant with stems and leaves, and the target feature is the top of the stems and leaves of the target crop. Step S2-2 specifically includes the following steps: S2-2-1, performing Gaussian filtering to reduce noise on the image, performing background threshold segmentation on the image, and extracting a relatively complete target crop image; S2-2-2, performing dimensionality reduction processing on the acquired color target crop image, converting the image into a grayscale image, retaining the shape distribution information of the trunk and stems and leaves of the target crop in the original image, performing Canny operator edge detection, and obtaining a binary image of the edge of the target crop and the edge of the trunk of the stems and leaves; S2-2-3, process the acquired binary image: perform straight line detection on the binary image, where the curved image is the leaf edge and the remaining image is the stem and leaf trunk, extract the straight line image as C1, and the remaining image as C2, and the pixel values ​​in images C1 and C2 are the edge information of the binary image; S2-2-4, let C1(x,y) and C2(x,y) be the pixel points at the coordinate positions in the straight line image C1 and the residual image C2, judge C1(x,y) and C2(x,y), if C1(x,y) = C2(x,y), it means (x,y) is the leaf tip coordinate, if C1(x,y)≠ C2(x,y), it means (x,y) is not the leaf tip coordinate, save all leaf tip coordinates {(x,y)}; S2-2-5. Convert the straight line image C1 into a vector, with the starting point being the leaf tip coordinate {(x,y)}, obtain the intersection coordinates {(x',y')} of all vectors, draw a circle with i pixels as the radius, and use each pixel in C1 as the center of the circle for judgment. If there are intersection coordinates (x',y') in the circle and there are more than n intersections, the intersections are considered to be clustered, and the average coordinates of all intersections in the circle are calculated (Σx / n, Σy / n). The coordinates are the center coordinates of the top of the stems and leaves of the target crop, that is, the position coordinates of the target feature; where i and n are both natural numbers greater than 0.

2. The precision agricultural robot operation method according to claim 1, characterized in that: In step S4, the agricultural robot comprehensively calculates the three-dimensional coordinates of the agricultural robot and its operating actuator by integrating the RTK real-time differential positioning of GPS, IMU information, and wheel speed meter information, and calculates its position change in the world coordinate system.

3. The precision agricultural robot operation method according to claim 1, characterized in that: In the steps S1-S6, the delay in the entire algorithm logic is eliminated through time synchronization technology.

Images