An intelligent identification and harvesting method suitable for broccoli

Abstract

The application discloses a kind of suitable for broccoli intelligent identification harvesting method, belong to agricultural production technical field.The method comprises the following steps: S1, the current harvesting area of broccoli field is collected RGB-D image, the image is input into target detection network, and the maturity type of broccoli ball is identified;S2, the actual spatial position of the mature broccoli ball is determined, and the harvesting mechanism is controlled to harvest the broccoli ball at the actual spatial position of positioning;S3, control the harvesting mechanism to move to the designated storage area, according to the size of broccoli ball grading unloading, judge whether there is no mature broccoli that has not been harvested in the current harvesting area, if so, return to step S2, otherwise execute step S4;S4, move the harvesting mechanism to the next harvesting area, repeat steps S1 to S3, realize intelligent harvesting operation.The application realizes the integration of detection, positioning, harvesting, grading, collection, navigation and other processes in selective harvesting operation of broccoli, improves the harvesting efficiency and intelligent degree.

Description

Technical Field

[0001] This invention belongs to the field of agricultural production technology and relates to an intelligent identification and harvesting method suitable for broccoli. Background Technology

[0002] Broccoli, also known as green cauliflower, belongs to the Brassicaceae family and the Brassica genus. Broccoli is mainly grown in open fields, resulting in generally poor uniformity in maturity. Selective harvesting by robots represents a future trend to replace manual labor.

[0003] CN 20231008787367.0 discloses a broccoli combine harvester and its control method, in which an ultrasonic ranging sensor controls the cutting height and a stepper motor drives a disc to rotate and cut the blades. However, this method cannot perform selective harvesting, and the harvesting method does not meet agronomic requirements. CN 202210160153.7 discloses a broccoli harvesting device, but it cannot perform intelligent identification of floret type or automatic control of the entire harvesting process, resulting in a low level of intelligence.

[0004] To address the problems existing in current mechanical control methods for broccoli harvesting, this paper proposes an intelligent, practical, and efficient intelligent identification and harvesting method suitable for broccoli. This method enables the detection, positioning, harvesting, grading, collection, and navigation processes during the operation, thereby freeing up labor and improving harvesting efficiency, which is of great significance. Summary of the Invention

[0005] The purpose of this invention is to overcome the problems existing in the prior art and provide an intelligent identification and harvesting method suitable for broccoli, realizing the integration of the entire process of detection, positioning, harvesting, grading, collection, and navigation in selective harvesting operations, thereby improving harvesting efficiency and intelligence.

[0006] The technical solution adopted by this invention to solve the technical problem is:

[0007] A smart identification and harvesting method for broccoli includes the following steps:

[0008] S1. Acquire RGB-D images of the current harvest area of ​​the broccoli field, input the RGB-D images into the target detection network, and identify the maturity type of the broccoli heads; the target detection network includes a target detection module, a maturity detection module, and a size classification module;

[0009] S2, determine the actual spatial location of the mature broccoli heads, and control the harvesting mechanism to harvest the broccoli heads at the located actual spatial location;

[0010] S3, control the harvesting mechanism to move to the designated collection area, grade and unload the broccoli heads according to their size, determine whether there are unharvested mature broccoli in the current harvesting area, if so, return to step S2, otherwise execute step S4.

[0011] S4, move the harvesting mechanism to the next harvesting area, and repeat steps S1 to S3 to achieve intelligent harvesting operation.

[0012] Furthermore, the RGB-D image of the current harvest area of ​​the broccoli field was taken vertically at a distance of 60-90cm directly above the crop, and the RGB-D image includes at least one row of broccoli.

[0013] Furthermore, the target detection module in the target detection network adopts an improved DeepLab-V3+ semantic segmentation network to identify broccoli balls and locate the vertex plane coordinates of the broccoli balls;

[0014] The encoder of the semantic segmentation network consists of a backbone feature extraction network, a pooling module, and an attention mechanism. The backbone feature extraction network adopts Mobile Net-V2, and the pooling module adopts a densely connected, hollow spatial pyramid pooling module. The output of the pooling module is supplemented with a spatial attention mechanism to adjust the weights of the feature channels. The decoder of the semantic segmentation network adopts the decoder structure in DeepLab-V3+.

[0015] Furthermore, the location coordinates of the broccoli bulb vertex plane include:

[0016] Based on the broccoli head region identified by the target detection module, mark the center point of the broccoli head in the RGB-D image;

[0017] By using a hand-eye coordinate transformation matrix and a camera parameter matrix for capturing RGB-D images, the center point of the broccoli ball in the RGB-D image is transformed into the spatial coordinates of the broccoli ball's vertices. The hand-eye coordinate transformation matrix can be obtained through hand-eye calibration methods, using the AX=XB model, based on a cost function, and optimized using a solver. This is common knowledge in the field and will not be elaborated further.

[0018] Furthermore, the maturity detection module and size classification module in the target detection network take the broccoli head region within the detection frame output by the target detection module as input. The maturity detection module is used to classify the maturity type of the pixels in the broccoli head region and take the maturity type with the most pixels as the maturity type of the broccoli head. The size classification module is used to estimate the diameter of the broccoli head based on the pixel area of ​​the broccoli head region and the shooting distance between the broccoli head and the RGB-D image, and classify the size based on the estimated value.

[0019] Furthermore, before the harvesting mechanism harvests the broccoli heads at their actual spatial locations, all the mature broccoli heads identified in the current harvesting area are sequentially designated as broccoli heads to be harvested, arranged from left to right and from top to bottom.

[0020] Furthermore, the controlled harvesting mechanism harvests the broccoli heads at their actual spatial locations, including:

[0021] The Fast Search Random Tree algorithm is used to plan an optimal path from the center position of the bottom plane of the harvesting mechanism to the vertex position of the broccoli head to be harvested. If there is a collision in the first planned path, the path is replanned.

[0022] Control the harvesting mechanism to move along the planned path to the target location;

[0023] After the harvesting mechanism reaches the target position, it is controlled to descend vertically by 15cm to clamp and cut the broccoli head, thus completing the harvest.

[0024] Furthermore, the control of the harvesting mechanism to move to the designated storage area includes:

[0025] Based on the current harvested broccoli head diameter grade, confirm the location coordinates of the corresponding designated storage area;

[0026] The fast search random tree algorithm is used to plan the optimal path from the center position of the bottom plane of the harvesting mechanism to a position 5-15cm directly above the position coordinates of the designated storage area.

[0027] Control the harvesting mechanism to move along the planned path to the target location;

[0028] After the harvesting unit reaches the target location, it unloads the broccoli heads.

[0029] Furthermore, the mobile harvesting mechanism's movement to the next harvesting area includes:

[0030] Take a forward view image of the harvesting mechanism at an angle of -30° to 0° from 40-60cm above the crop, and the forward view image shall contain at least one row of broccoli.

[0031] Based on the current location of the harvesting area, plan a navigation path whose forward distance does not exceed the camera's field of view for capturing RGB-D images;

[0032] The harvesting mechanism is controlled to move along the navigation path to the next harvesting area, realizing unmanned driving and segmented cyclical operation in the field.

[0033] Furthermore, the Lane Detection navigation line extraction algorithm and the Pure Pursuit tracking algorithm are adopted as navigation path planning algorithms.

[0034] Furthermore, the harvesting mechanism is mounted on a robotic arm, which drives the harvesting mechanism to move in space, adopting a camera-robotic arm configuration with the eye outside the hand; the field of view of the camera used to capture RGB-D images should be smaller than the harvesting workspace of the robotic arm.

[0035] The beneficial effects of this invention are as follows: It provides an intelligent, practical, and efficient method for identifying and harvesting broccoli. The target detection algorithm can quickly and accurately obtain the maturity category and head size grade of broccoli, harvest mature broccoli, and unload them into designated locations of the corresponding grade. The unloading and harvesting process avoids collisions and shortens the time through optimal path planning. After the current harvesting area is completed, it automatically navigates to the next harvesting area, realizing the integration of detection, positioning, harvesting, grading, collection, and navigation processes in the selective harvesting of broccoli, thereby improving harvesting efficiency and intelligence. Attached Figure Description

[0036] Figure 1 This is an overview diagram of an intelligent identification and harvesting method applicable to broccoli;

[0037] Figure 2 This is a flowchart illustrating a smart identification and harvesting method suitable for broccoli.

[0038] Figure 3 This is a schematic diagram of a target detection network for an intelligent identification and harvesting method suitable for broccoli. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and specific examples, but these are not intended to limit the invention:

[0040] like Figure 1 As shown, a smart identification and harvesting method for broccoli includes the following steps:

[0041] S1. Collect RGB-D images of the current harvest area of ​​the broccoli field, input the RGB-D images into the target detection network, and identify the maturity type of the broccoli heads;

[0042] S2, determine the actual spatial location of the mature broccoli heads, and control the harvesting mechanism to harvest the broccoli heads at the located actual spatial location;

[0043] S3, control the harvesting mechanism to move to the designated collection area, grade and unload the broccoli heads according to their size, determine whether there are unharvested mature broccoli in the current harvesting area, if so, return to step S2, otherwise execute step S4.

[0044] S4, move the harvesting mechanism to the next harvesting area, and repeat steps S1 to S3 to achieve intelligent harvesting operation.

[0045] like Figure 2 As shown, in step S1 of a smart identification and harvesting method for broccoli, the type of broccoli head is identified through a target detection network. An optional implementation process specifically includes the following steps:

[0046] S1.1, an RGB-D camera is used to take vertical pictures 80cm directly above the crop. The camera's field of view is equal to or smaller than the harvesting workspace of the harvesting mechanism. The area within the camera's field of view is regarded as the current harvesting area. The captured image is transmitted to the target detection network. The target detection network adopts the improved DeepLab-V3+, with broccoli heads as the detection target, and includes maturity determination and size grading functions.

[0047] S1.2, the target detection network outputs the recognition results, which include the maturity type of all broccoli in the image, the coordinates (X,Y) of the vertex plane position of the broccoli head, and the diameter D of the head.

[0048] like Figure 3 As shown, in step S1.2, the target detection network mainly includes an improved DeepLab-V3+ semantic segmentation network, a maturity determination module, and a size classification module. In some embodiments, to improve the accuracy and speed of identifying the maturity type of broccoli heads, the encoder of the semantic segmentation network consists of a backbone feature extraction network, a pooling module, and an attention mechanism. The backbone feature extraction network uses MobileNet-V2, and the pooling module is replaced with a densely connected, hollow spatial pyramid pooling module. The output with a lower void ratio is connected to a layer with a higher void ratio as input through dense connections, resulting in denser pixel sampling and enhancing the network's feature extraction capability. A spatial attention mechanism is added to the output of the pooling module to adjust the weight share of the feature channels, solving the problem of weak feature representation in broccoli images and improving the image segmentation accuracy of the model. The decoder of the semantic segmentation network adopts the decoder structure in DeepLab-V3+.

[0049] Based on the decoding results, maturity and size are determined. The maturity determination module determines the type of broccoli by comparing the number of pixel types, while the size grading module estimates the actual size by converting the area of ​​the flower head pixels and the distance between the flower head and the camera.

[0050] In this embodiment, four maturity types are set: immature, semi-mature, mature, and over-mature, corresponding to four pixel colors: green, blue, red, and yellow. In the maturity determination module, the maturity type of the pixels in the broccoli head area is classified. A broccoli head area will be marked with multiple colors, and the maturity type with the most pixels is taken as the maturity type of the broccoli head. Broccoli heads classified as mature or over-mature are the objects to be harvested.

[0051] The size grading module estimates the diameter D of the broccoli head based on the pixel area of ​​the broccoli head region and the shooting distance between the broccoli head and the RGB-D image. It then divides the broccoli head into grades based on the diameter D, using 15cm as the dividing line to distinguish mature broccoli heads into two product grades.

[0052] like Figure 2 As shown, in step S2 of a smart identification and harvesting method for broccoli, the harvesting mechanism is guided to move along a planned path to harvest the broccoli heads located in their actual spatial positions. An optional implementation process includes the following steps:

[0053] S2.1 Select a mature broccoli head that has been identified in the current harvesting area as the broccoli head to be harvested. In this embodiment, all the broccoli heads identified in the current harvesting area are selected as broccoli heads to be harvested in order from left to right and from top to bottom.

[0054] S2.2, obtain the spatial coordinates (X, Y, Z) of the apex of the broccoli head to be harvested using the hand-eye coordinate system transformation matrix and RGB-D camera parameters. The hand-eye coordinate system transformation matrix is ​​the transformation relationship between the planar pixel position in the RGB-D image and the actual spatial position in the harvesting mechanism coordinate system, which can be obtained through TASI or Zhang Zhengyou calibration method.

[0055] S2.3, based on the spatial coordinates (X,Y,Z) of the apex of the broccoli head to be harvested and the position of the harvesting mechanism, plan the path from the bottom of the harvesting mechanism to the point directly above the coordinates (X,Y,Z) to obtain a collision-free path with optimal time and optimal distance. In this embodiment, the harvesting mechanism is controlled by a robotic arm. By controlling the robotic arm's movements, the harvesting mechanism moves to the target harvesting position. The optimal path is obtained through the fast search random tree algorithm in the path planning library of the robotic arm operating system.

[0056] S2.4, control the harvesting mechanism to descend vertically by 15cm, which is the optimal cutting position for the broccoli, and clamp and cut the broccoli head.

[0057] like Figure 2As shown, in step S3, the harvesting mechanism is moved to the designated collection area, and the broccoli heads are graded and unloaded according to their size. An optional implementation process specifically includes the following steps:

[0058] S3.1, Based on the recognition results of the target detection network, the diameter D of the flower head is classified into grades. In this embodiment, 15cm is used as the dividing line. Those with a diameter greater than 15cm are mature Grade 1 broccoli, and those with a diameter less than 15cm are mature Grade 2 broccoli.

[0059] S3.2, Based on the spatial coordinates (X,Y,Z) of the apex of the clamped and cut broccoli head and the location of the corresponding level collection area, plan the path for the harvesting mechanism to reach 10cm above the corresponding level collection area; In this embodiment, the harvesting mechanism is controlled by a robotic arm to move. By controlling the robotic arm's movements, the harvesting mechanism moves to the location of the corresponding level collection area. The optimal path is obtained through the fast search random tree algorithm in the path planning library of the robotic arm operating system.

[0060] S3.3, the harvesting mechanism moves to 10cm above the corresponding level collection area according to the planned path to complete the discharge.

[0061] S3.4, repeat the above harvesting action until there are no more mature broccoli florets to be harvested in the camera's field of view.

[0062] like Figure 2 As shown, in step S4, moving the harvesting mechanism to the next harvesting area, an optional implementation process specifically includes the following steps:

[0063] S4.1, the navigation camera acquires an image taken at -15° from 50cm above the crop. The image field of view should include the top of the broccoli head in one row and two ridges to facilitate the extraction of the crop row navigation line.

[0064] S4.2, Based on the image captured in step S4.1, a field navigation algorithm is used to generate a navigation path for the next harvesting area. In this embodiment, the field navigation algorithm consists of a navigation line extraction algorithm of Lane Detection and a Pure Pursuit tracking algorithm.

[0065] S4.3, drive the harvesting mechanism to move along the navigation path generated in step S4.2 to the next harvesting area, with the forward distance being less than the width of the camera's field of view.

[0066] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A smart identification and harvesting method for broccoli, characterized in that, Includes the following steps: S1. Acquire RGB-D images of the current harvest area of ​​the broccoli field, input the images into a target detection network, and identify the maturity type of the broccoli heads. The target detection network includes a target detection module, a maturity detection module, and a size grading module. The RGB-D images of the current harvest area of ​​the broccoli field are obtained by vertically shooting from 60-90cm directly above the crops, and the RGB-D images include at least one row of multiple broccoli heads. The target detection module in the target detection network adopts an improved DeepLab-V3+ semantic segmentation network to identify broccoli heads and locate the vertex plane coordinates of the broccoli heads. The encoder of the semantic segmentation network consists of a backbone feature extraction network, a pooling module, and an attention mechanism. The backbone feature extraction network adopts MobileNet-V2, and the pooling module adopts a densely connected, hollow spatial pyramid pooling module. The output of the pooling module is supplemented with a spatial attention mechanism to adjust the weights of the feature channels. The decoder of the semantic segmentation network adopts the decoder structure in DeepLab-V3+. The maturity detection module and size classification module in the target detection network take the broccoli head region within the detection frame output by the target detection module as input. The maturity detection module is used to classify the maturity type of the pixels in the broccoli head region and take the maturity type with the most pixels as the maturity type of the broccoli head. The size classification module is used to estimate the diameter of the broccoli head based on the pixel area of ​​the broccoli head region and the shooting distance between the broccoli head and the RGB-D image, and classify the size based on the estimated value. The method for locating the planar position coordinates of the broccoli head vertex includes: marking the center point of the broccoli head in the RGB-D image based on the broccoli head region identified by the target detection module; and converting the center point of the broccoli head in the RGB-D image into the spatial position coordinates of the broccoli head vertex using a hand-eye coordinate system transformation matrix and a camera parameter matrix used to capture the RGB-D image. S2, determine the actual spatial location of mature broccoli heads, and control the harvesting mechanism to harvest the broccoli heads at the located spatial location, including: The fast search random tree algorithm is used to plan the optimal path from the center position of the bottom plane of the harvesting mechanism to the vertex position of the broccoli head to be harvested. Control the harvesting mechanism to move along the planned path to the target location; After the harvesting mechanism reaches the target position, it is controlled to descend vertically by 15cm to clamp and cut the broccoli head, thus completing the harvest. S3, control the harvesting mechanism to move to the designated storage area, and discharge broccoli heads according to their size, including: determining the location coordinates of the corresponding designated storage area based on the diameter of the currently harvested broccoli heads; using a fast search random tree algorithm to plan an optimal path from the center of the bottom plane of the harvesting mechanism to a position 5-15cm directly above the location coordinates of the designated storage area; controlling the harvesting mechanism to move along the planned path to the target location; and discharging the broccoli heads after the harvesting mechanism reaches the target location. Determine if there are any unharvested mature broccoli in the current harvesting area. If yes, return to step S2; otherwise, proceed to step S4. S4, moving the harvesting mechanism to the next harvesting area, including: taking a forward view image of the harvesting mechanism at an angle of -30° to 0° from 40-60cm above the crop, wherein the forward view image contains at least one row of broccoli; planning a navigation path based on the current harvesting area location, with a forward distance not exceeding the camera's field of view width used for capturing RGB-D images; and controlling the harvesting mechanism to move along the navigation path to the next harvesting area. Repeat steps S1 to S3 to achieve intelligent harvesting operations.

2. The intelligent identification and harvesting method for broccoli according to claim 1, characterized in that, Before the harvesting mechanism harvests the broccoli heads at their actual spatial locations, all mature broccoli heads identified in the current harvesting area are arranged in order from left to right and from top to bottom as broccoli heads to be harvested.

3. The intelligent identification and harvesting method for broccoli according to claim 1, characterized in that, The navigation line extraction algorithm based on Lane Detection and the Pure Pursuit tracking algorithm are used as the navigation path planning algorithms.

Images