A method, apparatus, equipment, and storage medium for chestnut identification and harvesting planning.

Abstract

This invention provides a chestnut identification and harvesting planning method, apparatus, device, and storage medium. The method includes acquiring a first chestnut identification image using a first camera; segmenting and masking the first chestnut identification image to address the small size and easily obscured growth characteristics of chestnuts; acquiring a second chestnut identification image using a second camera; extracting color components to determine maturity to address the difficulty in judging chestnut maturity due to their color blending with the environment; and using a multi-objective path planning algorithm to plan the harvesting path for a harvesting machine to address the high growth position and large quantity of chestnuts. After the harvesting machine approaches the chestnuts, the path planning algorithm further plans the harvesting path for a harvesting mechanism, which then approaches the chestnuts according to the planned path to complete the harvesting operation. This invention can automatically identify chestnuts that meet the requirements, saving time and labor compared to manual harvesting path planning, exhibiting strong robustness to complex environments, increasing harvesting speed, and reducing the likelihood of missed detections.

Description

Technical Field

[0001] This invention relates to the field of automatic chestnut identification and harvesting technology, specifically to a chestnut identification and harvesting planning method, apparatus, equipment, and storage medium. Background Technology

[0002] Traditional chestnut harvesting methods typically involve manual identification and picking. However, when chestnuts are not fully ripe, their spiky shells are usually a mix of green and yellow, similar to the color of tree leaves, making them difficult to distinguish manually. Furthermore, ordinary people often struggle to differentiate chestnuts at different stages, leading to frequent omissions during harvesting. While manual harvesting is acceptable for personal consumption, it is inefficient for commercial sales. Additionally, the height and spiky shells of chestnuts frequently cause injuries to harvesters, posing a high risk. To address the challenges of manual harvesting, harvesting machines have emerged. However, most of these machines rely on manual control of the harvesting path, which is time-consuming, labor-intensive, and unsuitable for complex environments. A satisfactory solution has yet to be found for the small size, easy obstruction, overlapping, and color-difficulty of chestnuts. Therefore, a safe, efficient, and intelligent method for chestnut identification and harvesting planning with low omission rates is crucial. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the present invention aims to provide a chestnut identification and harvesting planning method, apparatus, device, and storage medium. This method can completely encompass small-sized chestnuts in a chestnut image, overcome the problems of chestnuts being easily obscured and overlapping, acquire all target chestnuts in the image, and sequentially reach the positions of all target chestnuts using a multi-objective planning algorithm. The maturity of each target chestnut is then determined, and chestnuts that meet the maturity requirements are harvested. Compared to relying on human eyes to find chestnut targets, this method is more accurate and precise. Compared to manually planning harvesting paths, it saves time and effort, is robust to complex environments, increases harvesting speed, and reduces the likelihood of missed detections.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A method for chestnut identification and harvesting planning includes the following steps:

[0006] The first chestnut recognition image is acquired using the first camera. The first chestnut recognition image is then cut into several image blocks and enlarged by a certain scale so that all target chestnuts in the first chestnut recognition image are fully contained. Then, a mask reconstruction process is performed to identify all target chestnuts that are occluded or not.

[0007] The machine plans the recognition path according to the multi-objective programming algorithm, and then goes to the front of all target chestnut positions in the first chestnut recognition image in sequence according to the recognition path. If there is no target chestnut, it continues to take a new first chestnut recognition image.

[0008] Upon reaching the location of each target chestnut, a second camera is used to acquire a second chestnut recognition image. The maturity of the target chestnut is then determined. Colors of the target chestnut are extracted using RGB and HSV color spaces respectively. A hyperplane space is established based on the SVM model, and the maturity of the target chestnut in the hyperplane space is then determined.

[0009] If the target chestnuts reach the required maturity, a path planning algorithm is used to plan the harvesting path for the harvesting organization. The harvesting organization harvests the target chestnuts according to the harvesting path. If the target chestnuts do not reach the required maturity, the organization moves to the next target chestnut location until all target chestnuts that meet the requirements in the first chestnut identification image have been harvested. Then, a new first chestnut identification image is created, and the process continues until all target chestnuts that meet the requirements have been harvested.

[0010] Furthermore, before recognizing the first chestnut image, the first chestnut image or the first chestnut image is cut into several image blocks, and the horizontal and vertical pixel counts of each image block are expanded by a certain scale so that the image block containing the target chestnut can completely contain the target chestnut.

[0011] Taking the top left corner of the first chestnut recognition image as the origin, and the width W and height H of the image as the x-axis and y-axis respectively; the position of each image block in the first image is:

[0012]

[0013]

[0014] x2=x1+S p +ε

[0015] y2=y1+S p +ε

[0016] In the formula, x1 and y1 are the coordinates of the top-left vertex of the image block, x2 and y2 are the coordinates of the bottom-right vertex of the image block, and S P Let ε be the size of the image patch, and ε be the size of the overlapping pixel region between image patches. Based on the vertex coordinates of the image patches, the segmentation process of image patches in the first chestnut identification image can be represented as:

[0017] Image s =Image[x1:x2,y1:y2]

[0018] In the formula, Image and Images represent the first chestnut recognition image and the image blocks obtained from segmentation.

[0019] Furthermore, the first chestnut recognition image is first divided into multiple regular non-overlapping image block regions, and some image block regions are randomly sampled and selected, while the unselected image block regions are masked.

[0020] Then, feature encoding is performed on the masked first chestnut recognition image, and the next step is to reconstruct the encoded features to output the complete first chestnut recognition image;

[0021] Finally, the similarity between the reconstructed image and the original first chestnut recognition image is calculated using the mean squared error loss function, and the parameters of the image reconstruction are optimized through gradient backpropagation.

[0022] Furthermore, a chestnut maturity discrimination model was established using an SVM model. For the target chestnut, the R, G, and B components were used to extract the mature area of ​​the chestnut fruit, and the S component was used to extract the area of ​​the semi-ripe and unripe fruit. The second chestnut recognition image dataset was input into the SVM model for training. The R, G, B, and S components of each target chestnut were data points in the data space, representing the maturity coordinates of each target chestnut. The minimum distance of each coordinate relative to the separating hyperplane was calculated, and the slope and intercept of the separating hyperplane were solved to establish a separating hyperplane for classifying the data points. Multiple SVM models were used to complete the classification of the target chestnuts.

[0023] Furthermore, a first chestnut identification image is obtained by a first camera mounted on a harvesting machine, and a harvesting mechanism is mounted on the harvesting machine. A second chestnut identification image is obtained by a second camera mounted on the harvesting mechanism. The first camera is a regular camera, and the second camera is a macro camera. Multiple macro cameras are arranged circumferentially relative to the harvesting mechanism to capture chestnut images from various angles.

[0024] Furthermore, after identifying all target chestnuts in the first chestnut recognition image, a multi-objective planning algorithm based on ant colony algorithm is used to plan the harvesting machine to reach the front of all target chestnut positions in sequence;

[0025] In the ant colony algorithm process, the initial setting process for pheromone concentration is as follows:

[0026]

[0027] C = m / C nn

[0028] in, The pheromone intensity ranges from i1 to j1; C represents the pheromone; m is the number of ants; C nn This represents the length of the shortest path from the starting point to the ending point.

[0029] The pheromone update is shown below:

[0030]

[0031] in, Let ρ be the pheromone intensity from i1 to j1 at time t; m be the number of ants; 0 < ρ ≤ 1 be the pheromone evaporation rate. The pheromone left by the k1th ant on the path from i1 to j1.

[0032] A chestnut identification and harvesting planning device includes:

[0033] The target chestnut recognition module is used to acquire a first chestnut recognition image using a first camera, cut the first chestnut recognition image into several image blocks and enlarge them by a certain scale so that all target chestnuts in the first chestnut recognition image are fully contained, and then perform mask reconstruction processing to identify all target chestnuts that are occluded or not occluded.

[0034] The multi-objective planning algorithm planning and recognition module is used to plan the recognition path according to the multi-objective planning algorithm. The harvesting machine will sequentially reach the front of all target chestnut positions in the first chestnut recognition image according to the recognition path. If there is no target chestnut, it will continue to take a new first chestnut recognition image.

[0035] The target chestnut maturity discrimination module is used to acquire a second chestnut recognition image using a second camera after reaching the front of each target chestnut location, and to discriminate the maturity of the target chestnut. The module extracts the color of the target chestnut according to the RGB and HSV color spaces respectively, establishes a hyperplane space based on the SVM model, and discriminates the maturity of the target chestnut in the hyperplane space.

[0036] In the chestnut harvesting module, if the target chestnuts meet the maturity requirements, a path planning algorithm is used to plan the harvesting path for the harvesting organization. The harvesting organization harvests the target chestnuts according to the harvesting path. If the target chestnuts do not meet the maturity requirements, the organization moves forward to the next target chestnut location until all target chestnuts that meet the requirements in the first chestnut recognition image have been harvested. Then, a new first chestnut recognition image is created, and the process continues until all target chestnuts that meet the requirements have been harvested.

[0037] A chestnut identification and harvesting planning device includes a processor and a memory for storing a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the chestnut identification and harvesting planning method described above.

[0038] A storage medium storing a computer program, which, when executed by a processor, implements the steps of the chestnut identification and harvesting planning method described above.

[0039] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0040] (1) The present invention provides a chestnut identification and harvesting planning method, apparatus, equipment, and storage medium. It improves the identification method based on the specific growth characteristics of chestnuts. Given the small size of chestnuts, they are not fully displayed at the image edges. To identify all chestnuts, before identifying the first chestnut identification image, the first chestnut identification image is cut into several image blocks and enlarged by a certain scale, so that all target chestnuts in the first chestnut identification image are fully contained. This increases the speed of chestnut identification, taking into account the small-scale growth characteristics of chestnuts. Furthermore, mask reconstruction processing is performed to ensure that chestnuts in the first chestnut identification image are completely identified, addressing the easy occlusion and overlap of chestnuts. Furthermore, since chestnut shells are typically yellow-green with spiky textures, and the shells of nearly mature or already mature chestnuts are similar in color to chestnut leaves, they are difficult to identify. Therefore, a special identification method is required. This invention extracts the color of target chestnuts using RGB and HSV color spaces, establishes a hyperplane space based on an SVM model, and identifies target chestnuts in the hyperplane space to obtain their maturity values. Target chestnuts with maturity values ​​exceeding a set threshold are identified as having reached the required maturity level. This method replaces the traditional method of relying on human eyes to find chestnuts and determine their maturity. It can adapt to various complex environments, enhances the robustness of the system, and reduces the missed rate of chestnut harvesting.

[0041] (2) The present invention provides a chestnut identification and harvesting planning method, device, equipment and storage medium. It adopts a multi-objective planning algorithm to reach the front of all target chestnut positions in sequence, formulate the optimal path for identifying the maturity of all target chestnuts in the first chestnut identification image, and then harvest the target chestnuts that have reached the required maturity. This realizes autonomous harvesting, high-efficiency harvesting, saves time and effort, and reduces damage to chestnut trees caused by misharvesting. Attached Figure Description

[0042] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate specific embodiments of the invention and are intended to explain the invention, but do not constitute an undue limitation thereof. In the drawings:

[0043] Figure 1 This is a flowchart of a chestnut identification and harvesting planning method according to the present invention.

[0044] Figure 2 This is a schematic diagram of a chestnut identification and harvesting planning device according to the present invention.

[0045] Figure 3 This is a schematic diagram of a harvesting machine in a specific embodiment of the present invention.

[0046] Among them, 1. Industrial control computer; 2. Drone; 3. Robotic arm; 4. LiDAR; 5. Ordinary camera; 6. Macro camera; 7. Target chestnut. Detailed Implementation

[0047] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0048] To facilitate the description of specific implementation methods, this embodiment establishes corresponding chestnut detection models and chestnut maturity discrimination models to meet the requirements for chestnut detection and judgment.

[0049] The chestnut of this invention refers to the chestnut fruit with a spiky outer shell from a chestnut tree.

[0050] This invention provides a method for chestnut identification and harvesting planning, such as... Figure 1 and Figure 3 As shown, it includes the following steps:

[0051] Step S100: Use the first camera to acquire the first chestnut recognition image, cut the first chestnut recognition image into several image blocks and enlarge them by a certain scale so that all target chestnuts 7 in the first chestnut recognition image are fully contained, and then perform mask reconstruction processing to identify all target chestnuts 7 that are occluded or not occluded.

[0052] Step S200: Plan the recognition path according to the multi-objective programming algorithm. The harvesting machine will arrive in front of all target chestnuts 7 in the first chestnut recognition image in sequence according to the planned recognition path. If there is no target chestnut 7, continue to take a new first chestnut recognition image.

[0053] Step S300: After reaching the front of each target chestnut 7, use the second camera to acquire the second chestnut recognition image of the target chestnut 7, and judge the maturity of the target chestnut 7. Extract the color of the target chestnut 7 according to the RGB and HSV color spaces respectively, establish a hyperplane space based on the SVM model, and judge the maturity of the target chestnut 7 in the hyperplane space.

[0054] Step S400: If the maturity of the target chestnut 7 meets the requirements, the harvesting mechanism is planned using a path planning algorithm. The harvesting mechanism harvests the target chestnut 7 according to the harvesting path. If the maturity of the target chestnut 7 does not meet the requirements, the mechanism moves forward to the next target chestnut 7 position until all target chestnuts 7 that meet the requirements in the first chestnut recognition image have been harvested. Then, a new first chestnut recognition image is created until all target chestnuts 7 that meet the requirements have been harvested.

[0055] This invention provides a chestnut identification and harvesting planning method. It improves the identification method based on the specific growth characteristics of chestnuts. Given the small size of chestnuts, they are not fully displayed at the image edges. To identify all chestnuts, before identifying the first chestnut identification image, it is segmented into several image blocks and enlarged by a certain scale, ensuring that all target chestnuts 7 in the first chestnut identification image are fully contained. This increases the speed of chestnut identification, addressing the small-scale growth characteristics of chestnuts. Furthermore, mask reconstruction processing is performed to ensure complete identification of chestnuts in the first chestnut identification image, addressing the issues of easy occlusion and overlap. Additionally, due to the chestnut shell... Typically, chestnuts have a yellow-green, spiky shell structure. The shells of nearly mature or fully mature chestnuts are similar in color to chestnut leaves, making them difficult to identify. Therefore, special identification methods are needed. This invention extracts the color of the target chestnut 7 using both RGB and HSV color spaces, establishes a hyperplane space based on an SVM model, and identifies the target chestnut 7 within the hyperplane space to obtain its maturity value. Target chestnuts with maturity values ​​exceeding a set threshold are identified as having reached the required maturity level. This method replaces the traditional method of relying on human eyes to find chestnuts and determine their maturity. It can adapt to various complex environments, enhances the robustness of the system, and reduces the missed rate during chestnut harvesting.

[0056] The present invention provides a chestnut identification and harvesting planning method, which adopts a multi-objective planning algorithm to sequentially reach the positions of all target chestnuts 7, formulate the optimal path for identifying the maturity of all target chestnuts 7 in the first chestnut identification image, and then harvest the target chestnuts 7 that have reached the required maturity. This achieves autonomous and efficient harvesting, saves time and effort, and reduces damage to chestnut trees caused by misharvesting.

[0057] In step S100 of the present invention, during the process of identifying all occluded or unoccluded target chestnuts 7, a first set of chestnut image datasets is collected and applied to the identification of target chestnuts 7. In each first chestnut identification image, the background occupies a certain part, and the rest is the chestnut target. A second set of chestnut image datasets is applied to the determination of chestnut maturity. In each second chestnut identification image, chestnuts occupy the majority.

[0058] In step S300 of this invention, the first set of chestnut image datasets is processed to outline the corresponding chestnuts, and the second set of chestnut image datasets is processed to classify and label the chestnut targets as chestnut-immature, chestnut-semi-mature, chestnut-mature, and chestnut-bad fruit.

[0059] In this invention, in step S100, to address the difficulty of detecting small-scale chestnuts in the first chestnut recognition image, a block-based preprocessing method is employed. This method divides the image into multiple regular image blocks, effectively increasing the proportion of image pixels occupied by small-scale target chestnuts and reducing interference from complex background features. Furthermore, target chestnuts located at the boundaries of image blocks are cut off by adjacent image blocks, resulting in the loss of chestnut image features. This phenomenon can be avoided by expanding the image blocks outwards to accommodate the largest target chestnuts in the overlapping areas.

[0060] Specifically, taking the top left corner of the first chestnut recognition image as the origin, and the width W and height H of the image as the x-axis and y-axis, respectively, the position of each image block in the first image is as follows:

[0061]

[0062]

[0063] x2=x1+S p +ε

[0064] y2=y1+S p +ε

[0065] In the formula, x1 and y1 are the coordinates of the top-left vertex of the image block, x2 and y2 are the coordinates of the bottom-right vertex of the image block, and S P Let be the size of the image patch, and ε be the size of the overlapping pixel region between image patches. Based on the vertex coordinates of the image patches, the segmentation process of image patches in the first chestnut identification image can be represented as:

[0066] Images = Image[x1:x2,y1:y2]

[0067] In the formula, Image and Images represent the first chestnut recognition image and the image blocks obtained from segmentation.

[0068] In step S100, the first set of labeled and preprocessed chestnut image datasets is input into the chestnut detection model for training. The first set of chestnut image datasets is divided into a training set, a validation set, and a test set according to a certain ratio. The training set is used for training, the validation set is used to adjust appropriate parameters, and the test set is used to test the target chestnuts and the detection speed and accuracy of the chestnut detection model.

[0069] The chestnut detection model is based on a visual self-attention model (Vision Transformer, ViT), an image reconstruction module, and an attention decoding module. The chestnut feature learning process includes two stages: pre-training and fine-tuning training.

[0070] In the pre-training phase, the visual self-attention model and the image reconstruction module are trained, enabling the visual self-attention model to learn chestnut image features. Specifically, the first chestnut recognition image is first divided into multiple regular non-overlapping image patch regions. A portion of these regions are randomly sampled, while the unselected regions are masked. Then, the visual self-attention model encodes features from the masked first chestnut recognition image, and the image reconstruction module reconstructs the complete chestnut image from the encoded features. Finally, the mean squared error loss function is used to calculate the similarity between the reconstructed and original first chestnut recognition images, and the parameters of the visual self-attention model and the image reconstruction module are optimized through gradient backpropagation. Masking the chestnut image simulates the feature loss phenomenon caused by leaves obscuring the chestnut, thereby improving the model's detection performance for chestnuts obscured by leaves.

[0071] During the fine-tuning training phase, the pre-trained visual self-attention model and attention decoding module are trained. The first chestnut recognition image is input into the visual self-attention model for feature encoding. The attention decoding module decodes the features to obtain the category and location results of the first chestnut recognition image. The coordinate information of the category and location results are optimized using the cross-entropy loss function and the mean squared error loss function, respectively.

[0072] The inference phase follows the same workflow as the fine-tuning training phase, yielding the category and location results for each chestnut in the first chestnut recognition image. The category result represents the confidence level of the detection result as a chestnut; detection results with a confidence level greater than 0.5 are identified as target chestnut 7. The location result represents the position of target chestnut 7 in the first chestnut recognition image.

[0073] In step S300 of this invention, the target chestnut is color extracted according to the RGB and HSV color spaces respectively to obtain the R, G, B, and S values ​​of the target chestnut, where R, G, B, and S represent red, green, blue, and saturation respectively. A hyperplane space is established based on the SVM model, and the target chestnut in the hyperplane space is judged to obtain a quantitative value of the maturity of the target chestnut. Target chestnuts with maturity values ​​exceeding a set threshold are identified as target chestnuts that have reached the required maturity.

[0074] Specifically, a chestnut maturity discrimination model was established using an SVM model. For the chestnut target to be judged, the R, G, and B components are beneficial for extracting the mature area of ​​the chestnut fruit; the S component is beneficial for extracting the area of ​​semi-ripe and unripe fruits. Further image processing methods such as binarization and morphological processing were used to establish a chestnut maturity discrimination model based on the SVM model. The SVM model, as a chestnut maturity discrimination model, is a binary classification model whose purpose is to establish a hyperplane space to discriminate and classify chestnut maturity data in the space.

[0075] The SVM image processing process is as follows:

[0076] The second set of chestnut image datasets was input into the SVM model for training. The R, G, B, and S components of each target chestnut represent data points in the data space, indicating the maturity coordinates of each target chestnut. The minimum distance of each coordinate relative to the separating hyperplane was calculated, and the slope and intercept of the separating hyperplane were determined to establish a separating hyperplane for classifying the data points. Six SVM models were used to classify target chestnuts into immature, semi-mature, mature, and spoiled types, specifically immature-semi-mature, immature-mature, immature-spoiled, semi-mature-mature, semi-mature-spoiled, and mature-spoiled. For target chestnuts whose maturity needs to be determined, the extracted R, G, B, and S components can be input to determine the maturity level.

[0077] In addition, to increase the accuracy of detection, MATLAB can be used to extract nine components (R, G, B, H, S, V, L, a, b) from the second set of chestnut image datasets. By inputting the extracted nine components (R, G, B, H, S, V, L, a, b) into the SVM model, the maturity of the target chestnut can be further determined. Among them, R represents red, G represents green, and B represents blue. In the RGB color space, R, G, and B represent red, green, and blue, respectively. In the HSV color space, H, S, and V represent hue, saturation, and brightness, respectively. In the Lab color space, the L component represents the brightness of the pixel, with a value range of [0, 100], representing pure black to pure white. a represents the range from red to green, with a value range of [127, -128]. b represents the range from yellow to blue, with a value range of [127, -128].

[0078] In this invention, the first camera is a regular camera 5, the second camera is a macro camera 6, and multiple macro cameras 6 are arranged circumferentially relative to the harvesting mechanism to capture chestnut images from various angles.

[0079] The first camera is used to take pictures of chestnuts from a distance, mainly focusing on the chestnut's external features to determine its presence and approximate location on the chestnut tree. The second camera is used to take pictures of chestnuts at close range, mainly focusing on specific features such as the chestnut's color and the presence of cracks in its burr-like shell, in order to determine the chestnut's maturity.

[0080] Specifically, the set of macro cameras 6 consists of three macro cameras 6, which are set at the end of the chestnut harvesting robotic arm 3 and distributed circumferentially. The three macro cameras 6 are 120° apart to achieve 360° coverage of the chestnuts and to collect detailed features of the chestnuts. The collected images are input into the control computer for chestnut classification.

[0081] In step S500 of the present invention, after identifying all target chestnuts 7 in the first chestnut identification image through the chestnut detection model, the positions of all target chestnuts 7 are obtained through the lidar 4, and the order of harvesting all target chestnuts 7 is further planned according to the multi-target path planning algorithm.

[0082] Specifically, the lidar 4 scans the area detected as the target chestnut 7 and emits a laser beam. By receiving the reflected laser beam, it obtains the distance data of the target chestnut 7. Based on the distance data and the scanning angle data, it determines the chestnut coordinates. In this process, the lidar 4 and the ordinary camera 5 first need to be registered in coordinate systems. Then, the lidar 4 emits a laser beam at the target chestnut 7. Finally, the distance data is converted from the sensor coordinate system to the three-dimensional coordinate system established by the system. Based on the converted distance data, each measured point is represented as a three-dimensional coordinate (x0, y0, z0). The generated point cloud data is further filtered to remove the influence of noisy point clouds. Then, the obtained point cloud data is segmented to distinguish different targets. Finally, the processed chestnut target point cloud data is obtained.

[0083] Specifically, the ant colony algorithm is used in the multi-objective planning algorithm of UAV 2. The ant colony algorithm is an intelligent biomimetic algorithm. Its basic principle is that when ants are foraging, they release a pheromone along their path. The concentration of the pheromone is inversely proportional to the length of the path. The higher the pheromone concentration, the shorter the path. Usually, ants will choose the path with a higher pheromone concentration and leave a certain amount of pheromone behind. In this way, the shorter the path, the higher the pheromone concentration. Finally, the ants can find the shortest foraging path, which is the optimal solution to the problem.

[0084] The ants select the next node according to a random ratio, and the random ratio rules are as follows:

[0085]

[0086] Where i1 and j1 are the starting point and the ending point, respectively; k represents the kth node; Visibility is the reciprocal of the distance between two points i1 and j1; The pheromone intensity from i1 to j1 at time t; allowed k The set of nodes that have not yet been visited; others represent other methods; The pheromone intensity from i1 to node k at time t; Visibility is the reciprocal of the distance between two points i1 and k; α and β are two constants, which are the weighted values ​​of information content and visibility, respectively.

[0087] In the algorithm process, the initial setting of pheromone concentration is very important; too high or too low a value will hinder the algorithm's search.

[0088]

[0089] C = m / C nn

[0090] in, The pheromone intensity ranges from i1 to j1; C represents the pheromone; m is the number of ants; C nn This represents the length of the shortest path from the starting point to the ending point.

[0091] To address the issue of pheromones slowly evaporating and dissipating, and to ensure that the ant colony leaves behind more information along the shortest possible path, the pheromone update is as follows:

[0092]

[0093] in, Let ρ be the pheromone intensity from i1 to j1 at time t; m be the number of ants; 0 < ρ ≤ 1 be the pheromone evaporation rate. The pheromone left by the k1th ant on the path from i1 to j1.

[0094] The role of pheromone evaporation is to prevent the algorithm from concentrating too quickly in local optima, which helps to expand the search area. The role of pheromone enhancement is to enable concentrated actions that a single ant cannot achieve.

[0095] In one specific embodiment of the present invention, the RRT* algorithm is used to plan the picking path of the picking mechanism.

[0096] In one specific embodiment of the present invention, such as Figure 3 As shown, a harvesting machine harvests chestnuts. The harvesting machine includes an industrial control computer 1, a drone 2, a robotic arm 3, a lidar 4, a regular camera 5, a macro camera 6, and target chestnuts 7. The robotic arm 3 is mounted on the drone 2, and the lidar 4 and regular camera 5 are also mounted on the drone 2. Multiple macro cameras 6 are arranged circumferentially at the end of the robotic arm 3. The industrial control computer 1 controls the drone 2 and the robotic arm 3, and uses the lidar 4, regular camera 5, and macro camera 6 to capture a first chestnut identification image and a second chestnut identification image, respectively, and completes the chestnut positioning. The industrial control computer 1 processes the first and second chestnut identification images, and combines them with the point cloud data from the lidar 4 to perform a multi-objective planning algorithm and a path planning algorithm based on ant colony optimization, and harvests the target chestnuts that meet the maturity requirements.

[0097] Specifically, industrial control computer 1 controls drone 2 to enter the chestnut orchard for harvesting. Drone 2 approaches the chestnut trees from top to bottom. After reaching a certain distance, drone 2 remains stationary while ordinary camera 5 takes pictures. Industrial control computer 1 determines whether there is a chestnut target based on the pictures. If a target chestnut 7 is confirmed, the next step is for lidar 4 to emit a laser beam at the target chestnut 7 and determine the three-dimensional coordinates of chestnut 7 in the three-dimensional coordinate system established by the system based on the reflected beam. Next, industrial control computer 1 plans the drone 2's operating path based on the three-dimensional coordinates of the target chestnut 7 using a multi-objective planning algorithm based on ant colony optimization. Drone 2 then approaches the first target chestnut 7 according to the planned path. After approaching the target chestnut 7, the drone 2 remains stationary. The industrial control computer 1 plans the picking path of the robotic arm 3 according to the RRT* algorithm. Next, the robotic arm 3 approaches the target chestnut 7 according to the planned picking path. After the robotic arm 3 gets close to the target chestnut 7, the macro camera 6 is mounted on the robotic arm 3. Next, the macro camera takes a close-up image of the second chestnut for identification. The industrial control computer 1 judges the maturity of the chestnut based on the second chestnut identification image. If it is a semi-ripe chestnut or a ripe chestnut, the industrial control computer 1 controls the robotic arm 3 to pick the target chestnut 7. After picking, the drone 2 turns to the next target chestnut 7. If it is an unripe chestnut or a rotten chestnut, the drone 2 directly turns to the next target chestnut 7.

[0098] Specifically, for ease of understanding, in this embodiment, the drone 2 is used as the working mechanism and the robotic arm is used as the harvesting mechanism. All feasible machines are included in the selection range of the working mechanism of this invention, and all feasible mechanisms are included in the selection range of the harvesting mechanism of this invention.

[0099] The present invention also provides a chestnut identification and harvesting planning device, such as... Figure 2 As shown, it includes:

[0100] The target chestnut recognition module is used to acquire a first chestnut recognition image using a first camera, cut the first chestnut recognition image into several image blocks and enlarge them by a certain scale so that all target chestnuts in the first chestnut recognition image are fully contained, and then perform mask reconstruction processing to identify all target chestnuts that are occluded or not occluded.

[0101] The multi-objective planning algorithm planning and recognition module is used to plan the recognition path according to the multi-objective planning algorithm. The harvesting machine will sequentially reach the front of all target chestnut positions in the first chestnut recognition image according to the planned multi-objective harvesting path. If there is no target chestnut, it will continue to take a new first chestnut recognition image.

[0102] The target chestnut maturity discrimination module is used to acquire a second chestnut recognition image using a second camera after reaching the front of each target chestnut location, and to discriminate the maturity of the target chestnut. The module extracts the color of the target chestnut according to the RGB and HSV color spaces respectively, establishes a hyperplane space based on the SVM model, and discriminates the maturity of the target chestnut in the hyperplane space.

[0103] In the chestnut harvesting module, if the target chestnuts meet the maturity requirements, a path planning algorithm is used to plan the harvesting path for the harvesting organization. The harvesting organization harvests the target chestnuts according to the planned harvesting path. If the target chestnuts do not meet the maturity requirements, the organization moves forward to the next target chestnut location until all target chestnuts that meet the requirements in the first chestnut recognition image have been harvested. Then, a new first chestnut recognition image is created, and the process continues until all target chestnuts that meet the requirements have been harvested.

[0104] The present invention also provides a chestnut identification and harvesting planning device, including a processor and a memory for storing a computer program that can run on the processor. When the processor runs the computer program, it executes the steps of the chestnut identification and harvesting planning method described above.

[0105] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the chestnut identification and harvesting planning method described above.

[0106] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for chestnut identification and harvesting planning, characterized in that, Includes the following steps: The first chestnut recognition image is acquired using the first camera. The first chestnut recognition image is then cut into several image blocks and enlarged by a certain scale so that all target chestnuts in the first chestnut recognition image are fully contained. Then, a mask reconstruction process is performed to identify all target chestnuts that are occluded or not. The machine plans the recognition path according to the multi-objective programming algorithm, and then goes to the front of all target chestnut positions in the first chestnut recognition image in sequence according to the planned recognition path. If there is no target chestnut, it continues to take a new first chestnut recognition image. Upon reaching the location of each target chestnut, a second camera is used to acquire a second chestnut recognition image. The maturity of the target chestnut is then determined. Colors of the target chestnut are extracted using RGB and HSV color spaces respectively. A hyperplane space is established based on the SVM model, and the maturity of the target chestnut in the hyperplane space is then determined. If the target chestnuts reach the required maturity, a path planning algorithm is used to plan the harvesting path for the harvesting organization. The harvesting organization harvests the target chestnuts according to the harvesting path. If the target chestnuts do not reach the required maturity, the organization moves to the next target chestnut location until all target chestnuts that meet the requirements in the first chestnut identification image have been harvested. Then, a new first chestnut identification image is created, and the process continues until all target chestnuts that meet the requirements have been harvested.

2. The chestnut identification and harvesting planning method according to claim 1, characterized in that: Before recognizing the first chestnut image, the first chestnut image or the first chestnut image is cut into several image blocks. The horizontal and vertical pixel counts of each image block are expanded by a certain scale so that the image block containing the target chestnut can completely contain the target chestnut. Taking the top left corner of the first chestnut recognition image as the origin, and the width W and height H of the image as the x-axis and y-axis respectively; the position of each image block in the first image is: x2=x1+Sp+ε y2=y1+Sp+ε In the formula, x1 and y1 are the coordinates of the top-left vertex of the image block, x2 and y2 are the coordinates of the bottom-right vertex of the image block, and S P Let ε be the size of the image patch, and ε be the size of the overlapping pixel region between image patches. Based on the vertex coordinates of the image patches, the segmentation process of image patches in the first chestnut identification image can be represented as: Images = Image[x1:x2,y1:y2] In the formula, Image and Images represent the first chestnut recognition image and the segmented image blocks.

3. The chestnut identification and harvesting planning method according to claim 1, characterized in that: The first chestnut recognition image is first divided into multiple regular non-overlapping image block regions. Some image block regions are randomly sampled and selected, while the unselected image block regions are masked. Then, feature encoding is performed on the masked first chestnut recognition image, and the next step is to reconstruct the encoded features to output the complete first chestnut recognition image; Finally, the similarity between the reconstructed image and the original first chestnut recognition image is calculated using the mean squared error loss function, and the parameters of the image reconstruction are optimized through gradient backpropagation.

4. The chestnut identification and harvesting planning method according to claim 1, characterized in that: A chestnut maturity discrimination model was established using an SVM model. For the chestnut target to be judged, the R, G, and B components were used to extract the mature area of ​​the chestnut fruit. The S component is used to extract the area of ​​semi-ripe and unripe fruits. The second chestnut recognition image dataset is input into the SVM model for training. The R, G, B, and S of each target chestnut are data points in the data space, representing the maturity coordinates of each target chestnut. The minimum distance of each coordinate relative to the separating hyperplane is calculated, and the slope and intercept of the separating hyperplane are solved to establish the separating hyperplane for classifying the data points. Multiple SVM models are used to complete the classification of target chestnuts.

5. The chestnut identification and harvesting planning method according to claim 1, characterized in that: A first chestnut identification image is obtained by a first camera mounted on a harvesting machine. A harvesting mechanism is mounted on the harvesting machine, and a second chestnut identification image is obtained by a second camera mounted on the harvesting mechanism. The first camera is a regular camera, and the second camera is a macro camera. Multiple macro cameras are arranged circumferentially relative to the harvesting mechanism to capture chestnut images from various angles.

6. The chestnut identification and harvesting planning method according to claim 1, characterized in that: After identifying all target chestnuts in the first chestnut recognition image, a multi-objective planning algorithm based on ant colony algorithm is used to plan the harvesting machine to reach the front of all target chestnut positions in sequence; In the ant colony algorithm process, the initial setting process for pheromone concentration is as follows: C=m / C nn in, The pheromone intensity ranges from i1 to j1; C represents the pheromone; m is the number of ants; C nn This represents the length of the shortest path from the starting point to the ending point. The pheromone update is shown below: in, The pheromone intensity from i1 to j1 at time t; m is the number of ants; 0 < ρ ≤ 1 is the pheromone evaporation rate. The pheromone left by the k1th ant on the path from i1 to j1.

7. A chestnut identification and harvesting planning device, characterized in that, include: The target chestnut recognition module is used to acquire a first chestnut recognition image using a first camera, cut the first chestnut recognition image into several image blocks and enlarge them by a certain scale so that all target chestnuts in the first chestnut recognition image are fully contained, and then perform mask reconstruction processing to identify all target chestnuts that are occluded or not occluded. The multi-objective planning algorithm planning and recognition module is used to plan the recognition path according to the multi-objective planning algorithm. The harvesting machine will sequentially reach the front of all target chestnut positions in the first chestnut recognition image according to the recognition path. If there is no target chestnut, it will continue to take a new first chestnut recognition image. The target chestnut maturity discrimination module is used to acquire a second chestnut recognition image using a second camera after reaching the front of each target chestnut location, and to discriminate the maturity of the target chestnut. The module extracts the color of the target chestnut according to the RGB and HSV color spaces respectively, establishes a hyperplane space based on the SVM model, and discriminates the maturity of the target chestnut in the hyperplane space. In the chestnut harvesting module, if the target chestnuts meet the maturity requirements, a path planning algorithm is used to plan the harvesting path for the harvesting organization. The harvesting organization harvests the target chestnuts according to the harvesting path. If the target chestnuts do not meet the maturity requirements, the organization moves forward to the next target chestnut location until all target chestnuts that meet the requirements in the first chestnut recognition image have been harvested. Then, a new first chestnut recognition image is created, and the process continues until all target chestnuts that meet the requirements have been harvested.

8. A chestnut identification and harvesting planning device, characterized in that, It includes a processor and a memory for storing a computer program that can run on the processor. When the processor runs the computer program, it performs the steps of the chestnut identification and harvesting planning method according to any one of claims 1-6.

9. A storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the steps of the chestnut identification and harvesting planning method according to any one of claims 1-6.

Images