Agaricus bisporus picking sequence planning method based on YOLOv8-USD instance segmentation and 3D point cloud

Abstract

The invention provides an agaricus bisporus picking sequence planning method based on YOLOv8-USD instance segmentation and 3D point cloud, and the method comprises the steps: processing color image information through employing a YOLOv8n-USD instance segmentation algorithm, and generating a segmentation mask corresponding to label information; performing registration on the generated segmentation mask and the corresponding depth image to generate a 3D point cloud; morphological processing is conducted on the color images, the circle center and the radius of each mushroom are calculated, whether picking is needed or not is judged according to the radius, and picking and sorting of the agaricus bisporus are achieved; mapping the circle center and radius information of the mushrooms into the 3D point cloud through the depth image, performing rapid neighbor search by using a KD tree, and segmenting the mushrooms in each mushroom cluster; a global picking sequence planning strategy and a local picking sequence planning strategy are provided, the global picking sequence strategy is used for carrying out classification and priority ranking on different types of agaricus bisporus clusters, and the local picking sequence strategy is used for planning the picking sequence of each mushroom in the mushroom clusters. The picking sequence can be adjusted according to the overlapping degree of agaricus bisporus.

Description

Technical Field

[0001] The present invention relates to the technical field of picking sequence planning, and in particular to a method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud. Background Art

[0002] There are four growth cycles of Agaricus bisporus. The first-wave mushrooms grow very densely, with obvious height differences. The second-wave mushrooms are grown after the mature Agaricus bisporus in the first-wave mushrooms are picked. The density is smaller than that of the first-wave mushrooms, and the growth height is similar. The third-wave mushrooms are the mushrooms grown after the second-wave mushrooms are picked. The density is smaller than that of the second-wave mushrooms, and there are more flush mushrooms. The fourth-wave mushrooms are the remaining mushrooms after the third-wave mushrooms are picked, and there are more mushrooms growing alone. Domestic and foreign researchers are committed to the research of intelligent picking robots for Agaricus bisporus, focusing on the collaborative work of high-precision visual recognition and flexible end effectors to reduce the damage caused by the mechanical picking process. However, for densely growing and overlapping Agaricus bisporus groups, this method still has the risk of damaging surrounding mushrooms. The distribution of mushroom clusters at different growth stages and different densities of Agaricus bisporus varies greatly. Using only one picking sequence strategy cannot meet the entire picking process. Therefore, further research is needed on how to identify Agaricus bisporus of different densities in the picking area and formulate a reasonable picking sequence plan to achieve accurate picking of Agaricus bisporus and reduce damage to surrounding Agaricus bisporus.

[0003] The existing patent number CN108805875B discloses a method for identifying the position of Agaricus bisporus and selecting the direction of pushing down. For overlapping mushrooms, light reflection is used to distinguish overlapping mushrooms, and it is judged whether they can be pushed down, and then they are pushed down and picked after judgment. This method has a disadvantage: for dense and overlapping mushroom clusters, the higher ones will cover the Agaricus bisporus in the lower layer. If the upper mushrooms are pushed down, the lower mushrooms will be damaged, and if the lower mushrooms are pushed down, the upper mushrooms will be damaged.

[0004] Patent No. CN117426255B discloses an automatic mushroom picking system and method based on vision and force feedback. In the overlap ratio calculation process, if the calculation is inaccurate, two mushrooms that are actually not overlapping may be misjudged as overlapping and thus mistakenly merged into one mushroom cluster. When picking in the overlapping area, even if a merging strategy is adopted, other mushrooms may be damaged during the picking process. This is especially true for dense and uneven mushrooms.

[0005] There is also a patent number CN111986203B that discloses a depth image segmentation method and device. The document mainly focuses on the accurate segmentation of depth images, and does not provide a detailed arrangement of the picking sequence, especially when dealing with dense or overlapping Agaricus bisporus. Solve the current problem of inaccurate depth image segmentation

[0006] There is also a patent number CN111445482B that discloses a method for segmenting and identifying overlapping Agaricus bisporus. A method for segmenting and identifying overlapping Agaricus bisporus is provided. The method overcomes the influence of uneven illumination and different heights of mushroom culture soil on the contour extraction of Agaricus bisporus during image acquisition, ensures the accuracy of the contour extraction of the Agaricus bisporus region, and improves the recognition rate of Agaricus bisporus in overlapping situations. The specific picking sequence planning is not detailed. Summary of the invention

[0007] The purpose of the present invention is to provide a method for planning the picking order of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud, which can overcome the problems of damage to surrounding Agaricus bisporus in the prior art and inability to adjust the picking order according to the overlapping degree of Agaricus bisporus, and realize instance segmentation and picking order planning of Agaricus bisporus with different densities.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is: a method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud, comprising the following steps: Step 1: Use the YOLOv8n-USD instance segmentation algorithm to process the color image information and generate a segmentation mask corresponding to the label information; Step 2: Generate a 3D point cloud by registering the generated segmentation mask with its corresponding depth image; Step 3: Perform morphological processing on the color image to calculate the center and radius of each mushroom, and determine whether it needs to be picked according to the size of the radius, so as to realize the picking and sorting of Agaricus bisporus; Step 4: Map the center and radius information of the mushrooms to the 3D point cloud through the depth image, use the KD tree to perform fast neighbor search, and segment the mushrooms in each mushroom cluster; Step 5: Propose a global picking and local picking sequence planning strategy. The global picking sequence strategy is to classify and prioritize different types of Agaricus bisporus clusters, and the local picking sequence strategy is to plan the picking sequence for each mushroom in the mushroom cluster.

[0009] Preferably, the YOLOv8n-USD instance segmentation algorithm in step 1 replaces the backbone network of yolov8 with the backbone network and jump connection module of U-Netv2 on the basis of YOLOv8n, retains the SPPF module in the original YOLOv8n, replaces the Concat module with the SDI module in the U-Netv2 segmentation network at the neck of YOLOv8n, and replaces the C2f module with the C2f_DWRSeg module.

[0010] Preferably, the backbone network of the U-Netv2 includes an encoder, an attention mechanism, and a feature transformation layer. The encoder uses PVT_v2 as the backbone network. The attention mechanism includes channel attention and spatial attention modules. The size and number of channels of these enhanced feature maps are adjusted through the feature transformation layer to match the YOLOv8 neck.

[0011] Preferably, the jump connection module transfers the enhanced feature map to the neck part of YOLOv8 for multi-scale fusion, ensuring that the feature map retains more detail information during the transmission process; the DWRSeg module includes a DWR module of a high-level network and a SIR module of a single-path structure of a low-level network.

[0012] Preferably, the DWR module uses a two-step method to draw multi-scale context information and fuse feature maps with multi-scale receptive fields. The first step is to generate relevant residuals from the density features of Agaricus bisporus, which are achieved by convolution combined with BN layers and ReLU layers to generate concise feature maps of different sizes as materials for morphological filtering in the second step; the second step uses multi-rate extended depth-directional convolution to perform morphological filtering on the density feature map of Agaricus bisporus, and each convolution channel uses a different expansion rate to filter the regional feature map to capture multi-scale feature information; after drawing the multi-scale context information, multiple outputs are aggregated, and BN in the path map is performed, and the point-wise convolution method is used to merge features to form residuals, and the residuals are added to the input feature map to construct a stronger and more comprehensive feature representation; the SIR module is based on the DWR module, removes the multi-branch dilated convolution structure, and only retains the first branch to compress the receptive field; removes the deep convolution, and uses a single-step feature extraction to quickly capture effective low-level features.

[0013] Preferably, in step 2, the generated mask image and the aligned depth image are combined to generate 3D point cloud data through camera intrinsic parameters and depth scaling factors, the 3D coordinates of each point are calculated through camera intrinsic parameters and the depth value of the pixel, and the color information corresponding to each point and the label in the mask image are recorded to distinguish different types of mushroom clusters; In the step 3, the color image is processed, the circular object in the image is detected, and the OTSU algorithm is used to automatically determine the optimal threshold for binarization processing, so as to highlight the bright and dark areas in the image, and the morphological closing operation is applied to the binary image to fill the small holes and cracks in the object, so as to make the boundary of the object clearer and more complete; the image is distance transformed, and the distance from each non-zero pixel to the nearest pixel is calculated to highlight the relationship between the center and the edge; an elliptical template is created and distance transformed, and the circular area in the image is quickly identified in the template matching, and the peak detection is performed on the template matching result to find the area with a higher matching degree, and the contour is found in the peak image, and each contour is extracted and its center and radius are calculated.

[0014] Preferably, in step 3, the geometric center of the mushroom cluster is calculated using the np.mean function of NumPy; The centers of the mushrooms in the mushroom cluster are labeled. After morphological processing of the color image, the coordinates and radius of each center in the image are obtained and converted into a three-dimensional point cloud. A is_center_in_range function is set. This function receives the coordinates of the center point and the coordinates of the mask bounding box. By comparing whether the coordinates of the center point are within the range of the mask bounding box, it is determined whether the center points are within the specified bounding box range, and their data are merged into the three-dimensional point cloud data with label information. To segment each mushroom point cloud within a mushroom cluster point cloud, first construct a KD tree and perform a fast nearest neighbor search, traversing each mushroom center and radius; for each mushroom center, use the KD tree to find the actual point cloud point closest to the center point, and use it as the seed point; set the search radius to be the actual radius multiplied by the distance threshold, use the KD tree to find points within the search radius, and then, through the region growing method, gradually expand to find qualified candidate points, and add these points to the point cloud set of the current mushroom, and finally segment each mushroom in this way.

[0015] Preferably, in the step 1, according to the growth characteristics of Agaricus bisporus and the gaps between different clusters, Agaricus bisporus is divided into three categories: dense Agaricus bisporus clusters, flush Agaricus bisporus clusters and single Agaricus bisporus; for single Agaricus bisporus, picking is carried out in the order of the closest distance to the origin; for flush Agaricus bisporus clusters, picking is carried out in a spiral order, according to the distance from the center of the Agaricus bisporus circle inside the mushroom cluster to the center of the mushroom cluster, and picking is carried out in descending order, and the outermost Agaricus bisporus is picked first; for dense Agaricus bisporus clusters, a genetic algorithm is used to plan the picking order.

[0016] Preferably, for a dense Agaricus bisporus cluster, a depth camera is used to measure the height of all mushrooms in the dense Agaricus bisporus cluster from the camera, and the maximum H and the minimum h are found, the height difference ΔH between the two is calculated, and a difference threshold ′ and a reference height H′ are set. The difference threshold refers to the height of a mature mushroom cap, and the reference height refers to the sum of the minimum value h of the mushroom from the camera and the difference threshold. If ΔH>ΔH′, it means that there are overlapping mushrooms with large height differences in the mushroom cluster, and mushrooms with heights lower than the reference height H′ are picked first. After picking the tall mushrooms, the remaining mushrooms are picked in order from the outer edge to the center, and the distance from each mushroom to the center of the minimum circumscribed circle to the center of the mushroom cluster is calculated, and the mushrooms are arranged in descending order, and mushrooms with larger distances are picked first; if ΔH<ΔH′, it indicates that the mushrooms in the mushroom cluster have the same growth height, and a spiral picking method is adopted to pick from the outer edge to the center.

[0017] Preferably, in the selection step, excellent individuals are selected for reproduction according to the fitness value, and individuals with high fitness are more likely to be selected; in the crossover step, new individuals are generated by exchanging part of the genes of the selected individuals, and the mutation step randomly changes part of the genes of the new individuals to increase the diversity of the population; the main loop optimizes the population by continuously iterating these steps until a satisfactory solution is found.

[0018] The beneficial effects of the present invention are:

[0019] This scheme can segment the Agaricus bisporus into different classes according to the growth state of the Agaricus bisporus and the growth gap between the Agaricus bisporus. The different classes of mushrooms are segmented by improving the YOLOv8 instance segmentation algorithm. This improvement enables the model to accurately detect and segment targets when dealing with complex scenes and densely arranged Agaricus bisporus, reducing missed detection and false detection. The improved YOLOv8 instance segmentation algorithm can more accurately segment the Agaricus bisporus and generate a better mask, which provides good support for the subsequent generation of point clouds. Through the KD tree nearest neighbor search and region growing method, this method can efficiently segment the mushroom point cloud in each mushroom cluster. Even when the Agaricus bisporus is densely arranged or partially overlapped, it can accurately distinguish and segment independent mushrooms, ensuring that the picking robot can operate in an orderly and accurate manner. Global and local picking order strategies. This method first performs global classification and priority sorting according to the type of Agaricus bisporus cluster, ensuring that the robot can first pick mushroom clusters that are easy to touch and have little impact, and then deal with more complex dense clusters. This global planning can give priority to picking those that are easy to pick in the picking area, while leaving more picking space in the picking area. In local picking, spiral picking is used for level mushrooms, and genetic algorithm picking is used for dense mushroom clusters, which can effectively reduce mushroom damage during picking. Especially when dealing with dense clusters, the picking sequence can be optimized to avoid collision or crushing of unpicked mushrooms by the robot. This not only improves the quality of picking, but also reduces the quality degradation of Agaricus bisporus caused by damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 The present invention provides a technical roadmap for the picking sequence planning method.

[0022] Figure 2 Schematic diagram of three different types of Agaricus bisporus images of the present invention.

[0023] Figure 3 This is the network structure diagram of YOLOv8n-USD of the present invention.

[0024] Figure 4 This is a network structure diagram of the SDI module of the present invention.

[0025] Figure 5 This is the C2f_DWRSeg network structure diagram of the present invention.

[0026] Figure 6 Schematic diagram of generating point cloud of the present invention.

[0027] Figure 7 This is the result diagram after the image morphological transformation of the present invention.

[0028] Figure 8 Schematic diagram of the segmentation of mushroom point cloud of the present invention.

[0029] Fig. 9 This is a flow chart of the picking sequence strategy of the present invention.

[0030] Fig.10 Schematic diagram of the dense Agaricus bisporus cluster picking strategy of the present invention.

[0031] Fig.11 This is a diagram of the picking order of all mushrooms of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0033] The present invention discloses a method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud. The images are taken using an Obbec Gemini2 depth camera, which can support the alignment and synchronous output of D2C depth images and color images. D2C refers to mapping each pixel point on the depth map to the corresponding position of the color image according to the internal and external parameters of the depth camera and the color camera, thereby obtaining an RGBD image. The camera is 350mm away from the mushroom bed of Agaricus bisporus, the size of the captured image is 1280×720 pixels, and a total of 875 images are captured.

[0034] In order to improve the robustness and generalization ability of the model, the dataset is enhanced. We use OpenCV and other library functions to enhance the dataset. (1) Change brightness: simulate images obtained under different lighting conditions to improve the robustness of the model to lighting changes. (2) Add Gaussian noise: simulate the noise of the image during transmission and acquisition, so that the neural network in the model shields high-frequency features. (3) Add motion blur: simulate the vibration of the robotic arm during rapid movement to enhance the neural network's feature extraction of the fruit boundary. (4) Rotate: rotate the original image. Through the above operations, the original image is expanded to 2750.

[0035] like Figure 1 As shown, the embodiment includes the following steps:

[0036] Step 1: Use the YOLOv8n-USD instance segmentation algorithm to process the color image information and generate a segmentation mask corresponding to the label information; Step 2: Generate a 3D point cloud by registering the generated segmentation mask with its corresponding depth image; Step 3: Perform morphological processing on the color image to calculate the center and radius of each mushroom, and determine whether it needs to be picked according to the size of the radius, so as to realize the picking and sorting of Agaricus bisporus; Step 4: Map the center and radius information of the mushrooms to the 3D point cloud through the depth image, use the KD tree to perform fast neighbor search, and segment the mushrooms in each mushroom cluster; Step 5: Propose a global picking and local picking sequence planning strategy. The global picking sequence strategy is to classify and prioritize different types of Agaricus bisporus clusters, and the local picking sequence strategy is to plan the picking sequence for each mushroom in the mushroom cluster.

[0037] Agaricus bisporus grows at different densities, and a single picking strategy will cause more damage to the mushrooms. Since the first two waves of Agaricus bisporus grow densely and overlap a lot, after two harvests, the height difference of the subsequent Agaricus bisporus growth is small, and there are more parallel and single ones. According to the growth characteristics of Agaricus bisporus and the gaps between different clusters, Agaricus bisporus is divided into three categories: dense Agaricus bisporus clusters, parallel Agaricus bisporus clusters and single Agaricus bisporus, which are named dense, level and one respectively. Figure 2 As shown in the figure, dense Agaricus bisporus cluster: Agaricus bisporus clusters grow densely, with obvious overlap or high density. Level Agaricus bisporus: Agaricus bisporus clusters have fewer number of 2-7, and the height difference of overlapping Agaricus bisporus is small. One Agaricus bisporus: There are no adjacent Agaricus bisporus around. Use Llabelme image annotation software to annotate the Agaricus bisporus images according to the gaps at the boundaries of different Agaricus bisporus clusters, such as Figure 2 shown.

[0038] The present invention aims at the instance segmentation of Agaricus bisporus clusters with different densities. On the basis of YOLOv8n, the backbone network of yolov8 is replaced with the backbone network of U-Net v2, while retaining the SPPF module in the original YOLOv8n. The purpose is to effectively extract multi-scale features and complex detail information, so that the network can better understand and process Agaricus bisporus with different densities. In order to improve the precision of feature fusion and better combine semantic information with detail information, the Concat module is replaced with the semantic and detail injection module SDI in the U-Net v2 segmentation network at the neck of YOLOv8n. The SDI module can better capture the feature information of small targets and improve the accuracy of detection and segmentation through meticulous feature fusion. The C2f module is replaced with the C2f_DWRSeg module. The low-level features can be extracted more efficiently through dilated convolution. The gradient flow and information integrity can be maintained through residual connection, thereby improving the perception and feature extraction of the Agaricus bisporus cluster boundary. The improved YOLOv8n network structure is as follows: Figure 3 shown.

[0039] Although the backbone network of yolov8 has the ability to extract image features, the backbone of UNet v2 has more advantages when dealing with complex and high-precision image segmentation tasks. In order to enable the yolov8 algorithm model to more comprehensively extract the features of the boundaries of Agaricus bisporus clusters with different densities, the backbone network of yolov8 is replaced with the backbone network and jump connection module of U-Net v2. The backbone network of U-Net v2 is mainly composed of encoders, attention mechanisms, feature transformation layers, etc. The encoder uses PVT_v2 as the backbone network. PVT_v2 is an efficient visual transformer that can better capture the details and semantic information in the image from the input image. In this way, the multi-level feature extraction capability can better capture the boundaries and internal structures of Agaricus bisporus clusters and improve segmentation accuracy. The attention mechanism includes channel attention and spatial attention modules, which can enhance important features and suppress irrelevant background information during feature extraction, and can better retain the detailed information of Agaricus bisporus clusters with different densities to enhance the feature map. The size and number of channels of these enhanced feature maps are adjusted through the feature transformation layer to match the YOLOv8 neck. The skip connection module transfers the enhanced feature map to the neck part of YOLOv8 for multi-scale fusion, which can ensure that the feature map retains more detail information during the transmission process.

[0040] In the YOLOv8 network structure, the neck part usually uses the path aggregation feature pyramid network PAN-FPN, in which Concat is responsible for splicing feature maps of different scales to achieve feature fusion, information transmission and feature enhancement. However, the Concat module has certain limitations in multi-scale information fusion. Its fusion effect is relatively rough, and it cannot fully capture the details of the target. It also loses some contextual information, which affects the detection performance in complex scenes. In order to better solve these problems, the SDI mechanism is introduced, and the Concat module in the YOLOv8 neck structure is replaced by the SDI module. The SDI module combines high-level semantic information and low-level detail information, and through spatial and channel attention mechanisms, it realizes a more detailed fusion of features, effectively improves the detection accuracy of small targets, better retains and utilizes contextual information, and improves the overall detection and segmentation accuracy. Its structure is as follows Figure 4 shown.

[0041] In the neck part of the yolov8 network, the C2f module can achieve multi-scale feature fusion, but in complex scenes and small targets, the feature representation ability is not strong enough, and it cannot fully utilize the feature information of different scales, resulting in insufficient robustness for targets of various sizes. In addition, the information transmission in the deep network may not be sufficient, and the gradient disappearance problem is prone to occur, which affects the model training effect. Therefore, a fused dilated residual DWRSeg module is introduced, and a new module C2f_DWRSeg module is designed. The C2f_DWRSeg structure is Figure 5 As shown in the figure. The structural feature of C2f_DWRSeg is that the DWRSeg module is added on the basis of the original C2f module. DWRSeg is mainly composed of the DWR module of the high-level network and the SIR module of the single-path structure of the low-level network. The DWR module can adapt to the features of different scales more flexibly. The use of two-step residualization is aimed at improving the efficiency of multi-scale information capture. The SIR module can efficiently extract low-level features, such as edge details, and can also reduce the computational complexity and the number of parameters.

[0042] The DWR module uses a two-step method to draw multi-scale context information and then fuses the feature maps with multi-scale receptive fields. Its structure is as follows: Figure 5As shown in the figure. The first step is to generate relevant residuals from the dense features of Agaricus bisporus, which is called regional residualization. This step is achieved by a 3x3 convolution combined with a BN layer and a ReLU layer to generate concise feature maps of different sizes as the material for the second step of morphological filtering. The second step is to use multi-rate extended depth-wise convolution to perform morphological filtering on the dense feature map of Agaricus bisporus, which is called semantic residualization. Each convolution channel uses a different expansion rate to filter the regional feature map to capture multi-scale feature information. Each channel feature only applies one desired receptive field to prevent redundancy. After drawing the multi-scale context information, multiple outputs are aggregated and the BN in the path map is performed. Then, the point-wise convolution method is used to merge the features to form a residual. Finally, the residual is added to the input feature map to build a stronger and more comprehensive feature representation.

[0043] The DWR module is very effective in processing high-level features, but for processing low-level features, the use of complex dilated convolution structures will lead to a waste of computing resources. The SIR module is based on the DWR module and reduces the computational complexity by simplifying the dilated product structure. Its structure is as follows: Figure 5 As shown in the figure. Two modifications were made based on the DWR module: (1) The multi-branch dilated convolution structure was removed, and only the first branch was retained to compress the receptive field. (2) The 3x3 depth convolution semantic residual was removed. Since the input feature map is large in size and has weak semantic information, a single-channel convolution can collect less information. The single-step feature extraction can quickly capture effective low-level features. The SIR module improves the performance and efficiency of low-level feature extraction while maintaining effective feature extraction by optimizing the structure, reducing computational complexity, and improving processing efficiency.

[0044] like Figure 6 As shown, the generated mask image and the aligned depth image are used to generate three-dimensional point cloud data through the camera intrinsic parameters and the depth scaling factor. The three-dimensional coordinates of each point are calculated through the camera intrinsic parameters and the depth value of the pixel, and the color information corresponding to each point and the label in the mask image are recorded to distinguish different types of mushroom clusters. The color image is processed to detect circular objects in the image. The OTSU algorithm is used to automatically determine the optimal threshold for binarization, which is convenient for highlighting the bright and dark areas in the image. The morphological closing operation is applied to the binary image to fill the small holes and cracks in the object to make the object boundary clearer and more complete. Then, the image is distance transformed to calculate the distance from each non-zero pixel to the nearest pixel to highlight the relationship between the center and the edge. An elliptical template is created and distance transformed, so that the circular area in the image can be quickly identified in the template matching. The peak detection is performed on the template matching results to find the area with a higher matching degree, and the contour is found in the peak image. Each contour is extracted and its center and radius are calculated. Finally, the detected center and radius are used to generate three-dimensional point cloud data through the depth image and mapped to the three-dimensional point cloud. As shown Figure 7 Shown are some of the results of the graphic transformation.

[0045] According to the above steps, the data and information of the point cloud have been obtained. The next step is to segment the point cloud. It is necessary to calculate the geometric center of each mushroom cluster, segment each mushroom in each mushroom cluster, and find the geometric center of each mushroom. These data provide data support for the subsequent picking order.

[0046] (1) Calculate the geometric center of the mushroom cluster. Use the np.mean function of NumPy. This function will calculate the average value along the specified axis. The present invention is set to calculate along the axis "0", that is, the average value of the "x", "y", and "z" coordinates of all points is calculated respectively. The result obtained is the geometric center of all points, such as Figure 8 Individual mushrooms and the center of a mushroom cluster are shown.

[0047] (2) Label the centers of the mushrooms in the mushroom cluster. After morphological processing of the color image, the coordinates and radius of each center in the image are obtained and converted into a three-dimensional point cloud. However, the coordinates of the center and the corresponding radius do not carry label category information. Set up an is_center_in_range function, which receives the coordinates of the center point and the coordinates of the mask bounding box. By comparing whether the coordinates of the center point are within the range of the mask bounding box, it is determined whether the center point is within the specified bounding box range and merged into the three-dimensional point cloud data with label information, such as Figure 8 The center of the circle of the mushroom is shown.

[0048] (3) Segment the mushroom point cloud within each mushroom cluster point cloud. First, construct a KD tree and perform a fast nearest neighbor search, traversing each mushroom center and radius. For each mushroom center, use the KD tree to find the actual point cloud point closest to the center point and use it as the seed point. Set the search radius to the actual radius multiplied by the distance threshold, and use the KD tree to find points within the search radius. Then, through the region growth method, gradually expand to find qualified candidate points, and add these points to the current mushroom point cloud set. Finally, segment each mushroom in this way. As follows Figure 8 Mushroom segmentation at different heights shown.

[0049] The block diagram of the entire picking sequence strategy is as follows: Fig. 9 As shown in the figure, this strategy of combining global and local picking planning can effectively reduce the damage to Agaricus bisporus during the picking process and improve the picking efficiency.

[0050] The present invention proposes a strategy that combines global and local picking planning. The global picking strategy is to classify and prioritize different types of Agaricus bisporus clusters, giving priority to picking one Agaricus bisporus, followed by level Agaricus bisporus clusters, and finally dense Agaricus bisporus clusters. For mushroom clusters with the same label, the distance from the center of each mushroom cluster to the origin of the camera's internal coordinates is calculated, and the clusters are sorted according to the size of the distance. This method determines the picking order between different mushroom clusters and mushroom clusters of the same type. In the local picking order planning, for single Agaricus bisporus, the mushrooms are picked in the order of the closest distance to the origin; for level Agaricus bisporus clusters, they are picked in a spiral order, based on the distance from the center of the Agaricus bisporus circle inside the mushroom cluster to the center of the mushroom cluster, and picked in descending order, with the outermost Agaricus bisporus being picked first; for dense Agaricus bisporus clusters, a genetic algorithm is used to plan the picking order. A population is initialized, and each individual represents a possible mushroom picking order. A fitness function is designed, which combines the height of the mushrooms and the rules of the picking order. Fig.10 The schematic diagram of the dense mushroom cluster picking strategy shown in the figure uses a depth camera to measure the height of all mushrooms in the dense mushroom cluster from the camera, and finds the maximum H and the minimum h, calculates the height difference ΔH between the two, sets a difference threshold ΔH′ and a reference height H′, the difference threshold refers to the height of a mature mushroom cap, and the reference height refers to the sum of the minimum value h of the mushroom from the camera and the difference threshold. If ΔH>ΔH′, it means that there are overlapping mushrooms with large height differences in the mushroom cluster, and mushrooms with heights lower than the reference height H′ are picked first. After picking the high-growing mushrooms, the remaining mushrooms are picked in order from the outer edge to the center, and the distance from each mushroom to the center of the minimum circumscribed circle to the center of the mushroom cluster is calculated, and they are arranged in descending order, and mushrooms with larger distances are picked first. If ΔH<ΔH′, it means that the mushrooms in the mushroom cluster have the same growth height, and a spiral picking method is used to pick from the outer edge to the center. In the selection step, excellent individuals are selected for reproduction according to the fitness value, and individuals with high fitness have a high probability of being selected. In the crossover step, new individuals are generated by exchanging some genes of the selected individuals, and the mutation step randomly changes some genes of the new individuals to increase the diversity of the population. The main loop optimizes the population by continuously iterating these steps until a satisfactory solution is found. Fig.11 Final order of presentation. Fig.11 Each mushroom has its own ID: Ⅰ: single mushroom, Ⅱ: hierarchical mushroom cluster, Ⅲ: dense mushroom cluster. The middle number indicates the global order, and the last number indicates the local order.

[0051] It should be noted that the parts not described in detail in the above embodiments are all prior art.

[0052] The above description is only a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for Agaricus bisporus picking sequence planning based on YOLOv8-USD instance segmentation and 3D point cloud, characterized in that: The following steps are involved: Step 1: Use the YOLOv8n-USD instance segmentation algorithm to process the color image information and generate a segmentation mask corresponding to the label information; Step 2: Generate a 3D point cloud by registering the generated segmentation mask with its corresponding depth image; Step 3: Perform morphological processing on the color image to calculate the center and radius of each mushroom, and determine whether it needs to be picked according to the size of the radius, so as to realize the picking and sorting of Agaricus bisporus; Step 4: Map the center and radius information of the mushrooms to the 3D point cloud through the depth image, use the KD tree to perform fast neighbor search, and segment the mushrooms in each mushroom cluster; Step 5: Propose a global picking and local picking sequence planning strategy. The global picking sequence strategy is to classify and prioritize different types of Agaricus bisporus clusters, and the local picking sequence strategy is to plan the picking sequence for each mushroom in the mushroom cluster.

2. According to claim 1, a method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud, characterized in that: In the step 1, the YOLOv8n-USD instance segmentation algorithm replaces the backbone network of yolov8 with the backbone network and jump connection module of U-Netv2 on the basis of YOLOv8n, retains the SPPF module in the original YOLOv8n, replaces the Concat module with the SDI module in the U-Netv2 segmentation network at the neck of YOLOv8n, and replaces the C2f module with the C2f_DWRSeg module.

3. The method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud according to claim 2, characterized in that: The backbone network of the U-Netv2 includes an encoder, an attention mechanism, and a feature transformation layer. The encoder uses PVT_v2 as the backbone network. The attention mechanism includes channel attention and spatial attention modules. The size and number of channels of these enhanced feature maps are adjusted through the feature transformation layer to match the YOLOv8 neck.

4. The method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud according to claim 2, characterized in that: The jump connection module transfers the enhanced feature map to the neck part of YOLOv8 for multi-scale fusion, ensuring that the feature map retains more detail information during the transmission process; the DWRSeg module includes a DWR module of a high-level network and a SIR module of a single-path structure of a low-level network.

5. The method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud according to claim 4, characterized in that: The DWR module uses a two-step method to draw multi-scale context information and fuse feature maps with multi-scale receptive fields. The first step is to generate relevant residuals from the density features of Agaricus bisporus, which are implemented by convolution combined with BN layer and ReLU layer to generate concise feature maps of different sizes as materials for morphological filtering in the second step; the second step uses multi-rate extended depth-direction convolution to perform morphological filtering on the density feature map of Agaricus bisporus, and each convolution channel uses different dilation rates to filter the regional feature map to capture multi-scale feature information; After drawing the multi-scale context information, multiple outputs are aggregated and the BN in the path graph is performed. The point-wise convolution method is used to merge the features to form a residual, which is added to the input feature map to build a stronger and more comprehensive feature representation. The SIR module is based on the DWR module, removing the multi-branch dilated convolution structure and only retaining the first branch to compress the receptive field. Remove the depth convolution and use single-step feature extraction to quickly capture effective low-level features.

6. The method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud according to claim 1, characterized in that: In step 2, the generated mask image and the aligned depth image are combined to generate 3D point cloud data through camera intrinsic parameters and depth scaling factors, the 3D coordinates of each point are calculated through camera intrinsic parameters and the depth value of the pixel, and the color information corresponding to each point and the label in the mask image are recorded to distinguish different types of mushroom clusters; In the step 3, the color image is processed to detect circular objects in the image, and the OTSU algorithm is used to automatically determine the optimal threshold for binarization processing, so as to highlight the bright and dark areas in the image, and the morphological closing operation is applied to the binary image to fill the small holes and cracks in the body, so as to make the boundary of the object clearer and more complete; Perform distance transformation on the image and calculate the distance from each non-zero pixel to the nearest pixel to highlight the relationship between the center and the edge. Create an elliptical template and perform distance transformation. Quickly identify circular areas in the image during template matching. Perform peak detection on the template matching results to find areas with higher matching degrees. Find contours in the peak image, extract each contour and calculate its center and radius.

7. The method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud according to claim 6, characterized in that: In step 3, the geometric center of the mushroom cluster is calculated using the np.mean function of NumPy; The centers of the mushrooms in the mushroom cluster are labeled. After morphological processing of the color image, the coordinates and radius of each center in the image are obtained and converted into a 3D point cloud. A is_center_in_range function is set. This function receives the coordinates of the center point and the coordinates of the mask bounding box. By comparing whether the coordinates of the center point are within the range of the mask bounding box, it is determined whether the center points are within the specified bounding box range and their data are merged into the 3D point cloud data with label information. To segment each mushroom point cloud within a mushroom cluster point cloud, first construct a KD tree and perform a fast nearest neighbor search, traversing each mushroom center and radius; for each mushroom center, use the KD tree to find the actual point cloud point closest to the center point, and use it as the seed point; set the search radius to be the actual radius multiplied by the distance threshold, use the KD tree to find points within the search radius, and then, through the region growing method, gradually expand to find qualified candidate points, and add these points to the point cloud set of the current mushroom, and finally segment each mushroom in this way.

8. The method for planning the picking sequence of Agaricus bisporus based on YOLOv8-USD instance segmentation and 3D point cloud according to claim 1, characterized in that: In the step 1, according to the growth characteristics of Agaricus bisporus and the gaps between different clusters, Agaricus bisporus is divided into three categories: dense Agaricus bisporus clusters, flush Agaricus bisporus clusters and single Agaricus bisporus; for single Agaricus bisporus, picking is carried out in the order of the closest distance to the origin; for flush Agaricus bisporus clusters, picking is carried out in a spiral order, according to the distance from the center of the Agaricus bisporus circle inside the mushroom cluster to the center of the mushroom cluster, and picking is carried out in descending order, and the outermost Agaricus bisporus is picked first; for dense Agaricus bisporus clusters, a genetic algorithm is used to plan the picking order.

9. The method for planning the picking sequence of Agaricus bisporus based on YOL0v8-USD instance segmentation and 3D point cloud according to claim 8, characterized in that: For dense Agaricus bisporus clusters, a depth camera is used to measure the height of all mushrooms in the dense Agaricus bisporus cluster from the camera, and the maximum H and minimum h are found, the height difference ΔH between the two is calculated, and a difference threshold ΔH′ and a reference height H′ are set. The difference threshold refers to the height of a mature mushroom cap, and the reference height refers to the sum of the minimum value h of the mushroom from the camera and the difference threshold. If ΔH>ΔH′, it means that there are overlapping mushrooms with large height differences in the mushroom cluster. Mushrooms with heights lower than the reference height H′ are picked first. After picking the tall mushrooms, the remaining mushrooms are picked in order from the outer edge to the center. The distance from each mushroom to the center of the smallest circumscribed circle to the center of the mushroom cluster is calculated, and they are arranged in descending order, and mushrooms with larger distances are picked first; if ΔH<ΔH′, it indicates that the mushrooms in the mushroom cluster have the same growth height, and a spiral picking method is used to pick from the outer edge to the center.

10. The method for planning the picking sequence of Agaricus bisporus based on YOL0v8-USD instance segmentation and 3D point cloud according to claim 9, characterized in that: In the selection step, excellent individuals are selected for reproduction according to the fitness value, and individuals with high fitness are more likely to be selected; in the crossover step, new individuals are generated by exchanging part of the genes of the selected individuals, and the mutation step randomly changes part of the genes of the new individuals to increase the diversity of the population; the main loop optimizes the population by continuously iterating these steps until a satisfactory solution is found.

Images