Mushroom bud detection method, system, medium and device based on image instance segmentation

Abstract

The invention provides a mushroom bud detection method, system, medium and device based on image instance segmentation, and the method, system, medium and device can achieve the accurate recognition and automatic collection of mushroom buds through an accurate detection technology and an efficient collection means in the production process of edible mushrooms. By introducing an advanced sensing technology, an image recognition algorithm and automatic control equipment, the system can monitor the growth state and orientation of mushroom buds in real time, accurately control the collection action, optimize the collection process, reduce damage to edible mushrooms and improve the collection efficiency and quality. Therefore, intelligent and standardized operation of mushroom bud detection and collection in edible mushroom production is realized, and powerful guarantee is provided for high-quality production of edible mushrooms based on image instance segmentation. By introducing an advanced sensing technology, an image recognition algorithm and automatic control equipment, the system can monitor the growth state and orientation of mushroom buds in real time, accurately control the collection action, optimize the collection process, reduce damage to edible mushrooms and improve the collection efficiency and quality. Therefore, intelligent and standardized operation of mushroom bud detection and collection in edible mushroom production is realized, and powerful guarantee is provided for high-quality production of edible mushrooms.

Description

Technical Field

[0001] The present application relates to the technical field of smart agriculture and intelligent detection and collection, and in particular to a mushroom bud detection method, system, medium and device based on image instance segmentation. Background Art

[0002] As the market demand for edible fungi continues to rise, it is urgent to improve its production efficiency and quality, and the accurate detection and efficient collection of mushroom buds is undoubtedly a key link in the entire production process. For a long time, the detection and collection of mushroom buds have relied on manual observation and manual operation. This traditional model has significant disadvantages: low efficiency, frequent misjudgments and missed collections, and due to the subjective awareness and experience of operators, it is difficult to ensure the accuracy and consistency of the detection and collection results, which greatly restricts the development of the edible fungi industry.

[0003] Faced with the growing market demand for edible fungi, it is an inevitable trend to promote the production of edible fungi towards automation and intelligence. Although the existing mushroom bud detection and collection equipment has made some breakthroughs in work efficiency, there are still many problems that need to be overcome: the detection accuracy cannot meet the needs of refined production, and it is difficult to accurately judge the growth status and maturity of mushroom buds; the collection methods are not sophisticated and flexible enough, which makes edible fungi susceptible to physical damage during the picking process; in addition, most existing systems have relatively single functions and cannot conduct all-round and comprehensive detection and collection of mushroom buds, making it difficult to maximize industrial benefits. Summary of the invention

[0004] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a mushroom bud detection method, system, medium and device based on image instance segmentation, so as to solve the technical problem that the prior art cannot accurately detect the growth status and production direction of mushroom buds.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a method for detecting edible mushroom buds based on image instance segmentation, comprising: collecting images of the growth environment of edible mushrooms through a pre-configured image acquisition device; performing image instance segmentation on the collected images of the growth environment of edible mushrooms based on an image instance segmentation algorithm to separate the mushroom bud area and the background area; intercepting the depth map according to the instance segmentation result to extract the depth information of the mushroom bud area; performing three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; generating a directed three-dimensional frame based on the reconstructed target point cloud to calculate the orientation and coordinates of the target frame, and generating 3D position coordinates and posture morphology information of the mushroom buds based on this; generating corresponding execution instructions based on the received 3D position coordinates and posture morphology information of the mushroom buds, and sending the execution instructions to the execution body that executes the mushroom bud collection task.

[0006] In some embodiments of the first aspect of the present application, the image instance segmentation algorithm is used to perform image instance segmentation on the collected images of the growth environment of edible fungi, including: annotating the collected mushroom bud images, the annotation content including the instance outline of each mushroom bud; using the Mask R-CNN model as the basic model architecture for instance segmentation, dividing the annotated mushroom bud image data set into a training set, a validation set and a test set, and using the training set to train the Mask R-CNN model; in each training iteration, performing online data enhancement operations on the training data; the data enhancement operations include: any one or more of rotation operations, scaling operations, flipping operations and adding operations; after each training cycle, using the validation set to evaluate the model, and calculating the evaluation index to measure the accuracy and completeness of the model's segmentation of mushroom buds; using the test set to perform a final test on the optimized model to evaluate the model's performance in mushroom bud segmentation in real scenarios.

[0007] In some embodiments of the first aspect of the present application, the image instance segmentation algorithm is based on which the collected images of the growth environment of edible fungi are segmented, including performing a background subtraction operation on the collected mushroom bud images, which includes: converting the collected RGB image containing mushroom buds into HSV or Lab color space; determining the threshold range in the new color space based on prior research on the color of mushroom buds and prior knowledge; selecting a deep learning model architecture suitable for image segmentation, and inputting the images after preliminary threshold segmentation and the corresponding annotation data as training data into the deep learning model for training; applying the trained model to a new mushroom bud image dataset, and evaluating and optimizing the output results of the model based on feedback from actual applications.

[0008] In some embodiments of the first aspect of the present application, the image instance segmentation algorithm is used to perform image instance segmentation on the collected images of the growth environment of edible fungi, including performing illumination normalization processing on the collected mushroom bud images, which includes: using histogram equalization to perform illumination normalization on the original image containing mushroom buds; embedding an illumination invariant feature extraction module in the deep learning model to automatically identify and adjust the feature differences of the image caused by illumination changes, and extract features that are not affected by illumination changes; and using the image data after histogram equalization processing as input and sending it to the deep learning model with the illumination invariant feature extraction module added for training.

[0009] In some embodiments of the first aspect of the present application, the image instance segmentation algorithm is used to perform image instance segmentation on the collected images of the growth environment of edible fungi, including using multi-view imaging technology to obtain images of mushroom buds from different angles; obtaining the complete morphological information of mushroom buds through a three-dimensional reconstruction method, and then performing segmentation.

[0010] In some embodiments of the first aspect of the present application, the generating of a directed three-dimensional frame based on the reconstructed target point cloud includes: using a point cloud segmentation algorithm to identify a point cloud region representing mushroom buds from the entire point cloud, and separating the identified mushroom bud point cloud from the background and other objects; fitting to generate a minimum directed three-dimensional frame that encloses the target point cloud, and determining the orientation and coordinates of the minimum directed three-dimensional frame; optimizing the initially fitted minimum directed three-dimensional frame to match the shape and position of the target point cloud, and detecting whether the minimum directed three-dimensional frame collides with other objects.

[0011] In some embodiments of the first aspect of the present application, after the execution instruction is sent to the execution body that executes the mushroom bud collection task, it is also executed: recording the collected original image data, the results of segmentation processing, and the three-dimensional reconstructed data, and storing them in a database or file system; and analyzing the recorded data to evaluate the effectiveness of the current mushroom bud detection and collection strategy.

[0012] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides an edible mushroom bud detection system based on image instance segmentation, including: an image acquisition module, which is used to acquire images of the growth environment of edible mushrooms through a pre-configured image acquisition device; an instance segmentation module, which is used to perform image instance segmentation on the acquired images of the growth environment of edible mushrooms based on an image instance segmentation algorithm, so as to segment the mushroom bud area and the background area; a depth information extraction module, which is used to intercept the depth map according to the instance segmentation result, so as to extract the depth information of the mushroom bud area; a three-dimensional reconstruction module, which is used to perform three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; a target frame generation module, which is used to generate a directed three-dimensional frame based on the reconstructed target point cloud, so as to calculate the orientation and coordinates of the target frame, and generate the 3D position coordinates and posture morphology information of the mushroom bud accordingly; a control module, which is used to generate corresponding execution instructions according to the received 3D position coordinates and posture morphology information of the mushroom bud, and send the execution instructions to the execution body that executes the mushroom bud acquisition task.

[0013] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the edible mushroom bud detection method based on image instance segmentation.

[0014] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code is run on a computer, the computer implements the edible mushroom bud detection method based on image instance segmentation.

[0015] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the edible mushroom bud detection method based on image instance segmentation.

[0016] As described above, the mushroom bud detection method, system, medium and device based on image instance segmentation of the present application have the following beneficial effects: the present application can realize accurate identification and automatic collection of mushroom buds in the process of edible fungi production through precise detection technology and efficient collection means. By introducing advanced sensing technology, image recognition algorithm and automatic control equipment, the system can monitor the growth state and orientation of mushroom buds in real time, accurately control the collection action, optimize the collection process, reduce the damage to edible fungi, and improve the collection efficiency and quality. Thereby realizing the intelligent and standardized operation of mushroom bud detection and collection in edible fungi production, based on image instance segmentation, providing strong guarantee for the high-quality production of edible fungi. By introducing advanced sensing technology, image recognition algorithm and automatic control equipment, the system can monitor the growth state and orientation of mushroom buds in real time, accurately control the collection action, optimize the collection process, reduce the damage to edible fungi, and improve the collection efficiency and quality. Thereby realizing the intelligent and standardized operation of mushroom bud detection and collection in edible fungi production, providing strong guarantee for the high-quality production of edible fungi. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a method for detecting edible mushroom buds based on image instance segmentation in one embodiment of the present application.

[0018] Figure 2 Shown is a schematic diagram of the process of using the Mask R-CNN model for training in one embodiment of the present application.

[0019] Figure 3 Shown is a schematic diagram of a process of performing background subtraction operation using color space conversion in one embodiment of the present application.

[0020] Figure 4 Shown is a schematic diagram of the process of performing illumination normalization processing when illumination conditions are unstable in one embodiment of the present application.

[0021] Figure 5 Shown is a schematic diagram of the process of performing multi-view imaging with mushroom bud occlusion in one embodiment of the present application.

[0022] Figure 6 Shown is a structural schematic diagram of an edible mushroom bud detection system based on image instance segmentation in one embodiment of the present application.

[0023] Figure 7Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] The technical solution of the embodiment of the present application aims to achieve accurate detection and efficient collection of edible mushroom buds, and the scope of application covers the cultivation industry of various edible mushrooms, such as common varieties such as shiitake mushrooms, oyster mushrooms, and enoki mushrooms, as well as the cultivation process of some rare edible mushrooms such as morels and truffles. In the detection link, the present application adopts a combination of high-resolution image recognition technology and sensors. By rationally arranging high-definition cameras in the mushroom house and using advanced image analysis algorithms, the 3D position information such as the germination position, size, and morphological characteristics of the mushroom buds can be accurately identified. At the same time, the sensor monitors the temperature, humidity, light and other environmental factors in the mushroom house in real time, providing accurate data support for the growth of the mushroom buds, and further optimizing the accuracy and reliability of the detection.

[0026] To facilitate understanding of the embodiments of the present application, first Figure 1 Detailed description. Figure 1 The flowchart of a method for detecting edible mushroom buds based on image instance segmentation in an embodiment of the present invention is shown. The method for detecting and collecting edible mushroom buds in this embodiment mainly includes the following steps.

[0027] Step S1: Capturing the growth environment image of edible fungi by using a pre-configured image acquisition device.

[0028] The main task of the image acquisition equipment is to capture images of the growth environment of edible fungi, including the effects of different temperatures, humidity, light and other conditions on the growth of edible fungi.

[0029] In some examples, high-quality RGB-D camera devices can be used to capture images of the growth environment of edible fungi. These devices have high resolution, more accurate depth measurement, and better environmental adaptability.

[0030] In some examples, consumer-grade RGB-D camera devices can also be used to collect images of the growth environment of edible fungi to achieve cost reduction. Considering that consumer-grade devices have shortcomings in resolution, accuracy, noise level and environmental adaptability, the data may not be accurate or reliable enough in certain application scenarios. Therefore, the embodiments of the present application overcome the problem of data source quality by using data fusion technology or unsupervised deep learning networks, which includes: aligning and fusing depth image data from multiple sensors or the same sensor at different time points through a data fusion algorithm to reduce noise and errors to improve the overall data quality and reliability. In this way, existing consumer-grade devices can be used to improve data quality through software optimization without replacing hardware.

[0031] Exemplarily, the present embodiment may use a variety of algorithms to align and fuse multiple depth images to reduce noise, errors and improve the accuracy of depth information. For example: (1) Feature-based algorithms, SIFT (Scale Invariant Feature Transform) detects feature points in an image by finding extreme points in different scale spaces, and then matches based on the descriptors of the feature points. For depth images, corresponding feature areas in different depth images can be found to assist in alignment, and then fusion processing is performed. SURF (Speeded Up Robust Features) improves the computing speed while maintaining good feature detection and matching effects. It can extract key features in depth images more quickly for alignment between different depth images, thereby realizing fusion operations and reducing the impact of errors caused by feature mismatching. (2) Iterative Closest Point Algorithm (ICP): It iteratively finds the closest point pair between two point clouds (corresponding to different depth images), and then calculates the transformation matrix (such as rotation and translation matrix) based on the correspondence between these point pairs by least squares method, etc., to transform the source point cloud (the point cloud corresponding to one depth image) to the coordinate system of the target point cloud (the point cloud corresponding to another depth image) to complete the alignment. After that, a suitable fusion strategy, such as weighted average, can be used for fusion to effectively improve the overall depth information accuracy. (3) Algorithms based on graph optimization, such as Bundle Adjustment, which constructs the registration and fusion problems of multiple depth images into a graph structure. The nodes represent different depth images or features in the image, and the edges represent the geometric constraints between them. By minimizing the overall error function, the pose and depth information of each node are optimized to achieve high-precision alignment and fusion of multiple depth images. (4) Voxel fusion algorithm, which is used to convert the depth image into a voxel model, and then fuse the voxels corresponding to multiple depth images in the voxel space.

[0032] Step S2: performing image instance segmentation on the collected image of the growth environment of edible fungi based on an image instance segmentation algorithm to segment the mushroom bud area and the background area.

[0033] In some examples, considering that mushroom buds have different shapes, with a wide range of sizes, shapes, colors, etc., different types of mushroom buds may have different growth forms, for example, some mushroom buds are spherical, while others may be hemispherical or irregular. Moreover, the shape of mushroom buds will also change at different stages of growth, which makes segmentation methods based on fixed shapes or templates difficult to apply.

[0034] In view of this, this application adopts an instance segmentation algorithm based on deep learning, such as Mask R-CNN, which is trained by a large number of annotated mushroom bud images to learn the various morphological features of mushroom buds. The specific steps are as follows: Figure 2 As shown, it includes steps S21a~S21e.

[0035] Step S21a: annotate the collected mushroom bud images, where the annotated content includes the instance outline of each mushroom bud.

[0036] Specifically, images of mushroom buds can be obtained from multiple sources such as mushroom planting bases, laboratory culture environments, or natural growth areas. Ensure that the images cover different growth stages, different varieties, different environmental backgrounds, and various possible morphological changes, providing a rich and comprehensive data foundation for subsequent model training. Subsequently, professional image annotation tools are used to accurately annotate the collected mushroom bud images. The annotation content includes the instance outline of each mushroom bud, that is, the boundary of the mushroom bud is outlined using polygons or other appropriate annotation methods, and the corresponding category information is annotated for each instance (if there are multiple mushroom bud categories). Ensuring the accuracy and consistency of the annotation is crucial for the model to learn the correct mushroom bud features.

[0037] Step S21b: Using the Mask R-CNN model as the basic model architecture for instance segmentation, dividing the labeled mushroom bud image dataset into a training set, a validation set, and a test set according to a certain ratio, and using the training set to train the Mask R-CNN model.

[0038] The Mask R-CNN model is a powerful model based on convolutional neural networks. It adds pixel-level segmentation capabilities to target instances on the basis of target detection. It can simultaneously output the target category, bounding box, and mask information. It is very suitable for objects with diverse forms and that require precise segmentation, such as mushroom buds. The labeled mushroom bud image dataset is divided into training set, validation set, and test set, usually in a certain ratio, such as 8:1:1. The Mask R-CNN model is trained using the training set. During the training process, the model calculates the loss function between the predicted result and the labeled true value through forward propagation, and then updates the model parameters through back propagation, gradually learning the feature representation and segmentation method of mushroom buds. During the training process, appropriate hyperparameters are set, such as learning rate, number of iterations, batch size, etc. The selection of these hyperparameters will affect the training effect and convergence speed of the model. The optimal values ​​can be determined through multiple trials and tuning.

[0039] Step S21c: In each training iteration, an online data enhancement operation is performed on the training data; the data enhancement operation includes: any one or more of a rotation operation, a scaling operation, a flipping operation and an adding operation.

[0040] Specifically, in each training iteration, online data augmentation operations are performed on the training data. The rotation operation is to randomly rotate the image by a certain angle (for example, between -30 degrees and 30 degrees) to simulate the situation of mushroom buds at different shooting angles, so that the model can learn the rotation invariance characteristics of mushroom buds. The scaling operation is to randomly scale the image, and the scaling ratio is within a certain range (such as 0.8~1.2 times), which helps the model adapt to mushroom bud images of different sizes and enhances its robustness to scale changes. The flipping operation is to flip the image horizontally or vertically to increase the diversity of the data, and at the same time allow the model to learn the symmetric characteristics of mushroom buds. Adding noise is to add random noise, such as Gaussian noise, to the image to simulate the noise interference in the actual shooting environment, so that the model can accurately segment mushroom buds even when facing noisy images.

[0041] Step S21d: After each training cycle, the model is evaluated using the validation set, and evaluation indicators are calculated to measure the accuracy and completeness of the model's segmentation of mushroom buds.

[0042] Calculate evaluation indicators such as average precision (AP), average recall (AR), mask mean intersection over union (mIoU), etc. Based on the evaluation results, if the model performance does not meet expectations, analyze the deficiencies of the model, such as overfitting or underfitting. Optimize the model by adjusting hyperparameters, increasing the amount of training data, improving the model structure, and then continue training until the model performance on the validation set reaches a satisfactory effect.

[0043] Step S21e: Use the test set to conduct a final test on the optimized model to evaluate the performance of the model in mushroom bud segmentation in real scenarios. If the test results meet the requirements of the actual application, the trained model can be deployed to the actual mushroom bud image segmentation task, and the newly collected mushroom bud images can be accurately segmented to provide technical support for applications such as mushroom growth monitoring and yield evaluation.

[0044] In some examples, considering the complex environment in which mushroom buds grow, the background may include various objects such as soil, fallen leaves, and weeds. The colors and textures of these background objects may be similar to those of mushroom buds, resulting in misjudgment during segmentation. For example, some brown fallen leaves may be difficult to distinguish from brown mushroom buds in color, thus affecting the accuracy of segmentation.

[0045] In view of this, the embodiment of the present application performs a background subtraction operation, utilizes color space conversion, such as converting an RGB image into an HSV or Lab color space, and preliminarily separates possible mushroom bud areas and background areas through threshold segmentation based on prior knowledge of mushroom bud color. Then, fine segmentation is performed in combination with a deep learning model. During the model training process, learning of background and mushroom bud boundary features is increased, for example, by adding a boundary loss term to the loss function, so that the model can more accurately identify the boundary between mushroom buds and background. The specific steps are as follows: Figure 3 As shown, steps S22a to S22c are included.

[0046] Step S22a: converting the collected RGB image containing mushroom buds into HSV or Lab color space; determining the threshold range in the new color space based on the previous research on the color of mushroom buds and prior knowledge.

[0047] It should be understood that the collected RGB images containing mushroom buds are converted into HSV or Lab color spaces. These color spaces can better separate color information than RGB. For example, in HSV space, hue (H), saturation (S) and brightness (V) represent different color attributes, which are more in line with human perception of color. According to the previous research and prior knowledge of mushroom bud color, the threshold range in the new color space is determined. For example, through the analysis of a large number of mushroom bud images, it is known that the hue of mushroom buds in HSV space may be concentrated in a certain interval, and the saturation and brightness also have corresponding characteristic ranges. Use these thresholds to perform image segmentation operations, preliminarily divide the image into possible mushroom bud areas and background areas, and obtain a rough segmentation mask.

[0048] Step S22b: Select a deep learning model architecture suitable for image segmentation, and input the image after preliminary threshold segmentation and the corresponding annotation data as training data into the deep learning model for training.

[0049] Optional deep learning model architectures such as the U-Net segmentation model have the ability to deeply extract and learn image features, including further analysis of the mushroom buds and background features in the initial segmented area. After selecting the model, initialize the model to determine the number of layers, the number of neurons in each layer, the activation function and other parameters of the model to prepare for subsequent training. Subsequently, the image after the initial threshold segmentation and the corresponding annotation data (accurate mushroom bud area and background area annotation) are input into the deep learning model as training data. During the training process, the model's ability to learn the boundary features of the mushroom buds and background is optimized. By adding a boundary loss term to the loss function, such as using a boundary loss function based on distance transformation, the model pays more attention to the classification accuracy of boundary pixels during training, so that the boundary between the mushroom buds and the background can be more accurately identified, and the initial segmentation results can be refined.

[0050] Step S22c: Apply the trained model to the new mushroom bud image dataset, and evaluate and optimize the output results of the model based on feedback from actual applications.

[0051] Apply the trained model to a new mushroom bud image dataset. The model first extracts and analyzes the input image features, and uses the mushroom bud and background features learned during training, including boundary features, to output the final accurate segmentation result, that is, the accurate mushroom bud region mask. Evaluate and optimize the output results of the model based on feedback from actual applications. For example, if it is found in actual applications that the model has errors in segmenting specific backgrounds or mushroom bud morphology in some cases, more such samples can be collected in a targeted manner, and the model can be fine-tuned to further improve the generalization ability and segmentation accuracy of the model to meet the accuracy requirements for mushroom bud image segmentation in actual production or research.

[0052] In some examples, considering the unstable lighting conditions in the natural environment, changes in light intensity and angle will cause changes in the color, brightness and contrast of the mushroom bud image. Under strong light, the mushroom bud may have highlights, while the color will become darker in the shadow part, which will affect the segmentation algorithm's extraction of mushroom bud features.

[0053] In view of this, the embodiment of the present application performs illumination normalization processing on the image. The histogram equalization technology can be used to adjust the brightness and contrast of the image to make the illumination distribution of the image more uniform. In addition, a feature extraction module with illumination invariance is added to the deep learning model, for example, an illumination normalization layer based on a convolutional neural network (CNN) is used. This layer can perform illumination adaptive processing on the input image at the front end of the network to reduce the impact of illumination changes on the segmentation results. The specific steps are as follows: Figure 4 As shown, steps S23a to S23c are included.

[0054] Step S23a: using histogram equalization to perform illumination normalization on the original image containing the mushroom buds.

[0055] First, the original images containing mushroom buds are obtained. Since these images may have uneven light intensity and distribution, histogram equalization technology is used to process them. Histogram equalization redistributes the pixel grayscale values ​​of the image and stretches the contrast of the image, making the details of the originally darker or brighter areas clearer and making the light distribution of the entire image more uniform. Specifically, it counts the frequency of each grayscale level in the image, and then maps the pixel value according to the cumulative distribution function, thereby changing the overall brightness and contrast performance of the image, providing a more ideal image basis for subsequent segmentation operations, and reducing the problem of increased segmentation difficulty caused by excessive light differences.

[0056] Step S23b: embedding an illumination invariance feature extraction module in the deep learning model to automatically identify and adjust the feature differences of the image caused by illumination changes, and extracting features that are not affected by illumination changes.

[0057] Select a suitable deep learning model architecture (such as a common convolutional neural network model suitable for image segmentation) for the mushroom bud image segmentation task, and add a light normalization layer based on a convolutional neural network (CNN) at the front end of the model. The design purpose of this module is to enable the model to have the ability to perform adaptive lighting processing on the input image. When the image data flows into the model, the light normalization layer will first process it, and based on its built-in convolution operation and other mechanisms, it will automatically identify and adjust the feature differences of the image caused by lighting changes, extract features that are not affected by lighting changes, and enable subsequent network layers to learn mushroom bud image segmentation based on relatively stable features, thereby enhancing the overall robustness of the model to lighting changes.

[0058] Step S23c: The image data after histogram equalization processing is used as input and sent to the deep learning model with an illumination invariance feature extraction module added for training.

[0059] During the training process, by setting a suitable loss function (such as a loss function that combines segmentation accuracy-related indicators) and adjusting hyperparameters such as the learning rate and number of iterations, the model is continuously taught how to accurately segment mushroom bud images based on the image features after illumination normalization. At the same time, the performance of the model is evaluated using validation sets, etc., and the segmentation effect of the model under image data simulated under different illumination conditions is observed. Based on the evaluation results, the model structure is further optimized or hyperparameters are adjusted to ensure that the model can effectively reduce the impact of illumination changes on the segmentation results. Finally, a model that can stably and accurately segment mushroom bud images under a variety of illumination scenarios is obtained, and applied to the actual mushroom bud image segmentation task.

[0060] In some examples, mushroom buds often grow densely and may be occluded from each other. When mushroom buds occlude each other, the shape and boundary information of some mushroom buds are incomplete, which makes it difficult to accurately segment each mushroom bud instance.

[0061] In view of this, the embodiment of the present application uses multi-view imaging technology to obtain images of mushroom buds from different angles, and then obtains the complete morphological information of the mushroom buds through a three-dimensional reconstruction method, and then performs segmentation. For two-dimensional image segmentation, learning about occlusion relationships can be added to the model. For example, by marking the occluded part of the mushroom bud information in the training data, the model learns how to deal with occlusion situations, and in the reasoning process, uses contextual information and prior knowledge to infer the shape and position of the occluded mushroom buds. The specific steps are as follows: Figure 5 As shown, it includes steps S24a~A24c.

[0062] Step S24a: Based on the multi-view imaging technology, imaging devices with multiple shooting angles and positions are arranged around the mushroom buds, and images of the mushroom buds are acquired synchronously or sequentially from these different angles.

[0063] Since mushroom buds often grow densely and block each other, a single-view image is difficult to present their complete shape. Multi-view imaging can capture the characteristic information of each surface of the mushroom bud. Then, a suitable 3D reconstruction method is used, such as a reconstruction algorithm based on the principles of structured light, stereo vision, or multi-view geometry, to process the collected multi-view images, integrate the image information from various angles, and construct a 3D model of the mushroom bud, thereby obtaining the complete morphological information of the mushroom bud, laying the foundation for subsequent accurate segmentation.

[0064] Step S24b: for the two-dimensional image segmentation task, prepare an image data set for training the model; input the image data set with occlusion annotations into the image segmentation model for training.

[0065] In this process, the occlusion of mushroom buds in the image is annotated, and the parts of the mushroom buds that are occluded and the corresponding occlusion relationships are clearly marked, forming training data with detailed occlusion annotations. Next, a suitable image segmentation model (such as Mask R-CNN, etc.) is selected, and the image data with occlusion annotations is input into the model for training. During the training process, by designing the loss function and adjusting various parameters of the model, the model is guided to learn how to identify occlusion relationships, so that the model can accurately distinguish the occluded parts of the mushroom buds according to various features in the image, and master the ability to handle occlusion situations.

[0066] Step S24c: Apply the trained image segmentation model to the new mushroom bud image for inference segmentation.

[0067] When the trained model is applied to a new mushroom bud image for inference and segmentation, the model will first extract various features in the image. When encountering complex situations where mushroom buds are occluded, in addition to making judgments based on the learned occlusion relationship features, it will also make full use of the contextual information in the image, such as the distribution of surrounding mushroom buds, the overall growth trend, etc., as well as the prior knowledge about mushroom bud morphology accumulated from previous training, to comprehensively infer the shape and position of the occluded mushroom buds, thereby outputting more accurate and complete mushroom bud image segmentation results, and improving the segmentation accuracy in scenes with dense mushroom buds and occlusion.

[0068] Step S3: intercepting the depth map according to the instance segmentation result to extract the depth information of the mushroom bud area.

[0069] Specifically, the instance segmentation result refers to the identification of the specific position, contour boundary and other information of each mushroom bud in the image through the instance segmentation algorithm, distinguishing each mushroom bud as an independent instance, and generating a corresponding segmentation mask. This mask accurately marks which pixels belong to a specific mushroom bud instance and which belong to the background in the form of a binary image or a marking matrix, providing a precise basis for defining the area where the mushroom buds are located for subsequent operations. At the same time, depth map data corresponding to the image is required. A depth map is a type of data that represents the distance information of each object in the scene from an imaging device such as a camera in the form of an image. Usually, the grayscale value or color value of each pixel represents the depth value of the corresponding position in the real scene. The depth map can be obtained by directly shooting with a depth camera, or by generating it based on multiple ordinary images using algorithms such as multi-view stereo matching.

[0070] After obtaining the instance segmentation results and the corresponding depth map, the mushroom bud area mask information obtained by the instance segmentation is used to intercept the depth map. Specifically, the segmentation mask is overlaid on the depth map, and the pixel position of the mushroom bud marked by the mask is used to accurately locate the pixel area belonging to the mushroom bud in the depth map, and then the depth map portion corresponding to these mushroom bud areas is extracted, and the remaining depth map content belonging to the background and other non-mushroom bud areas is discarded. In this way, the intercepted depth map portion only contains the depth information of the mushroom bud area, and further analysis can be carried out based on the extracted depth information, such as studying the growth height differences of mushroom buds, volume estimation, and other depth-related mushroom bud growth feature analysis work. Through such a technical solution, the depth information of the target mushroom bud can be effectively separated from the depth map containing complex scenes, providing a key data basis for a more targeted understanding of the characteristics of mushroom buds in three-dimensional space.

[0071] Step S4: Perform three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud.

[0072] Specifically, structured light, stereo disparity, multi-view stereo matching or deep learning algorithms can be used for 3D reconstruction, and the converted 3D coordinate points are set to form a target point cloud. Among them, the structured light method calculates the depth information of the object by projecting a specific light pattern and capturing its deformation. The stereo disparity method uses two or more cameras to take images from different angles and calculates the depth through disparity. The multi-view stereo matching method combines images from multiple perspectives to reconstruct the 3D structure through feature matching and disparity calculation. The deep learning algorithm predicts the 3D shape of the object directly from the image by building a deep learning model without complex geometric calculations. It should be understood that a point cloud is a data set composed of a set of points in a three-dimensional space, each point containing its coordinates in space, as well as information such as color and intensity. The characteristics of point clouds include disorder, near-dense and far-sparse, and unstructured. They are the basic data structures that describe the shape and position of objects in three-dimensional space.

[0073] Step S5: Generate a directed three-dimensional frame based on the reconstructed target point cloud to calculate the orientation and coordinates of the target frame, and generate the 3D position coordinates and posture morphology information of the mushroom bud accordingly.

[0074] Preferably, before using the target point cloud to generate a directed three-dimensional box, the target point cloud is preprocessed, and the preprocessing methods include but are not limited to denoising, filtering, downsampling, segmentation, etc., to improve the quality of the point cloud and reduce the complexity of subsequent processing. Denoising refers to removing noise values ​​in the point cloud data. Filtering refers to filtering the point cloud data to reduce the impact of noise and outliers. Downsampling can reduce the density of point cloud data to increase processing speed and reduce storage space. Segmentation is to divide the point cloud into different areas or objects for further analysis and processing.

[0075] In an embodiment of the present application, the process of generating a target frame includes: First, a point cloud segmentation algorithm is used to identify the point cloud area representing the mushroom buds from the entire point cloud, and the identified mushroom bud point cloud is separated from the background and other objects.

[0076] Point cloud segmentation algorithms can be, for example, threshold-based segmentation algorithms, region-based segmentation algorithms, or machine learning-based segmentation algorithms. Threshold-based segmentation algorithms distinguish different point cloud regions by setting one or more thresholds. First, select the point with the lowest height value as the near-ground point set, determine the segmentation threshold based on the orthogonal distance from the point to the ground straight line model, and classify the points whose orthogonal distance from the point to the ground straight line model is less than the threshold as ground points; then, according to the characteristics of the point cloud data, such as the amplitude of road surface fluctuation, dynamically adjust the segmentation threshold to adapt to different point cloud data characteristics. Region-based segmentation algorithms divide the points in the point cloud into different regions according to their local characteristics. This method takes into account the local geometric characteristics of the point cloud and can better handle complex scenes. Machine learning-based segmentation algorithms refer to the use of machine learning models (such as deep learning models) to segment point clouds. These methods can learn the complex features of point cloud data and are suitable for large-scale and complex point cloud data sets. For example, PointNet / PointNet++ can be used as a deep learning network that inputs 3D point clouds and outputs segmentation results. It can handle the disorder of point cloud data and has good generalization ability.

[0077] Secondly, a minimum directed three-dimensional box surrounding the target point cloud is generated by fitting, and the orientation and coordinates of the minimum directed three-dimensional box are determined.

[0078] The method of determining the orientation and coordinates of the minimum directed three-dimensional frame includes: determining the principal axis direction of the point cloud by principal component analysis (PCA) to estimate the orientation of the target frame; calculating the center point coordinates of the minimum directed three-dimensional frame to determine the size of the minimum directed three-dimensional frame.

[0079] It should be understood that a directed 3D box refers to a 3D box with a direction (Oriented Bounding Box, OBB), which is aligned with the axis and can tightly surround the target point cloud, can more accurately adapt to the shape of the point cloud, and reduce the surrounding space. The center point of the minimum directed 3D box is usually the center of mass or center of gravity of the point cloud. The size of the minimum directed 3D box depends on the distribution range of the point cloud along the main axis, thus obtaining the length, width and height of the box.

[0080] The principle of principal component analysis is to transform the original data into a new coordinate system through linear transformation, so that in this new coordinate system, the first largest variance of any projection of the data is on the first coordinate (i.e., the first principal component), the second largest variance is on the second coordinate (i.e., the second principal component), and so on. In this way, the three main directions are the three directions with the largest data variance, which reveal the most significant trend of change in the data. In summary, PCA determines the principal component directions of the data by finding the eigenvectors of the data covariance matrix, which can best represent the variability of the data.

[0081] Finally, the initially fitted minimum directed three-dimensional box is optimized to match the shape and position of the target point cloud, and it is detected whether the minimum directed three-dimensional box collides with other objects.

[0082] It is worth noting that the initial fit is usually determined based on the geometric characteristics of the point cloud data (such as the centroid, the direction of the principal component, and the distribution range). It can provide a rough three-dimensional box, but it may not accurately match the complex shape of the target point cloud. Therefore, the three-dimensional box needs to be further optimized, and the goal of the optimization is to reduce the error between the three-dimensional box and the target point cloud. Specifically, the initial fit can be improved using optimization algorithms such as gradient descent, genetic algorithms, and simulated annealing. These algorithms find the best fit by iteratively adjusting the parameters of the box (such as position, size, and rotation).

[0083] In the embodiments of the present application, after optimizing the initially fitted minimum directed three-dimensional box, it is necessary to further detect whether the minimum directed three-dimensional box collides with other objects. Collision detection can prevent the collection equipment (such as a picking manipulator or other automated tools) from physically colliding with mushroom buds or other objects when approaching or picking mushroom buds, thereby avoiding damage to the equipment or damage to the mushroom buds. Specifically, geometric algorithms can be used for collision detection, such as the separating axis theorem (SAT) or the minimum distance test, to determine whether two or more objects intersect.

[0084] Step S6: Generate corresponding execution instructions according to the received 3D position coordinates and posture information of the mushroom buds, and send the execution instructions to the execution body that executes the mushroom bud collection task.

[0085] It is worth mentioning that issuing execution instructions is a key part of the mushroom bud detection system, which is responsible for converting the 3D position coordinates and morphological information provided by the target frame generation module into the actual acquisition action of the robotic arm.

[0086] In some examples, motion planning can be performed based on the 3D position and posture of the mushroom bud target to generate a motion path from the current position to the target position; the planned motion path is converted into specific control instructions and sent to the actuator of the robotic arm through a communication interface; the control instructions include PWM signals of the motor, CAN bus instructions or other communication protocols.

[0087] In some examples, state feedback information transmitted in real time during the execution of the task can be received from the robotic arm, including joint angle information, angular velocity information, acceleration information, etc., and the control instructions can be adjusted according to the state feedback information. Specifically, the current angle and angular velocity information of the joint can be obtained by the encoder installed at each joint of the robotic arm. The encoder can be an absolute encoder or an incremental encoder that can provide high-precision position feedback. The interaction force between the robotic arm and the environment can be monitored by the force / torque sensor installed at the end effector or joint of the robotic arm. Cameras or other visual sensors can also be used to monitor the movement and target position of the robotic arm and provide visual feedback for precise positioning and alignment.

[0088] In an embodiment of the present application, the edible fungus bud detection method based on image instance segmentation, after sending the execution instruction to the execution body that executes the mushroom bud collection task, also executes: recording the collected original image data, the results of segmentation processing and the three-dimensional reconstructed data, and storing them in a database or file system; and analyzing the recorded data to evaluate the effectiveness of the current mushroom bud detection and collection strategy.

[0089] Specifically, by analyzing the growth and collection effects of different batches of mushroom buds, the best image collection angle, instance segmentation parameters and collection actions are extracted, and the accuracy and efficiency of mushroom bud detection and collection are analyzed, such as analyzing the accuracy of segmentation results, evaluating the quality of three-dimensional reconstruction models, or monitoring the performance of robotic arm collection operations, etc. In this way, data support can be provided for the adjustment of mushroom bud detection and collection strategies. For example, if the analysis shows that the segmentation accuracy of a certain area is low, the parameters of the segmentation algorithm can be adjusted or an algorithm more suitable for the area can be selected. It should be understood that the autonomous learning algorithm can self-optimize based on historical collection data and real-time feedback. This algorithm continuously learns from actual operations through deep learning models such as YOLOv5 or YOLOv8 to improve the accuracy and efficiency of detection.

[0090] like Figure 6 As shown, a schematic diagram of the structure of an edible mushroom bud detection system based on image instance segmentation in an embodiment of the present invention is shown. The edible mushroom bud detection system based on image instance segmentation 600 includes: an image acquisition module 601, an instance segmentation module 602, a depth information extraction module 603, a three-dimensional reconstruction module 604, a target frame generation module 605, and a control module 606.

[0091] The image acquisition module 601 is used to acquire images of the growth environment of edible fungi through a pre-configured image acquisition device.

[0092] The instance segmentation module 602 is connected to the image acquisition module 601 and is used to perform image instance segmentation on the acquired image of the growth environment of edible fungi based on an image instance segmentation algorithm to segment the mushroom bud area and the background area.

[0093] The depth information extraction module 603 is connected to the instance segmentation module 602 and is used to intercept the depth map according to the instance segmentation result to extract the depth information of the mushroom bud area.

[0094] The three-dimensional reconstruction module 604 is connected to the instance segmentation module 602 and the depth information extraction module 603, and is used to perform three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud.

[0095] The target frame generation module 605 is connected to the 3D reconstruction module 604, and is used to generate a directed 3D frame based on the reconstructed target point cloud to calculate the orientation and coordinates of the target frame, thereby generating the 3D position coordinates and posture morphology information of the mushroom bud.

[0096] The control module 606 is connected to the target frame generation module 605, and is used to generate corresponding execution instructions according to the received 3D position coordinates and posture morphology information of the mushroom buds, and send the execution instructions to the execution body that executes the mushroom bud collection task.

[0097] Preferably, the edible fungus bud detection and collection system 600 in the embodiment of the present application also includes a data management module 607, which is connected to the image acquisition module 601, the three-dimensional reconstruction module 604, and the target frame generation module 605, and is used to record the collected original image data, the results of segmentation processing, and the three-dimensionally reconstructed data, and store them in a database or file system; and analyze the recorded data to evaluate the effectiveness of the current mushroom bud detection and collection strategy.

[0098] It should be understood that the specific process of each module executing the above-mentioned corresponding steps has been described in detail in the above-mentioned embodiment of the edible mushroom bud detection method based on image instance segmentation. For the sake of brevity, it will not be repeated here. It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application can be integrated in a processor, or it can exist physically separately, or two or more modules can be integrated in one module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0099] It should be noted that, in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0100] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0101] Figure 7 is a schematic block diagram of a computer device / equipment / system provided in an embodiment of the present application. Figure 7 As shown in FIG. 1 , the computer device includes: at least one processor 701, a memory 702, at least one network interface 703 and a user interface 705. The various components in the device are coupled together via a bus system 704. It can be understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 704 is not described in detail. Figure 7 In the specification, various buses are labeled as bus systems.

[0102] The user interface 705 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0103] It is understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0104] The memory 702 in the embodiment of the present invention is used to store various types of data to support the operation of the computer device 700. Examples of these data include: any executable program for operating on the computer device 700, such as an operating system 7021 and an application 7022; the operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 7022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The implementation of the XX method provided in the embodiment of the present invention can be included in the application 7022.

[0105] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 701 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0106] In an exemplary embodiment, the computer device 700 may be one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0107] According to the method provided in the embodiments of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes the edible mushroom bud detection method based on image instance segmentation of any embodiment.

[0108] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code is run on a computer, the computer executes the above-mentioned edible mushroom bud detection method based on image instance segmentation.

[0109] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program and / or a computer running on a processor. By way of illustration, both applications and computing devices running on a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0110] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0112] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0113] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0114] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0115] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0116] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0117] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0118] In summary, the present application provides a mushroom bud detection method, system, medium and device based on image instance segmentation, which can realize accurate identification and automatic collection of mushroom buds in the process of edible fungi production through precise detection technology and efficient collection means. By introducing advanced sensing technology, image recognition algorithm and automatic control equipment, the system can monitor the growth state and orientation of mushroom buds in real time, accurately control the collection action, optimize the collection process, reduce the damage to edible fungi, and improve the collection efficiency and quality. Thereby realizing the intelligent and standardized operation of mushroom bud detection and collection in edible fungi production, based on image instance segmentation, providing strong guarantee for the high-quality production of edible fungi. By introducing advanced sensing technology, image recognition algorithm and automatic control equipment, the system can monitor the growth state and orientation of mushroom buds in real time, accurately control the collection action, optimize the collection process, reduce the damage to edible fungi, and improve the collection efficiency and quality. Thereby realizing the intelligent and standardized operation of mushroom bud detection and collection in edible fungi production, providing strong guarantee for the high-quality production of edible fungi. Therefore, the present application effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0119] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for detecting edible mushroom buds based on image instance segmentation, characterized in that: include: Collecting images of the growth environment of edible fungi through a pre-configured image acquisition device; Based on the image instance segmentation algorithm, the collected edible fungus growth environment image is segmented to separate the mushroom bud area and the background area; The depth map is intercepted according to the instance segmentation result to extract the depth information of the mushroom bud area; The image information and depth information of the mushroom bud area are used for three-dimensional reconstruction to generate a target point cloud; Generate a directed three-dimensional frame based on the reconstructed target point cloud to calculate the orientation and coordinates of the target frame, and then generate the 3D position coordinates and posture morphology information of the mushroom bud; A corresponding execution instruction is generated according to the received 3D position coordinates and posture morphology information of the mushroom bud, and the execution instruction is sent to an execution body that executes the mushroom bud collection task.

2. The method for detecting edible mushroom buds based on image instance segmentation according to claim 1, characterized in that: The image instance segmentation algorithm is used to perform image instance segmentation on the collected image of the growth environment of edible fungi, including: Annotate the collected mushroom bud images, and the annotation content includes the instance outline of each mushroom bud; The Mask R-CNN model is used as the basic model architecture for instance segmentation. The labeled mushroom bud image dataset is divided into a training set, a validation set, and a test set. The training set is used to train the Mask R-CNN model. In each training iteration, an online data enhancement operation is performed on the training data; the data enhancement operation includes: any one or more of a rotation operation, a scaling operation, a flipping operation, and an adding operation; After each training cycle, the model is evaluated using the validation set, and evaluation indicators are calculated to measure the accuracy and completeness of the model's segmentation of mushroom buds; The optimized model is finally tested using the test set to evaluate the model's performance in mushroom bud segmentation in real scenarios.

3. The method for detecting edible mushroom buds based on image instance segmentation according to claim 1, characterized in that: The image instance segmentation algorithm is used to perform image instance segmentation on the collected edible fungus growth environment image, including performing a background subtraction operation on the collected mushroom bud image, which includes: Convert the collected RGB image containing mushroom buds into HSV or Lab color space; determine the threshold range in the new color space based on the previous research and prior knowledge of mushroom bud color; Select a deep learning model architecture suitable for image segmentation, and input the image after preliminary threshold segmentation and the corresponding annotation data as training data into the deep learning model for training; The trained model is applied to a new mushroom bud image dataset, and the output of the model is evaluated and optimized based on feedback from actual applications.

4. The method for detecting edible mushroom buds based on image instance segmentation according to claim 1, characterized in that: The image instance segmentation algorithm is used to perform image instance segmentation on the collected edible fungus growth environment image, including performing illumination normalization processing on the collected mushroom bud image, which includes: Use histogram equalization to normalize the illumination of the original image containing mushroom buds; Embed an illumination invariant feature extraction module in the deep learning model to automatically identify and adjust the feature differences of the image caused by illumination changes, and extract features that are not affected by illumination changes; The image data after histogram equalization is used as input and sent to the deep learning model with an illumination invariant feature extraction module for training.

5. The method for detecting edible mushroom buds based on image instance segmentation according to claim 1, characterized in that: The method performs image instance segmentation on the collected edible fungus growth environment image based on the image instance segmentation algorithm, including obtaining images of mushroom buds from different angles using multi-view imaging technology; obtaining complete morphological information of mushroom buds through a three-dimensional reconstruction method, and then performing segmentation.

6. The method for detecting edible mushroom buds based on image instance segmentation according to claim 1, characterized in that: The generating of a directed three-dimensional frame based on the reconstructed target point cloud includes: Use the point cloud segmentation algorithm to identify the point cloud area representing the mushroom buds from the entire point cloud, and separate the identified mushroom bud point cloud from the background and other objects; Fitting and generating a minimum directed three-dimensional box enclosing the target point cloud, and determining the orientation and coordinates of the minimum directed three-dimensional box; The initially fitted minimum directed three-dimensional box is optimized to match the shape and position of the target point cloud, and it is detected whether the minimum directed three-dimensional box collides with other objects.

7. The method for detecting edible mushroom buds based on image instance segmentation according to claim 1, characterized in that: After sending the execution instruction to the execution body that executes the mushroom bud collection task, it also executes: The collected original image data, the results of segmentation processing and the three-dimensional reconstructed data are recorded and stored in a database or file system; and the recorded data are analyzed to evaluate the effectiveness of the current mushroom bud detection and collection strategy.

8. An edible mushroom bud detection system based on image instance segmentation, characterized in that: include: An image acquisition module, used to acquire images of the growth environment of edible fungi through a pre-configured image acquisition device; An instance segmentation module, connected to the image acquisition module, is used to perform image instance segmentation on the collected image of the growth environment of the edible fungi based on an image instance segmentation algorithm to segment the mushroom bud area and the background area; A depth information extraction module, connected to the instance segmentation module, for intercepting the depth map according to the instance segmentation result to extract the depth information of the mushroom bud area; A three-dimensional reconstruction module, connected to the instance segmentation module and the depth information extraction module, for performing three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; A target frame generation module, connected to the 3D reconstruction module, is used to generate a directed 3D frame based on the reconstructed target point cloud, so as to calculate the orientation and coordinates of the target frame, and generate the 3D position coordinates and posture morphology information of the mushroom bud accordingly; The control module is connected to the target frame generation module, and is used to generate corresponding execution instructions according to the received 3D position coordinates and posture morphology information of the mushroom buds, and send the execution instructions to the execution body that executes the mushroom bud collection task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting edible mushroom buds based on image instance segmentation as described in any one of claims 1 to 7 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the edible mushroom bud detection method based on image instance segmentation as described in any one of claims 1 to 7.

Images