Edible mushroom bud detection collection system, method, medium and device

Abstract

The application provides a mushroom bud detection and collection system, method, medium and device. The application adopts advanced sensing technology and image recognition technology, can accurately detect the growth state and production direction of the mushroom bud, accurately control the collection action, optimize the collection process, reduce the damage to the edible fungi, and improve the collection efficiency and quality. Through the precise detection technology and efficient collection means, accurate identification and automatic collection of the mushroom bud are realized. Thus, intelligent and standardized operation of the mushroom bud detection and collection in the edible fungus production is realized, and strong guarantee is provided for high-quality production of the edible fungi.

Description

Technical Field

[0001] This application relates to the technical field of smart agriculture and intelligent detection and collection, and in particular to the detection and collection system, method, medium and device for edible fungi buds. Background Technology

[0002] With the increasing demand for edible fungi, improving the production efficiency and quality of edible fungi has become a top priority. The detection and collection of mushroom buds is a crucial step in the growth process, involving bud identification, growth status assessment, and collection techniques. Traditional methods of mushroom bud detection and collection rely mainly on manual observation and operation, which is not only inefficient but also prone to misjudgment and missed collection. Furthermore, manual operation is also affected by subjective factors and work experience, making it difficult to guarantee the accuracy and consistency of detection and collection.

[0003] To meet market demand for edible fungi and improve the efficiency and quality of mushroom bud detection and harvesting, automation and intelligentization of edible fungi production have become development trends. While existing mushroom bud detection and harvesting equipment has improved work efficiency to some extent, several problems remain. First, the detection accuracy of existing equipment is limited, making it difficult to accurately identify the growth status and maturity of mushroom buds. Second, the harvesting methods are not flexible enough and can easily damage the edible fungi. Furthermore, most existing systems are single-function and cannot achieve comprehensive detection and harvesting of mushroom buds. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a system, method, medium and device for detecting and collecting mushroom buds of edible fungi, so as to solve the technical problems of limited accuracy and lack of flexibility in the existing detection of mushroom buds of edible fungi.

[0005] To achieve the above and other related objectives, the first aspect of this application provides a mushroom bud detection and acquisition system based on the RT-DETR algorithm, comprising: an image acquisition module for acquiring images of the growth environment of edible fungi; a target detection module connected to the image acquisition module for processing the acquired images of the growth environment of edible fungi based on the RT-DETR algorithm and marking mushroom bud targets in the images using 2D bounding boxes; a depth map processing module connected to the target detection module for cropping the depth map according to the 2D bounding boxes to extract depth information of the mushroom bud region; and a three-dimensional reconstruction module connected to... The target detection module and depth map processing module are used to perform three-dimensional reconstruction using the image information and depth information of the mushroom bud region to generate a target point cloud; the target bounding box generation module, connected to the three-dimensional reconstruction module, is used to generate a directed three-dimensional target bounding box surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target bounding box to generate the 3D position coordinates and pose information of the mushroom bud; the control module, connected to the target bounding box generation module, is used to generate corresponding execution instructions based on the received 3D position coordinates and pose information of the mushroom bud, and send the execution instructions to the executor that performs the mushroom bud acquisition task.

[0006] In some embodiments of the first aspect of this application, the target detection module includes: a preprocessing submodule, used to preprocess the collected images of the edible fungi growth environment and label the preprocessed images to form an edible fungi growth environment image dataset; a model training submodule, used to train the RT-DETR model using the labeled edible fungi growth environment image dataset and adjust and optimize the model according to the training results; the trained and optimized RT-DETR model is used to process the real-time collected images of the edible fungi growth environment and output the detected mushroom bud target location and category.

[0007] In some embodiments of the first aspect of this application, the target bounding box generation module preprocesses the target point cloud before generating a directed 3D bounding box using the target point cloud. The preprocessing methods include denoising, filtering, downsampling, and segmentation.

[0008] In some embodiments of the first aspect of this application, the target bounding box generation module includes: a point cloud segmentation submodule, used to identify point cloud regions representing mushroom buds from the entire point cloud using a point cloud segmentation algorithm, and to separate the identified mushroom bud point clouds from the background and other objects; a 3D bounding box generation submodule, used to fit and generate a minimum directed 3D bounding box surrounding the target point cloud, and to determine the orientation and coordinates of the minimum directed 3D bounding box; and a 3D bounding box optimization submodule, used to optimize the initially fitted minimum directed 3D bounding box to match the shape and position of the target point cloud, and to detect whether the minimum directed 3D bounding box collides with other objects.

[0009] In some embodiments of the first aspect of this application, the 3D box generation submodule determines an initially fitted minimum directed 3D box based on the geometric characteristics of the point cloud data; and / or, the 3D box optimization submodule optimizes the initially fitted minimum directed 3D box using a gradient descent algorithm, a genetic algorithm, or a simulated annealing algorithm to iterate the parameters of the 3D box; the parameters of the 3D box include position, size, or rotation angle.

[0010] In some embodiments of the first aspect of this application, the control module performs motion planning based on the 3D position and pose of the mushroom bud target, generates a motion path from the current position to the target position, converts the planned motion path into specific control commands, and sends them to the actuator of the robotic arm through a communication interface.

[0011] In some embodiments of the first aspect of this application, the edible fungus bud detection and acquisition system based on the RT-DETR algorithm further includes: a data management module, connected to the image acquisition module, the three-dimensional reconstruction module, and the target box generation module, for recording the acquired raw image data, the results of segmentation processing, and the data after three-dimensional reconstruction, and storing them in a database or file system; and for analyzing the recorded data to evaluate the effectiveness of the current mushroom bud detection and acquisition strategy.

[0012] To achieve the above and other related objectives, a second aspect of this application provides a method for detecting and acquiring mushroom buds based on the RT-DETR algorithm, comprising: acquiring images of the growth environment of edible fungi; processing the acquired images of the growth environment of edible fungi based on the RT-DETR algorithm, and marking the mushroom bud targets in the images using 2D bounding boxes; cropping the depth map according to the 2D bounding boxes to extract depth information of the mushroom bud region; performing three-dimensional reconstruction using the image information and depth information of the mushroom bud region to generate a target point cloud; generating a directed three-dimensional target box surrounding the mushroom bud based on the target point cloud, and calculating the orientation and coordinates of the target box to generate 3D position coordinates and pose information of the mushroom bud; generating corresponding execution instructions according to the received 3D position coordinates and pose information of the mushroom bud, and sending the execution instructions to the executor performing the mushroom bud acquisition task.

[0013] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting and collecting edible fungus buds based on the RT-DETR algorithm.

[0014] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code, which, when executed on a computer, enables the computer to implement the method for detecting and collecting edible fungus buds based on the RT-DETR algorithm.

[0015] To achieve the above and other related objectives, a fifth aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the method for detecting and collecting edible fungus buds based on the RT-DETR algorithm.

[0016] As described above, the edible fungus bud detection and collection system, method, medium and device of this application have the following beneficial effects.

[0017] (1) Accurate detection: By using advanced image instance segmentation technology and three-dimensional reconstruction algorithm, this invention can accurately identify the position, shape and orientation of mushroom buds, significantly improve detection accuracy and ensure that every detection is accurate.

[0018] (2) High-efficiency collection: By generating a directed three-dimensional box, the present invention provides accurate positioning information for the collection operation, which greatly improves the collection efficiency and minimizes the damage to edible fungi, ensuring the high efficiency of the collection process and the quality of the product.

[0019] (3) Flexible adaptation: It can intelligently adjust and optimize according to different edible fungi varieties and changing growth environment, showing excellent adaptability and ensuring that it can maintain the best performance under various conditions. Attached Figure Description

[0020] Figure 1 The diagram shown is a structural schematic of a mushroom bud detection and acquisition system based on the RT-DETR algorithm in one embodiment of this application.

[0021] Figure 2 The diagram shown is a structural schematic of a target detection module in one embodiment of this application.

[0022] Figure 3 The diagram shown is a schematic representation of the forward propagation process during the training of an RT-DETR model in one embodiment of this application.

[0023] Figure 4 The diagram shown is a schematic representation of the backpropagation process during the training of an RT-DETR model in one embodiment of this application.

[0024] Figure 5 The diagram shown is a structural schematic of the target box generation module in one embodiment of this application.

[0025] Figure 6 The diagram shown is a flowchart illustrating a method for detecting and collecting mushroom buds based on the RT-DETR algorithm in one embodiment of this application.

[0026] Figure 7 The diagram shown is a structural schematic of a computer device according to an embodiment of this application. Detailed Implementation

[0027] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0028] The technical solution of this application aims to provide an intelligent mushroom bud detection and collection system, method, device, and storage medium based on the RT-DETR algorithm. Employing advanced sensing and image recognition technologies, it can accurately detect the growth status and orientation of mushroom buds, precisely control the collection action, optimize the collection process, reduce damage to edible fungi, and improve collection efficiency and quality. Through precise detection technology and efficient collection methods, it achieves accurate identification and automated collection of mushroom buds. This enables intelligent and standardized operation of mushroom bud detection and collection in edible fungi production, providing strong support for high-quality edible fungi production.

[0029] Figure 1 This document illustrates a schematic diagram of a mushroom bud detection and acquisition system based on the RT-DETR algorithm, according to an embodiment of the present invention. The mushroom bud detection and acquisition system 100 based on the RT-DETR algorithm in this embodiment includes: an image acquisition module 101, a target detection module 102, a depth map processing module 103, a 3D reconstruction module 104, a target bounding box generation module 105, a control module 106, and a data management module 107.

[0030] The image acquisition module 101 is used to acquire images of the growth environment of edible fungi. The main task of the image acquisition module 101 is to capture images of the growth environment of edible fungi, including the effects of different temperatures, humidity levels, and light levels on the growth of edible fungi.

[0031] In some examples, the image acquisition module 101 may use a high-resolution camera, infrared imager, or other similar device to acquire images. A high-resolution camera ensures high image clarity and captures details, which is crucial for subsequent image analysis and monitoring of the growth environment. Preferably, the image acquisition module 101 integrates autofocus and exposure control functions to ensure clear images are obtained under various conditions. The infrared imager uses multispectral imaging technology to monitor temperature changes in the edible fungi's growth environment, as different wavelengths of light can reveal different environmental information.

[0032] The target detection module 102 is used to process the acquired images of the growth environment of the edible fungi based on the RT-DETR algorithm, and to mark the mushroom bud targets in the images using 2D bounding boxes.

[0033] It's important to note that RT-DETR (Real-Time DETR) is a real-time object detection model based on the Transformer architecture. It combines the advantages of both Transformer and DETR (Detection Transformer), aiming to achieve high accuracy while maintaining real-time detection speed. The RT-DETR model employs an efficient hybrid encoder design, handling multi-scale features by decoupling intra-scale interactions and cross-scale fusion. It improves object query initialization by utilizing IoU (Intersection over Union)-aware query selection, enabling the model to focus on the most relevant objects in the scene, thereby improving detection accuracy. The RT-DETR model also supports flexible adjustment of inference speed through the use of different decoder layers without retraining.

[0034] In the embodiments of this application, the structure of the target detection module 102 is as follows: Figure 2 As shown, it also includes a preprocessing submodule 102a and a model training submodule 102b. These two submodules are used to process the collected images of the edible fungi's growth environment to detect the mushroom bud target. The specific content is as follows:

[0035] The preprocessing submodule 102a is used to preprocess the collected images of the growth environment of edible fungi and to annotate the preprocessed images to form an image dataset of the growth environment of edible fungi.

[0036] In some examples, the preprocessing of collected images of the growth environment of edible fungi includes: resizing the images to fit the input requirements of the RT-DERT model, normalizing the image data, and data augmentation. Resizing the images involves more than just simple scaling; it also involves interpolation algorithms to estimate pixel values ​​at the new size to maintain image quality and detail, such as nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation. Normalizing the image data involves scaling the image data proportionally to fall within a small, specific range, such as [0,1] or [-1,1]. This process can accelerate the convergence of machine learning algorithms and improve model performance. Image normalization typically involves two steps: data normalization (subtracting the mean) and scaling (dividing by the standard deviation or maximum / minimum value). Data augmentation is a technique that enhances the diversity and quantity of existing data by expanding and transforming it. In image processing, data augmentation methods include, but are not limited to, rotation, scaling, and color adjustment.

[0037] Preferably, the images in the edible fungus growth environment image dataset include mushroom buds of different sizes and growth stages, as well as possible occlusion (such as partial occlusion by other objects) and lighting variations (such as lighting conditions at different times of day). Such a dataset helps the model learn to identify mushroom buds in complex environments. It should be understood that model training is a core step in machine learning, involving training the model using a large amount of labeled data so that it can learn patterns and features from the data. In training the RT-DERT model, a high-quality labeled dataset is required, which should contain various situations in the edible fungus growth environment to ensure that the model can generalize to different real-world scenarios.

[0038] In some examples, labeling the preprocessed images means that each image needs to have an accurate label indicating the location and category of the mushroom buds. This is usually done by labeling with bounding boxes and indicating the type of mushroom bud. This labeling information will be used for loss calculation and optimization during the training process.

[0039] The model training submodule 102b is used to train the RT-DETR model using a labeled dataset of edible fungi growth environment images and to adjust and optimize the model based on the training results. The trained and optimized RT-DETR model is used to process the real-time acquired images of edible fungi growth environment and output the detected mushroom bud target location and category.

[0040] During training, the RT-DETR model predicts the location and category of mushroom buds through forward propagation, and then updates the model parameters through backpropagation to minimize the difference between the prediction and the true label.

[0041] The forward propagation process in the training of the RT-DETR model is as follows: Figure 3 As shown, it includes the following steps:

[0042] Step S31: Input the labeled edible fungus growth environment image dataset into the RT-DETR model; each image contains the target of the mushroom buds, as well as their category and location.

[0043] Step S32: Use an encoder based on the Transformer architecture to extract image features; the image features include shape features, texture features and color features, to capture information in the image related to mushroom bud identification and localization.

[0044] Step S33: Based on the extracted image features, the decoder of the RT-DETR model predicts the location and category of each target in the image.

[0045] Step S34: Compare the prediction results of the RT-DETR model with the true labels and calculate the loss function.

[0046] The loss functions that can be used include, but are not limited to, classification loss functions (such as cross-entropy loss function) and regression loss functions (such as IoU loss or L1 loss), which measure the difference between the predicted bounding boxes and categories and the ground truth labels.

[0047] The backpropagation process in the training of the RT-DETR model is as follows: Figure 4 As shown, it includes the following steps:

[0048] Step S41: Calculate the gradient of the model parameters and use an optimization algorithm to update the model parameters based on the calculated gradient.

[0049] The value of the loss function is obtained by calculating the gradient of the model's parameters. This gradient represents the rate of change of the loss function value relative to the model parameters and indicates how to adjust the parameters to reduce the loss. Optimization algorithms such as gradient descent or its variants (e.g., Adam or SGD) are used to update the model's parameters based on the calculated gradient, thus finding the parameter values ​​that minimize the loss function.

[0050] Step S42: In each iteration, the model's weights and biases are adjusted based on the gradient and learning rate.

[0051] The learning rate determines the step size of each update. A smaller learning rate means finer adjustments, while a larger learning rate may lead to oscillations during training.

[0052] It should be noted that the forward and backward propagation processes are repeated during training, with each iteration aiming to reduce the value of the loss function until the model's performance on the training set no longer improves, or the predetermined number of iterations is reached.

[0053] The depth map processing module 103 is connected to the target detection module 102 and is used to crop the depth map according to the 2D bounding box to extract the depth information of the mushroom bud region.

[0054] Specifically, the depth map processing module 103 converts the pixel coordinates into depth coordinates, i.e., three-dimensional spatial coordinates, based on the depth values ​​in the depth map and the pixel coordinates in the 2D image selected by the 2D bounding box, using camera intrinsic parameters (such as focal length and optical center coordinates). Then, it determines the region corresponding to the mushroom bud in the depth map and extracts the depth information of the corresponding mushroom bud region from the depth map based on the converted depth coordinates. It should be understood that the process of converting pixel coordinates to depth coordinates is existing technology and will not be elaborated upon here.

[0055] The 3D reconstruction module 104 is connected to the target detection module 102 and the depth map processing module 103, and is used to perform 3D reconstruction using the image information and depth information of the mushroom bud region to generate a target point cloud.

[0056] Specifically, the 3D reconstruction module 104 can use structured light, stereo parallax, multi-view stereo matching, or deep learning algorithms to perform 3D reconstruction, forming a target point cloud from the converted 3D coordinate points. Structured light calculates the object's depth information by projecting specific light patterns and capturing their deformation. Stereo parallax uses two or more cameras to capture images from different angles and calculates depth through parallax. Multi-view stereo matching combines images from multiple perspectives and reconstructs the 3D structure through feature matching and parallax calculation. Deep learning algorithms, on the other hand, directly predict the 3D shape of the object from the image by building a deep learning model, without requiring complex geometric calculations.

[0057] A point cloud is a dataset composed of points in three-dimensional space. Each point contains its coordinates in space, as well as information such as color and intensity. The characteristics of point clouds include disorder, denser points closer to the center and sparser points further away, and unstructured data. They are the basic data structures for describing the shape and position of objects in three-dimensional space.

[0058] The target bounding box generation module 105 is connected to the three-dimensional reconstruction module 104 and is used to generate a directional three-dimensional target bounding box surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target bounding box, thereby generating the 3D position coordinates and pose morphology information of the mushroom bud.

[0059] Preferably, before generating a directed 3D bounding box from the target point cloud, the target bounding box generation module 105 preprocesses the target point cloud. Preprocessing methods include, but are not limited to, denoising, filtering, downsampling, and segmentation to improve point cloud quality and reduce the complexity of subsequent processing. Denoising refers to removing noise values ​​from the point cloud data. Filtering refers to filtering the point cloud data to reduce the impact of noise and outliers. Downsampling reduces the density of the point cloud data, thereby improving processing speed and reducing storage space. Segmentation divides the point cloud into different regions or objects to facilitate further analysis and processing.

[0060] The structure of the target box generation module 105 is as follows: Figure 5 As shown, it includes a point cloud segmentation submodule 105a, a 3D bounding box generation module 105b, and a 3D bounding box optimization module 105c. The target bounding box generation module 105 uses these submodules to generate a directed 3D bounding box and calculates the orientation and coordinates of the mushroom bud target bounding box, thereby generating the 3D position coordinates and pose morphology information of the mushroom bud, as detailed below.

[0061] The point cloud segmentation submodule 105a is used to identify the point cloud region representing mushroom buds from the entire point cloud using a point cloud segmentation algorithm, and to separate the identified mushroom bud point cloud from the background and other objects.

[0062] Point cloud segmentation algorithms can be categorized into threshold-based, region-based, or machine learning-based algorithms. Threshold-based segmentation distinguishes different point cloud regions by setting one or more thresholds. First, the points with the lowest height values ​​are selected as the near-ground point set. The segmentation threshold is determined based on the orthogonal distance from these points to the ground's linear model; points with a distance less than the threshold are classified as ground points. Subsequently, the segmentation threshold is dynamically adjusted based on point cloud data characteristics, such as road surface undulations, to adapt to different point cloud data features. Region-based segmentation divides points in the point cloud into different regions based on their local characteristics. This method considers the local geometric properties of the point cloud and can better handle complex scenes. Machine learning-based segmentation algorithms utilize machine learning models (such as deep learning models) to segment point clouds. These methods can learn complex features of point cloud data and are suitable for large-scale and complex point cloud datasets. For example, PointNet / PointNet++, a deep learning network that can take 3D point clouds as input and output segmentation results, can handle the disorder of point cloud data and has good generalization ability.

[0063] The 3D bounding box generation submodule 105b is used to fit and generate a minimum directed 3D bounding box that surrounds the target point cloud, and to determine the orientation and coordinates of the minimum directed 3D bounding box.

[0064] The methods for determining the orientation and coordinates of the minimum directed 3D bounding box include: determining the principal axis direction of the point cloud through principal component analysis (PCA) to estimate the orientation of the target bounding box; calculating the coordinates of the center point of the minimum directed 3D bounding box to determine the size of the minimum directed 3D bounding box.

[0065] It should be understood that a directed 3D bounding box (OBB) is a 3D bounding box with a direction, aligned with the axis, capable of tightly enclosing the target point cloud, and can more accurately adapt to the shape of the point cloud, reducing the bounding space. The center point of the minimum directed 3D bounding box is usually the centroid or center of gravity of the point cloud. The size of the minimum directed 3D bounding box depends on the distribution range of the point cloud along the principal axis, thus obtaining the length, width, and height of the box.

[0066] The principle of Principal Component Analysis (PCA) is to transform the original data into a new coordinate system through linear transformation. In this new coordinate system, the largest variance of any projection of the data lies on the first coordinate (i.e., the first principal component), the second largest variance on the second coordinate (i.e., the second principal component), and so on. Thus, the three principal directions are the three directions with the largest data variance, revealing the most significant trends in the data. In summary, PCA determines the principal component directions of the data by finding the eigenvectors of the data's covariance matrix; these directions best represent the variability of the data.

[0067] The 3D bounding box optimization submodule 105c is used to optimize the initially fitted minimum directed 3D bounding box to match the shape and position of the target point cloud, and to detect whether the minimum directed 3D bounding box collides with other objects.

[0068] It's worth noting that the initial fit is usually determined based on the geometric characteristics of the point cloud data (such as centroid, principal component orientation, and distribution range), providing a rough 3D bounding box, but it may not accurately match the complex shape of the target point cloud. Therefore, further optimization of the 3D bounding box is needed, with the goal of reducing the error between the bounding box and the target point cloud. Specifically, optimization algorithms such as gradient descent, genetic algorithms, and simulated annealing can be used to improve the initial fit. These algorithms iteratively adjust the parameters of the bounding box (such as position, size, and rotation) to find the optimal fit.

[0069] In the embodiments of this application, after optimizing the initially fitted minimum directed 3D bounding box, it is necessary to further detect whether the minimum directed 3D bounding box collides with other objects. Collision detection can prevent the harvesting equipment (such as a harvesting robot or other automated tools) from physically colliding with mushroom buds or other objects when approaching or harvesting mushroom buds, thereby avoiding equipment damage or damage to mushroom buds. Specifically, geometric algorithms can be used for collision detection, such as the Separating Axis Theorem (SAT) or the minimum distance test, to determine whether two or more objects intersect.

[0070] The control module 106 is connected to the target box generation module 105 and is used to generate corresponding execution instructions based on the received 3D position coordinates and pose information of the mushroom buds, and send the execution instructions to the executor that performs the mushroom bud collection task.

[0071] It is worth noting that the control module is a key component of the mushroom bud detection system. It is responsible for converting the 3D position coordinates and morphological information provided by the target box generation module into the actual acquisition actions of the robotic arm. The control module 106 first receives the 3D position coordinates and pose morphological information of the mushroom bud provided by the target box generation module 105, parses the received information, and converts it into instructions that the robotic arm control system can understand and execute.

[0072] In some examples, the control module 106 performs motion planning based on the 3D position and pose of the mushroom bud target, generating a motion path from the current position to the target position; the planned motion path is converted into specific control commands, and sent to the actuator of the robotic arm through a communication interface; the control commands include PWM signals of the motor, CAN bus commands, or other communication protocols.

[0073] In some examples, the control module 106 receives real-time status feedback information from the robotic arm during task execution, including joint angle information, angular velocity information, acceleration information, etc., and adjusts control commands based on the status feedback information. Specifically, the current angle and angular velocity information of the joints can be obtained by encoders installed on each joint of the robotic arm. The encoders can be absolute encoders or incremental encoders, capable of providing high-precision position feedback. Force / torque sensors installed on the end effector of the robotic arm or at the joints can monitor the interaction forces between the robotic arm and the environment. Cameras or other vision sensors can also be used to monitor the movement of the robotic arm and the target position, providing visual feedback to facilitate precise positioning and alignment.

[0074] The data management module 107 connects to the image acquisition module 101, the 3D reconstruction module 104, and the target bounding box generation module 105. It records and stores the acquired raw image data, the segmentation results, and the 3D reconstructed data in a database or file system. Based on a self-learning algorithm, the data management module 107 optimizes detection and acquisition strategies according to historical acquisition data and real-time feedback. This includes analyzing the growth and acquisition effects of different batches of mushroom buds to extract the optimal image acquisition angle, instance segmentation parameters, and acquisition actions. It also analyzes the accuracy and efficiency of mushroom bud detection and acquisition, such as analyzing the accuracy of segmentation results, evaluating the quality of the 3D reconstruction model, or monitoring the performance of the robotic arm's acquisition operation. This provides data support for adjusting the mushroom bud detection and acquisition strategy. For example, if analysis shows that the segmentation accuracy of a certain area is low, the parameters of the segmentation algorithm can be adjusted or a more suitable algorithm for that area can be selected. It should be understood that the self-learning algorithm can self-optimize based on historical acquisition data and real-time feedback. This algorithm continuously learns from actual operation through deep learning models, such as YOLOv5 or YOLOv8, to improve the accuracy and efficiency of detection.

[0075] In summary, the workflow of the mushroom bud detection and acquisition system based on the RT-DETR algorithm is as follows: First, the image acquisition module 101 acquires images of the mushroom growth environment. The target detection module 102 processes the images using the advanced RE-DETR detection algorithm to accurately detect the entire mushroom bud region. The depth map processing module 103 extracts the depth map based on the segmentation results to obtain the depth information of the mushroom bud region. The 3D reconstruction module 104 reconstructs the point cloud of the mushroom bud target region using the image and depth information. The target bounding box generation module 105 generates a directed 3D bounding box based on the point cloud and calculates the orientation and coordinates of the bounding box. In the control module 106, the robotic arm receives the 3D position coordinates and pose generated by the target bounding box to realize the mushroom bud acquisition work, ensuring the stable operation and efficient operation of the mushroom bud detection system.

[0076] The module division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0077] It should also be noted that the edible fungus bud detection and collection system based on the RT-DETR algorithm provided in this application has the characteristics of modular design, autonomous learning and optimization, data collection and remote monitoring.

[0078] Modular design refers to the ability to flexibly adjust the configuration of each functional module according to the characteristics of different edible fungi varieties and growing environments. For example, the image acquisition module can be equipped with cameras of different resolutions to meet the detection needs of mushroom buds of different sizes; the target detection module can select different detection algorithms based on different mushroom bud morphologies to improve the accuracy of segmentation.

[0079] Autonomous learning and optimization refers to the introduction of autonomous learning algorithms, enabling the system to continuously optimize detection and acquisition strategies based on historical detection and acquisition data and real-time feedback. For example, by analyzing the growth and acquisition effects of different batches of mushroom buds, the system can summarize the optimal image acquisition angle, instance segmentation parameters, and acquisition actions, thereby improving the accuracy and efficiency of mushroom bud detection and acquisition.

[0080] Data acquisition and remote monitoring refer to the real-time uploading of data from all modules in the system to the cloud. Administrators can use the remote monitoring system to view the operational status of the mushroom bud detection and acquisition system at any time and perform remote operations and adjustments as needed. The data management module stores and analyzes historical data to provide a reference for future detection and acquisition plans.

[0081] After explaining in detail the structure, working principle and workflow of the edible fungus bud detection and acquisition system based on the RT-DETR algorithm provided by this invention, the following will combine two application cases to help those skilled in the art better understand the wide applicability of the technical solution of this invention.

[0082] Application Case 1: Used for the detection and collection of mushroom buds.

[0083] Common mushrooms are a prevalent edible fungus, and accurate detection and harvesting of their mushroom buds are crucial for improving mushroom yield and quality. In this case study, a highly efficient detection and harvesting system was specifically designed for common mushroom buds. In the image acquisition stage, a high-resolution camera is used to ensure clear capture of bud images within the mushroom's growth environment. The target detection module employs the advanced RT-DETR detection algorithm, accurately identifying the entire mushroom from a complex background. Through accurate detection of the entire mushroom, the depth map processing module extracts complete depth information. Next, the 3D reconstruction module uses this information to quickly construct a complete target point cloud of the mushroom. The target bounding box generation module generates a directed 3D bounding box based on the point cloud, accurately calculating the orientation and coordinates of the mushroom head, providing crucial data for subsequent pose detection. During the harvesting process, the system guides the harvesting device to precisely locate the mushroom bud based on the calculated pose information, achieving efficient and accurate harvesting. The entire process not only improves harvesting efficiency but also reduces damage to the mushrooms, ensuring their quality.

[0084] Application Case 2: Used for the detection and collection of enoki mushroom buds.

[0085] As a popular edible fungus, the accurate detection and harvesting of enoki mushroom buds is crucial for ensuring product quality and yield. Considering the slender nature of enoki mushrooms, simply detecting and segmenting the mushroom heads is a method with significant errors. This embodiment designs a system specifically for detecting and harvesting enoki mushroom buds. The image acquisition module uses a high-resolution camera with targeted parameter optimization to enhance image resolution and ensure clear capture of the slender enoki mushrooms. Advanced target detection algorithms are used to achieve cluster-level detection of enoki mushrooms, accurately separating them from complex backgrounds. The depth map processing module precisely extracts depth information from the mushroom clusters based on the segmentation results. Next, the 3D reconstruction module uses this information to construct a target point cloud of the mushroom clusters. The target bounding box generation module generates directed 3D bounding boxes based on the point cloud, accurately calculating the orientation and coordinates of the mushroom clusters, providing crucial data for pose detection.

[0086] Figure 6 The present invention illustrates a flowchart of a method for detecting and collecting mushroom buds based on the RT-DETR algorithm in an embodiment of the present invention, including steps S61 to S66.

[0087] Step S61: Collect images of the growth environment of edible fungi.

[0088] Step S62: Process the acquired images of the growth environment of the edible fungi based on the RT-DETR algorithm, and use 2D bounding boxes to mark the mushroom buds in the images.

[0089] Step S63: The depth map is cropped according to the 2D bounding box to extract the depth information of the mushroom bud region.

[0090] Step S64: Use the image information and depth information of the mushroom bud region to perform three-dimensional reconstruction to generate a target point cloud.

[0091] Step S65: Generate a directional 3D target bounding box surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target bounding box to generate the 3D position coordinates and pose morphology information of the mushroom bud.

[0092] Step S66: Generate corresponding execution instructions based on the received 3D position coordinates and pose information of the mushroom buds, and send the execution instructions to the executor that performs the mushroom bud collection task.

[0093] It should be noted that the implementation process and principle of the edible mushroom bud detection and collection method based on the RT-DETR algorithm provided in this embodiment of the invention are similar to those of the edible mushroom bud detection and collection system described above, and will not be repeated here.

[0094] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0095] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0096] Figure 7 This is a schematic block diagram of a computer device provided in an embodiment of this application. Figure 7 As shown, the computer device 700 includes at least one processor 701, a memory 702, at least one network interface 703, and a user interface 705. The various components in the device are coupled together via a bus system 704. It is understood that the bus system 704 is used to implement communication between these components. In addition to a data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 7 The general will label all buses as bus systems.

[0097] The user interface 705 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0098] It is understood that memory 702 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0099] In this embodiment of the invention, the memory 702 is used to store various categories of data to support the operation of the computer device 700. Examples of this data include: any executable program for operation on the computer device 700, such as operating system 7021 and application program 7022; operating system 7021 includes various system programs, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. Application program 7022 may include various applications, such as media player, browser, etc., for implementing various application services. The methods provided in this embodiment of the invention may be included in application program 7022.

[0100] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 701. Processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 701 or by instructions in software form. The processor 701 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 701 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 701 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0101] In an exemplary embodiment, the computer device 700 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0102] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute a method for detecting and collecting mushroom buds based on the RT-DETR algorithm.

[0103] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the above-described method.

[0104] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0105] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0110] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0111] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0113] In summary, this application provides a mushroom bud detection and acquisition system, method, medium, and device based on the RT-DETR algorithm. The technical solution of this invention features accurate detection, efficient acquisition, and flexible adaptation. Utilizing advanced image instance segmentation technology and 3D reconstruction algorithms, this invention can accurately identify the position, shape, and orientation of mushroom buds, significantly improving detection accuracy and ensuring error-free detection every time. By generating a directed 3D bounding box, this invention provides precise positioning information for the acquisition operation, greatly improving acquisition efficiency while minimizing damage to the edible fungi, ensuring high efficiency in the acquisition process and product quality. It can intelligently adjust and optimize according to different edible fungi varieties and varying growth environments, exhibiting excellent adaptability and ensuring optimal performance under various conditions. Therefore, this application effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0114] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A mushroom bud detection and acquisition system based on the RT-DETR algorithm, characterized in that, include: The image acquisition module is used to acquire images of the growth environment of edible fungi; the images of the growth environment of edible fungi include mushroom buds of different sizes and growth stages, as well as possible shading and changes in light. The target detection module, connected to the image acquisition module, processes the acquired images of the edible fungi's growth environment based on the RT-DETR algorithm and marks mushroom buds in the images using 2D bounding boxes. The RT-DETR algorithm is a real-time target detection model based on the Transformer architecture. It employs a hybrid coding design, decoupling intra-scale interactions and cross-scale fusion to handle multi-scale features. It improves target query initialization by utilizing IoU-aware query selection, enabling the model to focus on the most relevant targets in the scene. The RT-DETR algorithm also supports adjusting inference speed using different decoder layers without retraining. The target detection module includes a preprocessing submodule for preprocessing the acquired images of the edible fungi's growth environment and labeling the preprocessed images to form an edible fungi growth environment image dataset. Preprocessing includes resizing the images to fit the RT-DETR model. A depth map processing module, connected to the target detection module, is used to crop the depth map based on the 2D bounding box to extract the depth information of the mushroom bud region; The 3D reconstruction module connects the target detection module and the depth map processing module, and is used to perform 3D reconstruction using the image information and depth information of the mushroom bud region to generate a target point cloud; A target bounding box generation module, connected to the 3D reconstruction module, is used to generate a directed 3D target bounding box surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target bounding box to generate the 3D position coordinates and pose morphology information of the mushroom bud. The target bounding box is optimized using gradient descent, genetic, or simulated annealing algorithms to iterate parameters. The target bounding box generation module includes a 3D bounding box generation submodule and a 3D bounding box optimization submodule. The 3D bounding box generation submodule is used to fit and generate a minimum directed 3D bounding box surrounding the target point cloud, and determine the orientation and coordinates of the minimum directed 3D bounding box. The 3D bounding box optimization submodule is used to optimize the initially fitted minimum directed 3D bounding box to match the shape and position of the target point cloud, and to detect whether the minimum directed 3D bounding box collides with other objects. The 3D bounding box optimization submodule uses a geometric algorithm for collision detection to prevent physical collisions between the acquisition device and the mushroom bud or other objects when approaching or picking the mushroom bud. The geometric algorithm includes the separating axis theorem or the minimum distance test algorithm. The control module, connected to the target box generation module, is used to generate corresponding execution instructions based on the received 3D position coordinates and pose information of the mushroom buds, and to send the execution instructions to the executor that performs the mushroom bud collection task.

2. The edible fungus bud detection and acquisition system based on the RT-DETR algorithm according to claim 1, characterized in that, The target detection module includes: The model training submodule is used to train the RT-DETR model using a labeled dataset of edible fungi growth environment images and to adjust and optimize the model based on the training results. The trained and optimized RT-DETR model is used to process real-time acquired images of edible fungi growth environment and output the detected mushroom bud target location and category.

3. The edible fungus bud detection and acquisition system based on the RT-DETR algorithm according to claim 1, characterized in that, Before generating a directed 3D bounding box from the target point cloud, the target bounding box generation module first preprocesses the target point cloud. The preprocessing methods include denoising, filtering, downsampling, and segmentation.

4. The edible fungus bud detection and acquisition system based on the RT-DETR algorithm according to claim 1, characterized in that, The target bounding box generation module further includes: The point cloud segmentation submodule is used to identify the point cloud region representing mushroom buds from the entire point cloud using a point cloud segmentation algorithm, and to separate the identified mushroom bud point cloud from the background and other objects.

5. The edible fungus bud detection and acquisition system based on the RT-DETR algorithm according to claim 4, characterized in that, The 3D box generation submodule determines the initial fitted minimum directed 3D box based on the geometric characteristics of the point cloud data; and / or, the 3D box optimization submodule optimizes the initially fitted minimum directed 3D box using a gradient descent algorithm, a genetic algorithm, or a simulated annealing algorithm to iterate the parameters of the 3D box; the parameters of the 3D box include position, size, or rotation angle.

6. The edible fungus bud detection and acquisition system based on the RT-DETR algorithm according to claim 1, characterized in that, The control module performs motion planning based on the 3D position and pose of the mushroom bud target, generating a motion path from the current position to the target position; the planned motion path is converted into specific control commands and sent to the actuator of the robotic arm through the communication interface.

7. The edible fungus bud detection and acquisition system based on the RT-DETR algorithm according to claim 1, characterized in that, The edible mushroom bud detection and acquisition system based on the RT-DETR algorithm also includes: The data management module, connected to the image acquisition module, the 3D reconstruction module, and the target bounding box generation module, is used to record and store the acquired raw image data, the results of segmentation processing, and the data after 3D reconstruction in a database or file system; and to analyze the recorded data to evaluate the effectiveness of the current mushroom bud detection and acquisition strategy.

8. A method for detecting and collecting mushroom buds based on the RT-DETR algorithm, characterized in that, include: Images of the growth environment of edible fungi are collected; the images of the growth environment of edible fungi include mushroom buds of different sizes and growth stages, as well as possible shading and light changes. The RT-DETR algorithm is used to process the collected images of the edible fungi's growth environment, and 2D bounding boxes are used to label the mushroom buds in the images. The RT-DETR algorithm is a real-time object detection model based on the Transformer architecture. It processes multi-scale features by decoupling intra-scale interactions and cross-scale fusion, and improves target query initialization by utilizing IoU-aware query selection, enabling the model to focus on the most relevant targets in the scene. The RT-DETR algorithm also supports adjusting inference speed by using different decoder layers without retraining. The object detection process includes the following preprocessing sub-processes: preprocessing the collected images of the edible fungi's growth environment and labeling the preprocessed images to form an edible fungi growth environment image dataset; preprocessing includes adjusting image size to fit the RT-DETR model. The depth map is cropped based on the 2D bounding box to extract the depth information of the mushroom bud region; The image and depth information of the mushroom bud region are used to perform three-dimensional reconstruction to generate a target point cloud; Based on the target point cloud, a directed 3D target bounding box surrounding the mushroom bud is generated, and the orientation and coordinates of the target bounding box are calculated to generate the 3D position coordinates and pose morphology information of the mushroom bud. The target bounding box is optimized by iterative parameters using gradient descent, genetic, or simulated annealing algorithms. The target bounding box generation process also includes sub-processes of 3D bounding box generation and 3D bounding box optimization. The 3D bounding box generation sub-process includes fitting and generating a minimum directed 3D bounding box surrounding the target point cloud, and determining the orientation and coordinates of the minimum directed 3D bounding box. The 3D bounding box optimization sub-process includes optimizing the initially fitted minimum directed 3D bounding box to match the shape and position of the target point cloud, and detecting whether the minimum directed 3D bounding box collides with other objects. Among them, the 3D bounding box optimization uses a geometric algorithm for collision detection to prevent physical collisions between the acquisition device and the mushroom bud or other objects when approaching or picking the mushroom bud. The geometric algorithm includes the separating axis theorem or the minimum distance test algorithm. Based on the received 3D position coordinates and pose information of the mushroom buds, corresponding execution instructions are generated and sent to the executor that performs the mushroom bud collection task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the edible fungus bud detection and collection method based on the RT-DETR algorithm as described in claim 8.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the edible fungus bud detection and collection method based on the RT-DETR algorithm as described in claim 8.

Images