Edible mushroom bud detection and collection system, method, medium and device

Abstract

According to the edible mushroom bud detection and collection system, method, medium and device, the advanced sensing technology and the image recognition technology are adopted, the growth state and the production orientation of mushroom buds can be accurately detected, the collection action is accurately controlled, the collection process is optimized, damage to edible mushrooms is reduced, and the collection efficiency and quality are improved. Through an accurate detection technology and an efficient collection means, accurate identification and automatic collection of mushroom buds are realized. Therefore, intelligent and standardized operation of mushroom bud detection and collection in edible mushroom production is realized, and powerful guarantee is provided for high-quality production of edible mushrooms.

Description

Technical Field

[0001] The present application relates to the technical field of smart agriculture and intelligent detection and collection, and in particular to a system, method, medium and device for detecting and collecting edible mushroom buds. Background Art

[0002] With the increasing demand for edible fungi, improving the production efficiency and quality of edible fungi has become a top priority. In the growth process of edible fungi, the detection and collection of mushroom buds are important links, involving the identification of mushroom buds, growth status assessment and collection technology. Traditional mushroom bud detection and collection methods mainly rely on manual observation and manual operation, which is not only inefficient, but also prone to misjudgment and missed collection. In addition, manual operation is also affected by human subjective factors and work experience, making it difficult to ensure the accuracy and consistency of detection and collection.

[0003] In order to meet the market demand for edible fungi and improve the efficiency and quality of mushroom bud detection and collection, the automation and intelligence of edible fungi production has become a development trend. Although the existing mushroom bud detection and collection equipment has improved work efficiency to a certain extent, there are still some problems. First, the detection accuracy of existing equipment is limited, and it is difficult to accurately identify the growth status and maturity of mushroom buds. Secondly, the collection method is not flexible enough and is easy to cause damage to edible fungi. Furthermore, most of the existing systems have a single function and cannot achieve comprehensive detection and collection of mushroom buds. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a system, method, medium and device for detecting and collecting edible mushroom buds, so as to solve the technical problems of limited accuracy and lack of flexibility of the existing edible mushroom bud detection.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides an edible mushroom bud detection and acquisition system based on the RT-DETR algorithm, including: an image acquisition module for acquiring images of the growth environment of edible mushrooms; a target detection module, connected to the image acquisition module, for processing the acquired images of the growth environment of edible mushrooms based on the RT-DETR algorithm, and using a 2D bounding box to mark the mushroom bud targets in the image; a depth map processing module, connected to the target detection module, for intercepting the depth map according to the 2D bounding box to extract the depth information of the mushroom bud area; a three-dimensional reconstruction module, connected to the target detection module The target detection module and the depth map processing module are used to perform three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; the target frame generation module is connected to the three-dimensional reconstruction module, and is used to generate a directed three-dimensional target frame surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target frame, thereby generating the 3D position coordinates and posture morphology information of the mushroom bud; the control module is connected to the target frame generation module, and is used to generate corresponding execution instructions according to the received 3D position coordinates and posture morphology information of the mushroom bud, and send the execution instructions to the execution body that executes the mushroom bud collection task.

[0006] In some embodiments of the first aspect of the present application, the target detection module includes: a preprocessing submodule, used to preprocess the collected edible fungi growth environment images, and annotate the preprocessed images to form an edible fungi growth environment image dataset; a model training submodule, used to use the annotated edible fungi growth environment image dataset to train the RT-DETR model and adjust and optimize the model according to the training results; the trained and optimized RT-DETR model is used to process the edible fungi growth environment images collected in real time, and output the detected mushroom bud target position and category.

[0007] In some embodiments of the first aspect of the present application, the target frame generation module preprocesses the target point cloud before using the target point cloud to generate a directed three-dimensional frame. The preprocessing method includes denoising, filtering, downsampling, and segmentation.

[0008] In some embodiments of the first aspect of the present application, the target frame generation module includes: a point cloud segmentation submodule, which is used to use a point cloud segmentation algorithm to identify the point cloud area representing mushroom buds from the entire point cloud, and separate the identified mushroom bud point cloud from the background and other objects; a three-dimensional frame generation submodule, which is used to fit and generate a minimum directed three-dimensional frame that encloses the target point cloud, and determine the orientation and coordinates of the minimum directed three-dimensional frame; a three-dimensional frame optimization submodule, which is used to optimize the initially fitted minimum directed three-dimensional frame to match the shape and position of the target point cloud, and detect whether the minimum directed three-dimensional frame collides with other objects.

[0009] In some embodiments of the first aspect of the present application, the three-dimensional frame generation submodule determines an initially fitted minimum oriented three-dimensional frame based on the geometric characteristics of the point cloud data; and / or the three-dimensional frame optimization submodule optimizes the initially fitted minimum oriented three-dimensional frame using a gradient descent algorithm, a genetic algorithm, or a simulated annealing algorithm to iterate the parameters of the three-dimensional frame; the parameters of the three-dimensional frame include position, size, or rotation angle.

[0010] In some embodiments of the first aspect of the present application, the control module performs motion planning based on the 3D position and posture of the mushroom bud target, generates a motion path from the current position to the target position; converts the planned motion path into specific control instructions, and sends them to the actuator of the robotic arm through the communication interface.

[0011] In some embodiments of the first aspect of the present application, the edible fungus bud detection and acquisition system based on the RT-DETR algorithm also includes: a data management module, connected to the image acquisition module, the three-dimensional reconstruction module, and the target frame generation module, for recording the collected original image data, the results of segmentation processing, and the three-dimensionally reconstructed data, and storing them in a database or file system; and analyzing the recorded data to evaluate the effectiveness of the current mushroom bud detection and acquisition strategy.

[0012] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a method for detecting and collecting edible mushroom buds based on the RT-DETR algorithm, including: collecting images of the growth environment of edible fungi; processing the collected images of the growth environment of edible fungi based on the RT-DETR algorithm, and using a 2D bounding box to mark the mushroom bud targets in the image; intercepting the depth map according to the 2D bounding box to extract the depth information of the mushroom bud area; performing three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; generating a directed three-dimensional target frame surrounding the mushroom bud based on the target point cloud, and calculating the orientation and coordinates of the target frame to generate the 3D position coordinates and posture morphology information of the mushroom bud; generating corresponding execution instructions based on the received 3D position coordinates and posture morphology information of the mushroom bud, and sending the execution instructions to the execution body that executes the mushroom bud collection task.

[0013] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the edible mushroom bud detection and collection method based on the RT-DETR algorithm.

[0014] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code is run on a computer, the computer implements the edible mushroom bud detection and collection method based on the RT-DETR algorithm.

[0015] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a computer device, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the edible mushroom bud detection and collection method based on the RT-DETR algorithm.

[0016] As described above, the edible mushroom bud detection and collection system, method, medium and device of the present application have the following beneficial effects.

[0017] (1) Accurate detection: By using advanced image instance segmentation technology and three-dimensional reconstruction algorithms, the present invention can accurately identify the position, shape and orientation of mushroom buds, significantly improving detection accuracy and ensuring that every detection is accurate.

[0018] (2) Efficient collection: By generating a directed three-dimensional frame, the present invention provides accurate positioning information for the collection operation, greatly improving the collection efficiency while minimizing the damage to edible fungi, thus ensuring the efficiency of the collection process and the quality of the product.

[0019] (3) Flexible adaptability: It can intelligently adjust and optimize according to different edible fungi varieties and changing growth environments, showing excellent adaptability and ensuring optimal performance under various conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Shown is a structural schematic diagram of an edible mushroom bud detection and collection system based on the RT-DETR algorithm in one embodiment of the present application.

[0021] Figure 2 Shown is a structural schematic diagram of a target detection module in one embodiment of the present application.

[0022] Figure 3 Shown is a schematic diagram of the forward propagation process in RT-DETR model training in one embodiment of the present application.

[0023] Figure 4 Shown is a schematic diagram of the back propagation process in RT-DETR model training in one embodiment of the present application.

[0024] Figure 5 Shown is a structural diagram of a target frame generation module in one embodiment of the present application.

[0025] Figure 6 Shown is a flow chart of a method for detecting and collecting edible mushroom buds based on the RT-DETR algorithm in one embodiment of the present application.

[0026] Figure 7 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] The technical solution of the embodiment of the present application aims to provide an intelligent edible mushroom bud detection and collection system, method, device, and storage medium based on the RT-DETR algorithm, which adopts advanced sensing technology and image recognition technology, can accurately detect the growth status and production direction of mushroom buds, accurately control the collection action, optimize the collection process, reduce damage to edible fungi, and improve collection efficiency and quality. Through precise detection technology and efficient collection means, accurate identification and automated collection of mushroom buds are achieved. Thereby, intelligent and standardized operations of mushroom bud detection and collection in edible fungi production are realized, providing strong guarantee for the high-quality production of edible fungi.

[0029] Figure 1 The structural diagram of an edible mushroom bud detection and acquisition system based on the RT-DETR algorithm in an embodiment of the present invention is shown. The edible mushroom bud detection and acquisition system 100 based on the RT-DETR algorithm in the embodiment of the present application includes: an image acquisition module 101, a target detection module 102, a depth map processing module 103, a three-dimensional reconstruction module 104, a target frame generation module 105, a control module 106, and a data management module 107.

[0030] The image acquisition module 101 is used to acquire images of the growth environment of edible fungi. The main task of the image acquisition module 101 is to capture images of the growth environment of edible fungi, including the effects of different temperature, humidity, light and other conditions on the growth of edible fungi.

[0031] In some examples, the image acquisition module 101 may use a high-resolution camera, an infrared imager, or other device to acquire images. A high-resolution camera can ensure that the acquired images are clear and can capture details, which is essential for subsequent image analysis and growth environment monitoring. Preferably, the image acquisition module 101 is integrated with autofocus and exposure control functions to ensure that clear images can be obtained in various environments. The infrared imager uses multispectral imaging technology to monitor temperature changes in the growth environment of edible fungi, because different wavelengths of light can reveal different environmental information.

[0032] The target detection module 102 is used to process the collected image of the growth environment of the edible fungi based on the RT-DETR algorithm, and use a 2D bounding box to mark the mushroom bud target in the image.

[0033] It should be noted here that RT-DETR (Real-Time DETR) is a real-time target detection model based on the Transformer architecture. It combines the advantages of Transformer and DETR (Detection Transformer) to achieve high accuracy while maintaining real-time detection speed. The RT-DETR model adopts an efficient hybrid encoder design to process multi-scale features by decoupling intra-scale interactions and cross-scale fusion; it improves target query initialization by utilizing IoU (intersection over union)-aware query selection, allowing the model to focus on the most relevant targets in the scene, thereby improving detection accuracy. The RT-DETR model also supports flexible adjustment of inference speed by using different decoder layers without retraining.

[0034] In the embodiment of the present application, the structure of the target detection module 102 is as follows: Figure 2 As shown, it also includes a preprocessing submodule 102a and a model training submodule 102b, through which the collected edible fungus growth environment images are processed to detect mushroom bud targets, and the specific contents are as follows: The preprocessing submodule 102a is used to preprocess the collected edible fungus growth environment images, and annotate the preprocessed images to form an edible fungus growth environment image dataset.

[0035] In some examples, the collected images of the growth environment of edible fungi are preprocessed, which includes: adjusting the image size to adapt to the input requirements of the RT-DERT model, normalizing the image data, and data enhancement. The process of adjusting the image size is not just a simple scaling, it also involves an interpolation algorithm to estimate the pixel value at the new size to maintain the quality and details of the image, such as the nearest neighbor difference algorithm, the bilinear difference algorithm, and the bicubic interpolation algorithm. Normalizing image data is to scale the image data so that it falls into a small specific interval, such as [0,1] or [-1,1]. This process can speed up the convergence of the machine learning algorithm and improve the performance of the model. Image normalization usually includes two steps: data centering (subtracting the mean) and scaling (dividing by the standard deviation or the maximum and minimum values). Data enhancement is a technology that enhances the diversity and quantity of data by expanding and transforming existing data. In image processing, data enhancement methods include but are not limited to rotation, scaling, color adjustment, etc.

[0036] Preferably, the images in the edible fungus growth environment image dataset contain mushroom buds of different sizes and different growth stages, as well as possible occlusions (such as partial occlusion by other objects) and lighting changes (such as lighting conditions at different times). Such a dataset can help the model learn the ability to identify mushroom buds in complex environments. It should be understood that model training is a core step in machine learning, which involves using a large amount of labeled data to train the model so that it can learn patterns and features in the data. In the training of the RT-DERT model, a high-quality labeled dataset is required. This dataset should contain various situations in the edible fungus growth environment to ensure that the model can be generalized to different actual scenarios.

[0037] In some examples, labeling the preprocessed images means that each image needs to be accurately labeled to indicate the location and category of the mushroom buds, usually through bounding boxes and indicating the type of mushroom buds. These labeling information will be used for loss calculation and optimization during training.

[0038] The model training submodule 102b is used to use the labeled edible fungus growth environment image data set to train the RT-DETR model and adjust and optimize the model according to the training results; the trained and optimized RT-DETR model is used to process the edible fungus growth environment images collected in real time, and output the detected mushroom bud target position and category.

[0039] During the training process, the RT-DETR model predicts the location and category of mushroom buds through forward propagation, and then updates the model parameters through backpropagation to minimize the difference between the prediction and the true annotation.

[0040] The forward propagation process in RT-DETR model training is as follows Figure 3 As shown, the following steps are included: Step S31: input the labeled edible fungus growth environment image dataset into the RT-DETR model; each image contains mushroom bud targets, as well as their categories and locations.

[0041] Step S32: Use an encoder based on the Transformer architecture to extract image features; the image features include shape features, texture features and color features, which are used to capture information related to mushroom bud identification and positioning in the image.

[0042] Step S33: Based on the extracted image features, the decoder of the RT-DETR model predicts the location and category of each object in the image.

[0043] Step S34: Compare the prediction results of the RT-DETR model with the true annotations and calculate the loss function.

[0044] The available loss functions include, but are not limited to, classification loss functions (such as cross entropy loss) and regression loss functions (such as IoU loss or L1 loss), which measure the difference between the predicted bounding box and category and the true annotation.

[0045] The back propagation process in RT-DETR model training is as follows Figure 4 As shown, the following steps are included: Step S41: perform gradient calculation on the parameters of the model, and use an optimization algorithm to update the parameters of the model according to the calculated gradient.

[0046] The value of the loss function requires the calculation of the gradient of the model parameters. This gradient represents the rate of change of the loss function value relative to the model parameters, which is used to indicate how to adjust the parameters to reduce the loss. Using an optimization algorithm such as gradient descent or its variants (such as Adam or SGD), the parameters of the model are updated according to the calculated gradient, so that the parameter value that minimizes the loss function can be found.

[0047] Step S42: In each iteration, the weights and biases of the model are adjusted according to the gradient and learning rate.

[0048] The learning rate determines the step size of each update. A smaller learning rate means more refined adjustments, while a larger learning rate may lead to oscillations during training.

[0049] It should be noted that the processes of forward propagation and backward propagation are repeated during training, and each iteration aims to reduce the value of the loss function until the performance of the model on the training set no longer improves, or the predetermined number of iterations is reached.

[0050] The depth map processing module 103 is connected to the target detection module 102 and is used to intercept the depth map according to the 2D bounding box to extract the depth information of the mushroom bud area.

[0051] Specifically, the depth map processing module 103 converts the pixel coordinates into depth coordinates, i.e., three-dimensional space coordinates, based on the depth values ​​in the depth map and the pixel coordinates in the 2D image selected by the 2D bounding box, through the camera internal parameters (such as focal length, optical center coordinates). Subsequently, the area corresponding to the mushroom bud in the depth map is determined, and the depth information of the corresponding mushroom bud area is intercepted from the depth map according to the converted depth coordinates. It should be understood that the process of converting pixel coordinates into depth coordinates is a prior art and will not be repeated here.

[0052] The three-dimensional reconstruction module 104 is connected to the target detection module 102 and the depth map processing module 103, and is used to perform three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud.

[0053] Specifically, the 3D reconstruction module 104 can use structured light method, stereo disparity method, multi-view stereo matching method or deep learning algorithm to perform 3D reconstruction, and the converted 3D coordinate point set forms a target point cloud. Among them, the structured light method calculates the depth information of the object by projecting a specific light pattern and capturing its deformation. The stereo disparity method uses two or more cameras to capture images from different angles and calculates the depth through disparity. The multi-view stereo matching method combines images from multiple perspectives and reconstructs the 3D structure through feature matching and disparity calculation. The deep learning algorithm predicts the 3D shape of the object directly from the image by constructing a deep learning model without the need for complex geometric calculations.

[0054] Point cloud is a data set composed of a set of points in three-dimensional space, each point contains its coordinates in space, as well as information such as color and intensity. Point clouds have the characteristics of disorder, close density and far sparseness, and unstructured. They are the basic data structure for describing the shape and position of objects in three-dimensional space.

[0055] The target frame generation module 105 is connected to the 3D reconstruction module 104, and is used to generate a directed 3D target frame surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target frame to generate the 3D position coordinates and posture morphology information of the mushroom bud.

[0056] Preferably, the target frame generation module 105 pre-processes the target point cloud before using the target point cloud to generate a directed three-dimensional frame. The pre-processing methods include but are not limited to denoising, filtering, downsampling, segmentation, etc., to improve the quality of the point cloud and reduce the complexity of subsequent processing. Denoising refers to removing noise values ​​in the point cloud data. Filtering refers to filtering the point cloud data to reduce the impact of noise and outliers. Downsampling can reduce the density of point cloud data to increase processing speed and reduce storage space. Segmentation is to divide the point cloud into different areas or objects for further analysis and processing.

[0057] The structure of the target frame generation module 105 is as follows Figure 5 As shown, it includes a point cloud segmentation submodule 105a, a 3D frame generation module 105b, and a 3D frame optimization module 105c. The target frame generation module 105 generates a directed 3D frame by using these submodules and calculates the orientation and coordinates of the mushroom bud target frame, thereby generating the 3D position coordinates and posture morphological information of the mushroom bud, and the specific contents are as follows.

[0058] The point cloud segmentation submodule 105a is used to identify the point cloud region representing the mushroom buds from the entire point cloud using a point cloud segmentation algorithm, and separate the identified mushroom bud point cloud from the background and other objects.

[0059] Point cloud segmentation algorithms can be, for example, threshold-based segmentation algorithms, region-based segmentation algorithms, or machine learning-based segmentation algorithms. Threshold-based segmentation algorithms distinguish different point cloud regions by setting one or more thresholds. First, select the point with the lowest height value as the near-ground point set, determine the segmentation threshold based on the orthogonal distance from the point to the ground straight line model, and classify the points whose orthogonal distance from the point to the ground straight line model is less than the threshold as ground points; then, according to the characteristics of the point cloud data, such as the amplitude of road surface fluctuation, dynamically adjust the segmentation threshold to adapt to different point cloud data characteristics. Region-based segmentation algorithms divide the points in the point cloud into different regions according to their local characteristics. This method takes into account the local geometric characteristics of the point cloud and can better handle complex scenes. Machine learning-based segmentation algorithms refer to the use of machine learning models (such as deep learning models) to segment point clouds. These methods can learn the complex features of point cloud data and are suitable for large-scale and complex point cloud data sets. For example, PointNet / PointNet++ can be used as a deep learning network that inputs 3D point clouds and outputs segmentation results. It can handle the disorder of point cloud data and has good generalization ability.

[0060] The three-dimensional frame generation submodule 105b is used to fit and generate a minimum directional three-dimensional frame surrounding the target point cloud, and determine the orientation and coordinates of the minimum directional three-dimensional frame.

[0061] The method of determining the orientation and coordinates of the minimum directed three-dimensional frame includes: determining the principal axis direction of the point cloud by principal component analysis (PCA) to estimate the orientation of the target frame; calculating the center point coordinates of the minimum directed three-dimensional frame to determine the size of the minimum directed three-dimensional frame.

[0062] It should be understood that a directed 3D box refers to a 3D box with a direction (Oriented Bounding Box, OBB), which is aligned with the axis and can tightly surround the target point cloud, can more accurately adapt to the shape of the point cloud, and reduce the surrounding space. The center point of the minimum directed 3D box is usually the center of mass or center of gravity of the point cloud. The size of the minimum directed 3D box depends on the distribution range of the point cloud along the main axis, thus obtaining the length, width and height of the box.

[0063] The principle of principal component analysis is to transform the original data into a new coordinate system through linear transformation, so that in this new coordinate system, the first largest variance of any projection of the data is on the first coordinate (i.e., the first principal component), the second largest variance is on the second coordinate (i.e., the second principal component), and so on. In this way, the three main directions are the three directions with the largest data variance, which reveal the most significant trend of change in the data. In summary, PCA determines the principal component directions of the data by finding the eigenvectors of the data covariance matrix, which can best represent the variability of the data.

[0064] The 3D box optimization submodule 105c is used to optimize the initially fitted minimum directional 3D box to make it match the shape and position of the target point cloud, and detect whether the minimum directional 3D box collides with other objects.

[0065] It is worth noting that the initial fit is usually determined based on the geometric characteristics of the point cloud data (such as the centroid, the direction of the principal component, and the distribution range). It can provide a rough three-dimensional box, but it may not accurately match the complex shape of the target point cloud. Therefore, the three-dimensional box needs to be further optimized, and the goal of the optimization is to reduce the error between the three-dimensional box and the target point cloud. Specifically, the initial fit can be improved using optimization algorithms such as gradient descent, genetic algorithms, and simulated annealing. These algorithms find the best fit by iteratively adjusting the parameters of the box (such as position, size, and rotation).

[0066] In the embodiments of the present application, after optimizing the initially fitted minimum directed three-dimensional box, it is necessary to further detect whether the minimum directed three-dimensional box collides with other objects. Collision detection can prevent the collection equipment (such as a picking manipulator or other automated tools) from physically colliding with mushroom buds or other objects when approaching or picking mushroom buds, thereby avoiding damage to the equipment or damage to the mushroom buds. Specifically, geometric algorithms can be used for collision detection, such as the separating axis theorem (SAT) or the minimum distance test, to determine whether two or more objects intersect.

[0067] The control module 106 is connected to the target frame generation module 105, and is used to generate corresponding execution instructions according to the received 3D position coordinates and posture morphology information of the mushroom buds, and send the execution instructions to the execution body that executes the mushroom bud collection task.

[0068] It is worth noting that the control module is a key part of the mushroom bud detection system, which is responsible for converting the 3D position coordinates and morphological information provided by the target frame generation module into the actual acquisition action of the robot arm. The control module 106 first receives the 3D position coordinates and posture morphological information of the mushroom bud provided by the target frame generation module 105, parses the received information, and converts it into instructions that the robot arm control system can understand and execute.

[0069] In some examples, the control module 106 performs motion planning based on the 3D position and posture of the mushroom bud target to generate a motion path from the current position to the target position; converts the planned motion path into specific control instructions and sends them to the actuator of the robotic arm through the communication interface; the control instructions include the PWM signal of the motor, CAN bus instructions or other communication protocols.

[0070] In some examples, the control module 106 receives state feedback information transmitted in real time from the robotic arm during the execution of the task, including joint angle information, angular velocity information, acceleration information, etc., and adjusts the control instructions according to the state feedback information. Specifically, the current angle and angular velocity information of the joint can be obtained by an encoder installed at each joint of the robotic arm. The encoder can be an absolute encoder or an incremental encoder that can provide high-precision position feedback. The interaction force between the robotic arm and the environment can be monitored by a force / torque sensor installed at the end effector or joint of the robotic arm. A camera or other visual sensor can also be used to monitor the movement and target position of the robotic arm and provide visual feedback to facilitate precise positioning and alignment.

[0071] The data management module 107 is connected to the image acquisition module 101, the three-dimensional reconstruction module 104, and the target frame generation module 105, and is used to record the collected original image data, the results of the segmentation processing, and the data after the three-dimensional reconstruction, and store them in a database or a file system. The data management module 107 is based on an autonomous learning algorithm, and optimizes the detection and acquisition strategy according to historical acquisition data and real-time feedback, including: by analyzing the growth conditions and acquisition effects of different batches of mushroom buds, refining the best image acquisition angle, instance segmentation parameters and acquisition actions, analyzing the accuracy and efficiency of mushroom bud detection and acquisition, such as analyzing the accuracy of the segmentation results, evaluating the quality of the three-dimensional reconstruction model, or monitoring the performance of the acquisition operation of the mechanical arm, etc. In this way, data support can be provided for the adjustment of the mushroom bud detection acquisition strategy. For example, if the analysis shows that the segmentation accuracy of a certain area is low, the parameters of the segmentation algorithm can be adjusted or an algorithm more suitable for the area can be selected. It should be understood that the autonomous learning algorithm can be self-optimized according to historical acquisition data and real-time feedback. This algorithm continuously learns from actual operations through deep learning models, such as YOLOv5 or YOLOv8, to improve the accuracy and efficiency of detection.

[0072] In summary, the workflow of the edible mushroom bud detection and acquisition system based on the RT-DETR algorithm is as follows: First, the image acquisition module 101 acquires the image of the edible mushroom growth environment. The target detection module 102 uses the advanced RE-DETR detection algorithm to process the image and accurately detect the overall area of ​​the mushroom bud. The depth map processing module 103 intercepts the depth map according to the segmentation result to obtain the depth information of the mushroom bud area. The three-dimensional reconstruction module 104 uses the image information and depth information to reconstruct the point cloud of the mushroom bud target area. The target frame generation module 105 generates a directed three-dimensional frame based on the point cloud and calculates the orientation and coordinates of the target frame. In the control module 106, the robotic arm accepts the 3D position coordinates and posture generated by the target frame to realize the mushroom bud collection work and ensure the stable operation and efficient operation of the mushroom bud detection system.

[0073] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0074] It should also be noted that the edible mushroom bud detection and collection system based on the RT-DETR algorithm provided in the embodiment of the present application has the characteristics of modular design, autonomous learning and optimization, data collection and remote monitoring.

[0075] Modular design means that the configuration of each functional module can be flexibly adjusted according to the characteristics of different edible fungi varieties and growth environments. For example, the image acquisition module can replace cameras with different resolutions according to demand to meet the detection needs of mushroom buds of different sizes; the target detection module can select different detection algorithms according to different mushroom bud morphologies to improve segmentation accuracy.

[0076] Autonomous learning and optimization refers to the introduction of autonomous learning algorithms. The system can continuously optimize detection and collection strategies based on historical detection and collection data and real-time feedback. For example, by analyzing the growth and collection effects of different batches of mushroom buds, the system can summarize the best image collection angle, instance segmentation parameters and collection actions to improve the accuracy and efficiency of mushroom bud detection and collection.

[0077] Data collection and remote monitoring means that the data of all modules in the system can be uploaded to the cloud in real time. The manager can check the operating status of the mushroom bud detection and collection system at any time through the remote monitoring system, and perform remote operations and adjustments as needed. The data management module stores and analyzes historical data to provide reference for future detection and collection plans.

[0078] After explaining in detail the structure, working principle and workflow of the edible mushroom bud detection and collection system based on the RT-DETR algorithm provided by the present invention, two application cases will be combined below to assist technical personnel in this field to better understand the wide application of the technical solution of the present invention.

[0079] Application Case 1: Used for the detection and collection of common mushroom buds.

[0080] Common mushrooms are common edible fungi, and accurate detection and collection of their buds are crucial to improving mushroom yield and quality. In this case, an efficient detection and collection system is designed specifically for common mushroom buds. In the image acquisition process, a high-resolution camera is used to ensure that the image of the mushroom buds in the mushroom growth environment can be clearly captured. The target detection module uses the advanced detection algorithm RT-DETR to accurately detect the entire mushroom from the complex background. Through accurate detection of the entire mushroom, the depth map processing module can extract the complete depth information of the mushroom. Then, the 3D reconstruction module uses this information to quickly construct the target point cloud of the complete mushroom, and the target frame generation module generates a directed 3D frame based on the point cloud, thereby accurately calculating the orientation and coordinates of the mushroom head, providing key data for subsequent posture detection. During the acquisition process, the system can guide the acquisition equipment to accurately locate the mushroom buds based on the calculated posture information, realizing efficient and accurate acquisition operations. The whole process not only improves the acquisition efficiency, but also reduces damage to the mushrooms and ensures the quality of the mushrooms.

[0081] Application Case 2: Used for the detection and collection of Flammulina velutipes buds.

[0082] As a popular edible mushroom, accurate detection and collection of the buds of Flammulina velutipes is crucial to ensure product quality and yield. Considering the slender characteristics of Flammulina velutipes, only detecting and segmenting the head of the velutipes is a solution with large errors. This embodiment designs a detection and collection system specifically for the buds of Flammulina velutipes. The image acquisition module uses a high-resolution camera and performs targeted parameter optimization to enhance the image resolution and ensure that the slender Flammulina velutipes can be clearly captured. The advanced target detection algorithm is used to detect the velutipes at the cluster level, which can accurately separate the velutipes from the complex background. The depth map processing module accurately intercepts the depth map according to the segmentation results to obtain the depth information of the velutipes cluster. Then, the 3D reconstruction module uses this information to construct the target point cloud of the velutipes cluster, and the target frame generation module generates a directed 3D frame based on the point cloud, so as to accurately calculate the orientation and coordinates of the velutipes cluster, providing key data for posture detection.

[0083] Figure 6 A flow chart of a method for detecting and collecting edible mushroom buds based on the RT-DETR algorithm in an embodiment of the present invention is shown, including steps S61 to S66.

[0084] Step S61: Collecting images of the growth environment of edible fungi.

[0085] Step S62: Processing the collected image of the growth environment of the edible fungi based on the RT-DETR algorithm, and marking the mushroom bud targets in the image with a 2D bounding box.

[0086] Step S63: intercepting the depth map according to the 2D bounding box to extract depth information of the mushroom bud area.

[0087] Step S64: Perform three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud.

[0088] Step S65: Generate a directed three-dimensional target frame surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target frame to generate the 3D position coordinates and posture morphology information of the mushroom bud.

[0089] Step S66: Generate corresponding execution instructions according to the received 3D position coordinates and posture information of the mushroom buds, and send the execution instructions to the execution body that executes the mushroom bud collection task.

[0090] It should be noted that the implementation process and principle of the edible mushroom bud detection and collection method based on the RT-DETR algorithm provided in the embodiment of the present invention are similar to those of the edible mushroom bud detection and collection system mentioned above, and will not be repeated here.

[0091] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0092] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0093] Figure 7 is a schematic block diagram of a computer device provided in an embodiment of the present application. Figure 7 As shown, the computer device 700 includes: at least one processor 701, a memory 702, at least one network interface 703 and a user interface 705. The various components in the device are coupled together through a bus system 704. It can be understood that the bus system 704 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 704 is not used in the embodiment of the present invention. Figure 7 In the specification, various buses are labeled as bus systems.

[0094] The user interface 705 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0095] It is understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0096] The memory 702 in the embodiment of the present invention is used to store various types of data to support the operation of the computer device 700. Examples of these data include: any executable program for operating on the computer device 700, such as an operating system 7021 and an application 7022; the operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 7022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The implementation of the XX method provided in the embodiment of the present invention can be included in the application 7022.

[0097] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 701 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0098] In an exemplary embodiment, the computer device 700 may be one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0099] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes the edible mushroom bud detection and collection method based on the RT-DETR algorithm.

[0100] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the above method.

[0101] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program and / or a computer running on a processor. By way of illustration, both applications and computing devices running on a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0102] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0104] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0105] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0107] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0108] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0109] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0110] In summary, the present application provides a mushroom bud detection and collection system, method, medium and device based on the RT-DETR algorithm. The technical solution of the present invention has the characteristics of accurate detection, efficient collection and flexible adaptation. Utilizing advanced image instance segmentation technology and three-dimensional reconstruction algorithm, the present invention can accurately identify the position, shape and orientation of mushroom buds, significantly improve the detection accuracy, and ensure that each detection is accurate. By generating a directed three-dimensional frame, the present invention provides accurate positioning information for the collection operation, greatly improves the collection efficiency, and at the same time minimizes the damage to edible fungi, ensuring the efficiency of the collection process and the quality of the product. It can be intelligently adjusted and optimized according to different edible fungi varieties and changing growth environments, showing excellent adaptability, and ensuring that the best performance can be maintained under various conditions. Therefore, the present application effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0111] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. An edible mushroom bud detection and collection system based on RT-DETR algorithm, characterized in that: include: An image acquisition module, used to acquire images of the growth environment of edible fungi; A target detection module, connected to the image acquisition module, is used to process the collected image of the growth environment of the edible fungi based on the RT-DETR algorithm, and use a 2D bounding box to mark the mushroom bud target in the image; A depth map processing module, connected to the target detection module, for intercepting the depth map according to the 2D bounding box to extract the depth information of the mushroom bud area; A three-dimensional reconstruction module, connected to the target detection module and the depth map processing module, for performing three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; A target frame generation module, connected to the 3D reconstruction module, is used to generate a directed 3D target frame surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target frame, thereby generating the 3D position coordinates and posture morphology information of the mushroom bud; The control module is connected to the target frame generation module, and is used to generate corresponding execution instructions according to the received 3D position coordinates and posture morphology information of the mushroom buds, and send the execution instructions to the execution body that executes the mushroom bud collection task.

2. The edible mushroom bud detection and collection system based on RT-DETR algorithm according to claim 1, characterized in that: The target detection module comprises: A preprocessing submodule is used to preprocess the collected edible fungus growth environment images and annotate the preprocessed images to form an edible fungus growth environment image dataset; The model training submodule is used to train the RT-DETR model using the labeled edible fungus growth environment image dataset and adjust and optimize the model according to the training results; the trained and optimized RT-DETR model is used to process the edible fungus growth environment images collected in real time and output the detected mushroom bud target location and category.

3. The edible mushroom bud detection and collection system based on RT-DETR algorithm according to claim 1, characterized in that: The target frame generation module preprocesses the target point cloud before generating a directed three-dimensional frame using the target point cloud. The preprocessing method includes denoising, filtering, down sampling, and segmentation.

4. The edible mushroom bud detection and collection system based on RT-DETR algorithm according to claim 1, characterized in that: The target frame generation module includes: The point cloud segmentation submodule is used to identify the point cloud area representing the mushroom buds from the entire point cloud using a point cloud segmentation algorithm, and separate the identified mushroom bud point cloud from the background and other objects; A 3D box generation submodule is used to fit and generate a minimum directional 3D box surrounding the target point cloud, and determine the orientation and coordinates of the minimum directional 3D box; The 3D box optimization submodule is used to optimize the initially fitted minimum directed 3D box so that it matches the shape and position of the target point cloud, and detect whether the minimum directed 3D box collides with other objects.

5. The edible mushroom bud detection and collection system based on RT-DETR algorithm according to claim 4 is characterized in that: The 3D frame generation submodule determines the minimum directional 3D frame of the initial fit based on the geometric characteristics of the point cloud data; and / or, the 3D frame optimization submodule optimizes the minimum directional 3D frame of the initial fit using a gradient descent algorithm, a genetic algorithm or a simulated annealing algorithm to iterate the parameters of the 3D frame; the parameters of the 3D frame include position, size or rotation angle.

6. The edible mushroom bud detection and collection system based on RT-DETR algorithm according to claim 1, characterized in that: The control module performs motion planning according to the 3D position and posture of the mushroom bud target, generates a motion path from the current position to the target position; converts the planned motion path into specific control instructions, and sends them to the actuator of the robotic arm through the communication interface.

7. The edible mushroom bud detection and collection system based on RT-DETR algorithm according to claim 1, characterized in that: The edible mushroom bud detection and collection system based on the RT-DETR algorithm also includes: The data management module is connected to the image acquisition module, the three-dimensional reconstruction module, and the target frame generation module, and is used to record the acquired original image data, the results of the segmentation processing, and the three-dimensionally reconstructed data, and store them in a database or a file system; and to analyze the recorded data to evaluate the effectiveness of the current mushroom bud detection and acquisition strategy.

8. A method for detecting and collecting edible mushroom buds based on RT-DETR algorithm, characterized in that: include: Collect images of the growth environment of edible fungi; The collected image of the growth environment of the edible fungus is processed based on the RT-DETR algorithm, and the mushroom bud target in the image is marked with a 2D bounding box; Intercepting the depth map according to the 2D bounding box to extract depth information of the mushroom bud area; Performing three-dimensional reconstruction using the image information and depth information of the mushroom bud area to generate a target point cloud; Generate a directed three-dimensional target frame surrounding the mushroom bud based on the target point cloud, and calculate the orientation and coordinates of the target frame, thereby generating 3D position coordinates and posture morphological information of the mushroom bud; A corresponding execution instruction is generated according to the received 3D position coordinates and posture morphology information of the mushroom bud, and the execution instruction is sent to an execution body that executes the mushroom bud collection task.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, the edible mushroom bud detection and collection method based on the RT-DETR algorithm described in claim 8 is implemented.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the edible mushroom bud detection and collection method based on the RT-DETR algorithm described in claim 8.

Images