Agaricus bisporus picking robot

Abstract

According to the agaricus bisporus picking robot, RGB-D images of agaricus bisporus are collected through a depth camera, the RGB-D images are sent to an AI development board, the AI development board recognizes and positions the agaricus bisporus according to the RGB-D images through an improved Mask R-CNN model algorithm, the current pose of an SCARA mechanical arm and the picking position information of the agaricus bisporus are obtained, and the agaricus bisporus is picked according to the current pose of the SCARA mechanical arm and the picking position information of the agaricus bisporus. According to the method, the AI development board receives the agaricus bisporus, generates a corresponding picking track command and sends the picking track command to the AI development board, the AI development board controls the SCARA mechanical arm to move to the target position according to the picking track command and controls the end effector to pick and grab the agaricus bisporus corresponding to the target position, accurate identification and positioning of the agaricus bisporus can be achieved, the accuracy of rapidly grabbing the agaricus bisporus can be improved by adopting the SCARA mechanical arm, and the working efficiency of the agaricus bisporus is improved. And the agaricus bisporus picking efficiency is improved.

Description

Technical Field

[0001] This invention belongs to the field of button mushroom harvesting technology, and in particular relates to a button mushroom harvesting robot. Background Technology

[0002] The industrialized production of button mushrooms mainly includes the preparation of substrate, pre-wet mixing, primary fermentation, secondary fermentation, preparation of tertiary substrate, casing, bud induction, and mushroom cultivation. Except for harvesting, all stages are mechanized and automated. However, button mushroom harvesting still relies on manual labor, with labor costs accounting for nearly half of the total labor costs in the production process. Furthermore, there are problems such as an aging workforce, labor shortages during the harvest season, and the health risks of high humidity in mushroom houses. Therefore, manual harvesting has become a significant bottleneck restricting the development of the button mushroom industry, and the development of button mushroom harvesting robots has become an urgent need for this industry's development. Summary of the Invention

[0003] In view of this, the present invention provides a button mushroom harvesting robot, which can quickly and accurately identify and locate button mushrooms, thereby improving the efficiency of button mushroom harvesting. The specific technical solution adopted is as follows.

[0004] This invention provides a button mushroom harvesting robot, which includes a SCARA robotic arm, a depth camera, an AI development board, and an end effector. The SCARA robotic arm, the depth camera, and the end effector are all connected to the AI ​​development board. The SCARA robotic arm is connected to the end effector. The execution process of the harvesting robot includes: The harvesting robot enters working mode, acquires RGB-D images of button mushrooms captured by the depth camera, and sends the RGB-D images to the AI ​​development board. The RGB-D images include RGB images and depth images. The AI ​​development board uses an improved Mask R-CNN model algorithm to identify and locate the button mushroom based on the RGB-D image; The current pose of the SCARA robotic arm and the picking position information of the button mushroom are obtained, and a corresponding picking trajectory command is generated and sent to the AI ​​development board. The picking trajectory command includes the picking trajectory and picking point information. The AI ​​development board controls the SCARA robotic arm to move to the target position corresponding to the picking trajectory command according to the picking trajectory command, and controls the end effector to pick and grab the button mushrooms corresponding to the target position.

[0005] As a preferred embodiment of the above technical solution, the harvesting robot enters a working state, acquires RGB-D images of the button mushrooms captured by the depth camera, and sends the RGB-D images to the AI ​​development board, including: The RGB image and depth image captured by the depth camera are obtained, and the growth area of ​​Agaricus bisporus is obtained by using an instance segmentation network on the RGB image; The picking point of the button mushroom is identified based on Hough circle detection, and the growth posture of the button mushroom is estimated by combining it with the aligned depth map to obtain the picking location information of the button mushroom. Specifically, the picking point of the button mushroom is obtained from the RGB image, and the growth tilt angle and deviation angle are obtained by combining it with the depth map.

[0006] As a preferred embodiment of the above technical solution, before acquiring the RGB image and depth map captured by the depth camera, the following steps are included: The coordinate transformation is performed by acquiring the internal and external parameters of the depth camera. The internal parameters include the coordinate values ​​of the intersection of the camera optical axis and the imaging plane, the distortion coefficient of the camera lens, and the physical magnification from the imaging plane to the image coordinate system. The external parameters include the transformation relationship between the camera coordinate system and the world coordinate system. Constructing a pixel coordinate system Image coordinate system Camera coordinate system and world coordinate system The image coordinate system describes the position of each pixel in the image. Its origin is the top left corner of the image, the X-axis points horizontally to the right, and the Y-axis points vertically downwards. The position of each pixel is determined by its row and column number. The image coordinate system is expressed in millimeters, with its origin at the intersection of the optical axis and the image plane. The camera coordinate system is a three-dimensional Cartesian coordinate system with the optical center of the depth camera as its origin and the optical axis as its Z-axis. Its X and Y axes are parallel to the X and Y axes of the image coordinate system, respectively, and its Z-axis is perpendicular to the image plane and points in front of the camera. The world coordinate system describes the spatial position of the button mushroom in the real environment. It is transformed into the camera coordinate system using the external parameters of the depth camera. The corresponding mathematical expression is: (1) Where K is the obtained camera intrinsic parameter matrix, , For camera focal length, , Let R be the pixel coordinates of the projection center, and let R be the rotation matrix and T be the translation matrix, which are extrinsic parameters. This represents the Z-axis coordinate in the camera coordinate system. , and These are the coordinates in the world coordinate system.

[0007] As a preferred embodiment of the above technical solution, the Mask R-CNN model comprises three networks: a feature extraction network, an RPN, and a RoiHead. Its detection process is as follows: Features of button mushroom images are extracted using a feature extraction network, and then fused with feature maps of different scales using an FPN network to generate multi-scale information feature maps. Anchor boxes of various scales and shapes are set in the multi-scale information feature map, and preliminary target candidate regions are generated by combining them with the RPN network. The ROI Align algorithm is used to align the extracted target candidate regions with the feature map. The aligned candidate region feature map is then input into the RoiHead network, which includes a target detection branch and a mask prediction branch. The target detection branch is used to locate the bounding box of Agaricus bisporus, and the mask prediction branch is used for segmentation via FCN.

[0008] As a preferred embodiment of the above technical solution, the AI ​​development board uses the Mask R-CNN model algorithm to identify and locate the button mushroom based on the RGB-D image, including: The Mask R-CNN model is improved to be lightweight by replacing the backbone network in the Mask R-CNN model with MobileNetV3 and introducing a BiFPN feature fusion module. After feature extraction by the MobileNetV3 network, feature information at Layer 1, Layer 3, Layer 8, and Layer 11 is output respectively. The feature information at different layers is then integrated by the BiFPN feature fusion module and output to the RPN for candidate box extraction, so that the prediction results can be output through RoiHead. An improved Mask R-CNN model was used to identify and segment the growth regions of Agaricus bisporus to distinguish between independent and aggregated growth. The improved Mask R-CNN model was then used to identify Agaricus bisporus images.

[0009] As a preferred embodiment of the above technical solution, the model identifies and segments the growth regions of Agaricus bisporus to distinguish between independent growth and adhering growth, including: The original images of Agaricus bisporus collected are input into the Agaricus bisporus recognition model to obtain the detection results, and the category information is obtained as the mask image corresponding to the adhesion area; The mask image is ANDed with the original image of the button mushroom to obtain the button mushrooms growing in the adhesion region. The button mushrooms growing in the adhesion region are then subjected to median filtering to smooth the edge contour of the button mushrooms. The Canny operator was used to extract the contour information of the adhesion region of Agaricus bisporus, and the contour information of the adhesion region was subjected to Hough circle detection to separate the adhesion of Agaricus bisporus. Using the center of the circle as the harvesting point for button mushrooms, the sticky button mushrooms were separated to obtain the detection results.

[0010] As a preferred embodiment of the above technical solution, the picking point of the button mushroom is identified based on Hough circle detection, and the growth posture of the button mushroom is estimated in conjunction with the aligned depth map, including: Finding the picking point: Input the RGB image of the button mushroom into the button mushroom recognition model for prediction, extract the target button mushroom recognition mask, and use Hough circle detection to identify the center and radius of the button mushroom, where the center is the button mushroom picking point and the radius is the growth radius of the button mushroom; Sampling: Using the picking point as the center and the radius as the reference value, draw sampling circles of different radii at equal intervals. Align the RGB image with the depth image, obtain the depth information on the sampling circles respectively, and find the pixel with the smallest depth value in the sampling circle as the sampling depth value. Measurement of bias angle and tilt angle: When the sampling depth value is greater than the depth value at the center of the circle, it indicates that the button mushroom is growing without tilt. The angle between the line connecting the pixel corresponding to the sampling depth value and the center of the circle and the Y-axis of the image is the bias angle. The angle between the line connecting the depth value at the center of the circle and the sampling depth value and the XY plane of the image is the tilt angle.

[0011] As a preferred embodiment of the above technical solution, the end effector includes a backplate, a connector disposed on the backplate, and a reducer disposed at one end of the backplate via the connector. A gear is disposed on the backplate below the reducer, and a servo motor is disposed on the gear. A rack stop is disposed at the other end of the backplate, and an arc-shaped rack that meshes with the gear is disposed on the rack stop. A buffer suction cup bracket is disposed on the servo motor along the length direction of the backplate. A vacuum suction cup is disposed at the end of the buffer suction cup bracket away from the arc-shaped rack, and a PU air tube is disposed on the buffer suction cup bracket.

[0012] As a preferred embodiment of the above technical solution, the harvesting robot further includes an attitude adjustment mechanism connected to the end effector. The attitude adjustment mechanism converts the rotational motion output by the servo motor into a radial motion along an arc using gears and an arc-shaped rack. The corresponding attitude adjustment process includes: When the SCARA robotic arm drives the vacuum suction cup of the end effector to contact the surface of the button mushroom, the error in the contact distance between the vacuum suction cup and the surface of the button mushroom is compensated by the buffer suction cup bracket; The servo motor is used as the drive motor for the attitude adjustment mechanism, and the gear and rack are considered equivalent to a meshing transmission problem, where the driving gear is the gear mounted on the servo motor, and the driven gear is the rack; the servo motor's rotation range The rotation angle range of the rack is 0~180°. The temperature ranges from 0 to 70 degrees Celsius. The stepper motor is connected to the attitude adjustment mechanism via a reducer. The mechanism drives the stepper motor without microstepping and calculates the deflection angle in the horizontal direction. Adjust the resolution; A servo motor is used as the drive motor, and the tilt angle in the vertical direction is calculated. Adjust the resolution.

[0013] As a preferred embodiment of the above technical solution, the initialization process of the end effector includes: Upon power-on initialization, a communication task, a negative pressure sensor reading task, a negative pressure adjustment task, and a picking posture adjustment task are created. The communication task is used to receive control commands from the AI ​​development board and provide feedback on the picking and holding status. The negative pressure sensor reading task is used to read the sensor information of the picking robot in real time, perform filtering processing, and send the sensor information to the negative pressure adjustment task through a message queue. The negative pressure adjustment task is used to receive sensor information, detect the end effector's holding status in real time, dynamically adjust the negative pressure based on the holding status using a control algorithm, and feed back the holding status to the communication task; the picking posture adjustment task is used to receive the angle value from the communication task and control the stepper motor and servo motor to perform actions in real time.

[0014] This invention provides a button mushroom harvesting robot. It acquires RGB-D images of button mushrooms using a depth camera and sends these images to an AI development board. The AI ​​development board uses an improved Mask R-CNN model algorithm to identify and locate the button mushrooms based on the RGB-D images, obtaining the current pose of the SCARA robotic arm and the mushroom harvesting position information. It then generates a corresponding harvesting trajectory command and sends it to the AI ​​development board. The AI ​​development board controls the SCARA robotic arm to move to the target position corresponding to the harvesting trajectory command and controls the end effector to harvest and grab the button mushroom at the target position. The depth camera enables rapid and accurate identification and location of the button mushrooms, while the use of the SCARA robotic arm improves the accuracy of quickly grabbing the mushrooms, thus increasing the efficiency of button mushroom harvesting. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 The structural block diagram of the harvesting robot provided by the present invention; Figure 2A schematic diagram of the camera imaging principle provided by this invention; Figure 3 This is a schematic diagram of the basic Mask R-CNN model structure provided by the present invention; Figure 4 A schematic diagram of the improved Mask R-CNN model structure provided by this invention; Figure 5 This is a schematic diagram of the recognition results provided by the present invention; Figure 6 This is a schematic diagram of the Agaricus bisporus adhesion separation process provided by the present invention; Figure 7 This is a schematic diagram of the segmentation results of the adhesion region of Agaricus bisporus provided by the present invention; Figure 8 This is a schematic diagram of the growth posture of Agaricus bisporus provided by the present invention; Figure 9 This is a schematic diagram of the steps for measuring the deflection angle and tilt angle of Agaricus bisporus provided by the present invention; Figure 10 This is a schematic diagram of a button mushroom grown in a real environment, as provided by the present invention. Figure 11 A schematic diagram of the end effector provided by the present invention; Figure 12 A schematic diagram of the execution process of the end effector provided by the present invention; Figure 13 A schematic diagram of the working principle of the attitude adjustment mechanism provided by the present invention; Figure 14 The control system operation diagram of the end effector provided by the present invention.

[0017] The symbols for the main components are explained below: 1-Reducer; 2-Gear; 3-PU air hose; 4-Connector; 5-Backplate; 6-Servo motor; 7-Arc rack; 8-Rack stop; 9-Spring buffer suction cup bracket; 10-Vacuum suction cup; 20-SCARA robotic arm; 30-Depth camera; 40-AI development board; 50-End effector; 60-Attitude adjustment mechanism. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0019] It should be noted that when an element is said to be "fixed" to another element, it can be directly on the other element or there may be an intervening element. When an element is said to be "connected" to another element, it can be directly connected to the other element or there may be an intervening element. Conversely, when an element is said to be "directly" on another element, there is no intervening element. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0020] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0021] See Figure 1 This invention provides a button mushroom harvesting robot, which includes a SCARA robotic arm 20, a depth camera 30, an AI development board 40, and an end effector 50. The SCARA robotic arm 20, the depth camera 30, and the end effector 50 are all connected to the AI ​​development board 40. The SCARA robotic arm 20 is connected to the end effector 50. The execution process of the harvesting robot includes: The harvesting robot enters working mode, acquires RGB-D images of button mushrooms captured by the depth camera 30, and sends the RGB-D images to the AI ​​development board 40. The RGB-D images include RGB images and depth images. The AI ​​development board 40 uses an improved Mask R-CNN model algorithm to identify and locate the button mushroom based on the RGB-D image; The current pose of the SCARA robotic arm 20 and the picking position information of the button mushroom are obtained, and a corresponding picking trajectory command is generated and sent to the AI ​​development board 40. The picking trajectory command includes the picking trajectory and picking point information. The AI ​​development board 40 controls the SCARA robotic arm 20 to move to the target position corresponding to the picking trajectory command according to the picking trajectory command, and controls the end effector 50 to pick and grab the button mushrooms corresponding to the target position.

[0022] In this embodiment, the (button mushroom) harvesting robot mainly includes a SCARA robotic arm 20, a depth camera 30, an AI development board 40, an STM32 control board (not shown), an end effector 50, related components, and a power supply. These components are assembled. After grasping the mushroom, the SCARA robotic arm 20 places it in the harvesting basket and begins the next round of grasping. The mushroom harvesting robot's recognition and positioning system mainly consists of a hardware system and a software recognition algorithm. The hardware system mainly consists of the depth camera 30 and the AI ​​development board 40. The depth camera 30 uses a D435i camera, which provides visual information for target recognition and positioning tasks. The D435i camera has four circular holes on its front. From left to right, the first and third are infrared sensors, the second is an infrared laser emitter, and the fourth is an RGB camera. The harvesting robot enters working mode, acquires RGB-D images of button mushrooms captured by the depth camera 30, and sends the RGB-D images to the AI ​​development board 40. This includes: acquiring the RGB image and depth map captured by the depth camera 30; obtaining the growth area of ​​the button mushrooms through an instance segmentation network on the RGB image; identifying the harvesting point of the button mushrooms based on Hough circle detection; and estimating the growth posture of the button mushrooms in conjunction with the aligned depth map to obtain the harvesting location information of the button mushrooms. Specifically, the harvesting point of the button mushrooms is obtained from the RGB image, and the growth tilt angle and deviation angle are obtained in conjunction with the depth map.

[0023] It should be noted that, for reference Figure 2 Before acquiring the RGB image and depth map captured by the depth camera 30, the process includes: acquiring the internal and external parameters of the depth camera 30 and performing coordinate transformation, wherein the internal parameters include the coordinate values ​​of the intersection point of the camera optical axis and the imaging plane, the distortion coefficient of the camera lens, and the physical magnification factor from the imaging plane to the image coordinate system, and the external parameters include the transformation relationship between the camera coordinate system and the world coordinate system; and constructing a pixel coordinate system. Image coordinate system Camera coordinate system and world coordinate system The pixel coordinate system describes the position of each pixel in the image, with the top-left corner of the image as the origin, the horizontal axis to the right as the X-axis, and the vertical axis downwards as the Y-axis. The position of each pixel is determined by its row and column number. The image coordinate system is a coordinate system expressed in millimeters, with its origin located at the intersection of the optical axis and the image plane. The camera coordinate system is a three-dimensional rectangular coordinate system established with the optical center of the depth camera as the origin and the optical axis as the Z-axis. The X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system, respectively, and the Z-axis is perpendicular to the image plane and points in front of the camera. The world coordinate system describes the spatial position of the button mushroom in the real environment. The world coordinate system is transformed into the camera coordinate system through the external parameters of the depth camera 30. The corresponding mathematical expression is: (1) Where K is the obtained camera intrinsic parameter matrix, , For camera focal length, , Let R be the pixel coordinates of the projection center, and let R be the rotation matrix and T be the translation matrix, which are extrinsic parameters. This represents the Z-axis coordinate in the camera coordinate system. , and These are coordinates in the world coordinate system. The D435i depth camera is calibrated before leaving the factory, and its intrinsic and extrinsic parameters can be directly obtained via the RealSense SDK. It should be understood that the diverse growth postures and severe adhesion of button mushrooms increase the difficulty of spatial localization. To ensure accurate detection and localization of button mushrooms, RGB and depth maps were acquired using a depth camera 30. On the RGB map, the growth area of ​​button mushrooms was obtained through an instance segmentation network, distinguishing between two categories: independent growth and adhered growth. Adhesive-grown button mushrooms were separated, and the picking point was identified based on Hough circle detection. Combined with the aligned depth map, growth posture estimation was achieved, providing information on the position and posture of button mushrooms for picking. By acquiring RGB-D images of button mushrooms using a depth camera and sending these images to the AI ​​development board 40, the AI ​​development board 40 uses an improved Mask R-CNN model algorithm to identify and locate the button mushrooms based on the RGB-D images. It then obtains the current pose of the SCARA robotic arm 20 and the mushroom's harvesting position information, generates a corresponding harvesting trajectory command, and sends it to the AI ​​development board 40. The AI ​​development board 40 controls the SCARA robotic arm 20 to move to the target position corresponding to the harvesting trajectory command and controls the end effector 50 to harvest the button mushroom at the target position. This enables rapid and accurate identification and location of button mushrooms. Using a SCARA robotic arm improves the accuracy of quickly grasping button mushrooms and increases harvesting efficiency.

[0024] Optionally, the Mask R-CNN model comprises three networks: a feature extraction network, an RPN, and a RoiHead. Its detection process is as follows: Features of button mushroom images are extracted using a feature extraction network, and then fused with feature maps of different scales using an FPN network to generate multi-scale information feature maps. Anchor boxes of various scales and shapes are set in the multi-scale information feature map, and preliminary target candidate regions are generated by combining them with the RPN network. The ROI Align algorithm is used to align the extracted target candidate regions with the feature map. The aligned candidate region feature map is then input into the RoiHead network, which includes a target detection branch and a mask prediction branch. The target detection branch is used to locate the bounding box of Agaricus bisporus, and the mask prediction branch is used for segmentation via FCN.

[0025] In this embodiment, see Figure 3 and Figure 4 The AI ​​development board uses the Mask R-CNN model algorithm to identify and locate the button mushroom based on the RGB-D image. This includes: a lightweight improvement to the Mask R-CNN model, replacing the backbone network with MobileNetV3 and introducing a BiFPN feature fusion module; after feature extraction via the MobileNetV3 network, outputting feature information at Layer 1, Layer 3, Layer 8, and Layer 11 respectively, integrating the feature information from different layers via the BiFPN feature fusion module, and outputting it to the RPN for candidate box extraction, so that the prediction result is output through RoiHead; the improved Mask R-CNN model identifies and segments the growth region of the button mushroom to distinguish between independent and adhering growth, and uses the improved Mask R-CNN model to identify the button mushroom image. Button mushrooms grow densely and exhibit severe adhesion. Selecting a suitable visual recognition algorithm is a prerequisite for constructing a visual system. This invention uses an instance segmentation algorithm as the visual recognition algorithm. Compared with object detection algorithms and semantic segmentation algorithms, the instance segmentation algorithm simultaneously outputs the position and contour information of the button mushrooms, providing information support for the estimation of picking points and growth postures. This invention is based on an improved Mask R-CNN instance segmentation algorithm as the visual recognition model.

[0026] It should be noted that the Mask R-CNN model has high computational complexity, which limits its practicality in the context of button mushroom harvesting robots. Therefore, this invention first improves the Mask R-CNN model by replacing the backbone network with MobileNetV3, and then introduces a BiFPN feature fusion module to reduce the number of network parameters and improve the network's multi-scale feature fusion capability, thereby improving segmentation accuracy. This network structure consists of three parts: a feature extraction and fusion network, an RPN, and a RoiHead. After feature extraction by the MobileNetV3 network, feature information at Layer 1, Layer 3, Layer 8, and Layer 11 is output. Then, the BiFPN feature fusion module integrates the feature information from different layers and outputs it to the RPN for candidate box extraction. Finally, the RoiHead outputs the prediction result. The improved Mask R-CNN model's recognition result for button mushroom images is shown below. Figure 5As shown, the model can accurately identify and segment the growth regions of Agaricus bisporus, and can distinguish between independent growth and adhering growth. Furthermore, the improved Mask R-CNN model has significantly reduced parameters and computational cost, resulting in faster detection speeds and making it more suitable for deployment on embedded devices. Figure 5 Images (a0), (b0), and (c0) in the image are the original photos of the collected button mushrooms. Figure 5 (a1), (b1), and (c1) in the image represent the button mushroom image in the improved Mask R-CNN network.

[0027] Optionally, the model identifies segmented growth regions of Agaricus bisporus to distinguish between independent and aggregated growth, including: The original images of Agaricus bisporus collected are input into the Agaricus bisporus recognition model to obtain the detection results, and the category information is obtained as the mask image corresponding to the adhesion area; The mask image is ANDed with the original image of the button mushroom to obtain the button mushrooms growing in the adhesion region. The button mushrooms growing in the adhesion region are then subjected to median filtering to smooth the edge contour of the button mushrooms. The Canny operator was used to extract the contour information of the adhesion region of Agaricus bisporus, and the contour information of the adhesion region was subjected to Hough circle detection to separate the adhesion of Agaricus bisporus. Using the center of the circle as the harvesting point for button mushrooms, the sticky button mushrooms were separated to obtain the detection results.

[0028] In this embodiment, the picking point of the button mushroom is identified based on Hough circle detection, and the growth posture of the button mushroom is estimated using an aligned depth map, including: Finding the picking point: Input the RGB image of the button mushroom into the button mushroom recognition model for prediction, extract the target button mushroom recognition mask, and use Hough circle detection to identify the center and radius of the button mushroom, where the center is the button mushroom picking point and the radius is the growth radius of the button mushroom; Sampling: Using the picking point as the center and the radius as the reference value, draw sampling circles of different radii at equal intervals. Align the RGB image with the depth image, obtain the depth information on the sampling circles respectively, and find the pixel with the smallest depth value in the sampling circle as the sampling depth value. Measurement of bias angle and tilt angle: When the sampling depth value is greater than the depth value at the center of the circle, it indicates that the button mushroom is growing without tilt. The angle between the line connecting the pixel corresponding to the sampling depth value and the center of the circle and the Y-axis of the image is the bias angle. The angle between the line connecting the depth value at the center of the circle and the sampling depth value and the XY plane of the image is the tilt angle.

[0029] In this embodiment, the button mushroom recognition model can output button mushroom category labels and corresponding mask images. Further processing is needed to classify the button mushroom adhesion regions. This invention uses mask images and Hough circle detection to further segment the button mushroom adhesion regions. The process is as follows: Figure 6 As shown: First, the original image ( Figure 6 (a) Input the results into the button mushroom recognition model to obtain the detection results. Figure 6 (b)), obtain the category information as the mask image corresponding to the adhesion area ( Figure 6 (c) Perform an AND operation between the mask image and the original image to obtain the button mushrooms growing in the adhesion region. Figure 6 (d) Median filtering was applied to the Agaricus bisporus growing in the adhesion area to smooth the edge contour of the Agaricus bisporus. Figure 6 (e) The contour information of the adhesion region of Agaricus bisporus was extracted using the Canny operator. Figure 6 (f) The adhesion of button mushrooms was separated by Hough circle detection of the contour of the adhesion area, and the center of the circle was used as the picking point of the button mushroom. Figure 7 (g) The same method was used to treat the adherent growth of Agaricus bisporus, and the final detection area was as follows: Figure 8 As shown.

[0030] It should be noted that, for reference Figure 9 A four-degree-of-freedom SCARA robotic arm was used as the robotic arm for harvesting. This arm occupies only a flat vertical space during harvesting, offering advantages such as a large grasping area, compact and stable structure, and high operating speed. After identifying a single button mushroom target, further positioning is needed to obtain its coordinate position and growth posture in the camera coordinate system. The growth posture of the button mushroom is mainly determined by its tilt angle. and deflection angle Decision. RGB images do not contain depth information about the growth of button mushrooms, making it impossible to obtain their spatial location information. Therefore, this invention obtains the button mushroom harvesting point from the RGB image and then combines it with a depth image to obtain the growth tilt angle and deviation angle. The entire process is as follows: Figure 9 As shown in (a), the main steps are as follows: (1) Locating the picking point. First, input the RGB image of the button mushroom into the recognition model for prediction, extract the target button mushroom recognition mask, and identify the center and radius of the button mushroom by Hough circle detection, where the center is the picking point of the button mushroom and the radius is the growth radius of the button mushroom; (2) Sampling. Using the picking point as the center and the radius as the reference value, draw sampling circles of different radii at equal intervals. Align the RGB image with the depth image and obtain the depth information on the sampling circles respectively. Find the pixel with the smallest depth value in the sampling circle as the sampling depth value; (3) Measurement of skew angle and tilt angle. If the sampling depth values ​​are all greater than the depth value at the center, it indicates that the button mushroom is growing without tilt. The angle between the line connecting the pixel corresponding to the sampling depth value and the center of the circle and the Y-axis of the image is the skew angle. The angle between the line connecting the depth value at the center of the circle and the sampling depth value and the XY plane of the image is the tilt angle. The measurement results are as follows: Figure 1 As shown in (b).

[0031] Optionally, the end effector 50 includes a back plate 5, a connector 4 disposed on the back plate 5, and a reducer 1 disposed at one end of the back plate 5 via the connector 4. A gear 2 is disposed on the back plate 5 below the reducer 1, and a servo motor 6 is disposed on the gear 2. A rack stop 8 is disposed at the other end of the back plate 5, and an arc-shaped rack 7 that meshes with the gear 2 is disposed on the rack stop 8. A buffer suction cup bracket 9 is disposed along the length direction of the back plate 5. A vacuum suction cup 10 is disposed at the end of the buffer suction cup bracket 9 away from the arc-shaped rack 7, and a PU air pipe 3 is disposed on the buffer suction cup bracket 9.

[0032] In this embodiment, see Figure 13 and Figure 10 The harvesting robot also includes an attitude adjustment mechanism 60 connected to the end effector 50. The attitude adjustment mechanism 60 converts the rotational motion output by the servo motor 6 into a radial motion along an arc through the gear 2 and the arc rack 7. The corresponding attitude adjustment process includes: When the SCARA robotic arm 20 drives the vacuum suction cup 10 of the end effector 50 to contact the surface of the button mushroom, the buffer suction cup bracket 9 compensates for the error in the contact distance between the vacuum suction cup 10 and the surface of the button mushroom. The servo motor 6 is used as the drive motor of the attitude adjustment mechanism 60, and the rack of gear 2 is considered as an equivalent meshing transmission problem, wherein the driving gear is the gear mounted on the servo motor, and the driven gear is the rack; the angular range of the servo motor 6 The rotation angle range of the rack is 0~180°. The temperature ranges from 0 to 70 degrees Celsius. The stepper motor is connected to the attitude adjustment mechanism 60 via a reducer 1. The reducer drives the stepper motor without microstepping and calculates the deflection angle in the horizontal direction. Adjust the resolution; Servo motor 6 was used as the drive motor, and the tilt angle in the vertical direction was calculated. Adjust the resolution.

[0033] It should be noted that the performance of the end effector 50 directly affects the robot's harvesting efficiency and quality, and is a key component of the robot's harvesting process. This is because the growth posture of button mushrooms is quite complex, such as... Figure 11As shown. During harvesting, the end effector 50 can only contact specific harvesting areas to reduce damage to the button mushrooms and the impact on the growth of surrounding button mushrooms. The overall structure of the end effector 50 mainly consists of components such as gear 2, arc-shaped rack 7, servo motor 6, vacuum suction cup 10, and back plate 5. The end effector uses a silicone suction cup as the contact material, which can significantly reduce damage to the button mushrooms during harvesting. At the same time, in conjunction with the attitude adjustment mechanism 60, it can automatically adjust the harvesting attitude according to the growth status of the button mushrooms, reducing the impact on the growth of surrounding mushrooms. Its main structure is as follows. Figure 12 As shown in the figure. Among them, the end effector 50 was modeled and motion simulated, and the tilt angle adjustment range of ±35 degrees can be designed according to the growth angle of button mushrooms, which meets most of the harvesting requirements of button mushrooms.

[0034] Specifically, the harvesting process mainly follows a "suck-transfer-lift" method, first completing the "suck" action, such as... Figure 12 As shown in (a), based on the growth posture of the button mushroom, the servo motor and stepper motor are controlled to adjust the rotation angle in the horizontal and vertical directions respectively, thereby adjusting the harvesting posture of the end effector; after the harvesting posture is adjusted, the robotic arm is controlled to make the vacuum suction cup contact the surface of the button mushroom, as shown in (a). Figure 12 As shown in (b), contact pressure is formed on the surface of the button mushroom. During the adsorption process, the vacuum pump empties the air from the suction cup through the air tube, creating a pressure difference that allows the vacuum suction cup to firmly adhere to the surface of the button mushroom. After the adsorption action is completed, a "rotation" action is performed, as shown in (b). Figure 13 As shown in (c), the servo motor rotates to drive the gear-arc rack mechanism to align the suction cup to its initial position, while simultaneously controlling the stepper motor to rotate. During this process, the button mushroom is aligned by the suction cup and detaches from the substrate to complete the "rotation" action; finally, the vacuum suction cup lifts up to lift the button mushroom off the soil to complete the "lifting" action.

[0035] Optionally, the initialization process of the end effector includes: Upon power-on initialization, a communication task, a negative pressure sensor reading task, a negative pressure adjustment task, and a picking posture adjustment task are created. The communication task is used to receive control commands from the AI ​​development board and provide feedback on the picking and holding status. The negative pressure sensor reading task is used to read the sensor information of the picking robot in real time, perform filtering processing, and send the sensor information to the negative pressure adjustment task through a message queue. The negative pressure adjustment task is used to receive sensor information, detect the end effector's holding status in real time, dynamically adjust the negative pressure based on the holding status using a control algorithm, and feed back the holding status to the communication task; the picking posture adjustment task is used to receive the angle value from the communication task and control the stepper motor and servo motor to perform actions in real time.

[0036] In this embodiment, the attitude adjustment mechanism of the end effector converts the rotational motion output by the servo motor into a radial motion along an arc using a gear and rack structure. During this motion, only the attitude of the end effector suction cup changes without altering its spatial position. Figure 14 As shown. The main design steps are as follows: (1) Determine the pitch circle diameter and arc length of the rack. Due to positioning errors in the visual recognition system and the robotic arm, the contact distance between the suction cup and the mushroom surface cannot be precisely controlled during the process of the robotic arm driving the end effector suction cup to contact the mushroom surface. Therefore, a spring-buffered suction cup bracket was designed to compensate for the error in the contact distance between the suction cup and the mushroom surface through spring buffering. The design length of the spring-buffered suction cup bracket is 107.25 mm, therefore the pitch circle radius of the rack is determined to be 107.25 mm. Based on the growth angle range of the mushroom, the adjustment angle of the attitude adjustment mechanism is designed to be ±35°. The arc length corresponding to the pitch circle of the rack is calculated to be 112.31 mm using the arc length formula.

[0037] (2) Determine the parameters of the gear-rack transmission This invention uses a servo motor as the drive motor for the attitude adjustment mechanism, and the servo motor's rotation angle range... The rotation angle range of the rack is 0~180°. The angle is 0~70°. The gear-rack problem is equivalent to a meshing transmission problem, where the driving gear is the gear mounted on the servo motor and the driven gear is the rack. The parameters are shown in Table 1.

[0038] Table 1. Parameters of Gears and Curved Racks

[0039] The actual range of motion is the angular range of the servo motor. =0~180° corresponds to the rack's rotation angle range =0~68.2°, with an error of 2.57%, which meets the design requirements.

[0040] (3) Calculate the angular resolution Calculate the deflection angle in the horizontal direction. Adjusting the resolution. The stepper motor is connected to the attitude adjustment mechanism via a reducer. Without microstepping, the stepper motor is driven to move, with a step angle of 1.8°. After passing through a 50:1 reducer, the horizontal angle... The resolution is 0.036°; Calculate the tilt angle in the vertical direction. Resolution adjustment. A servo motor is used as the drive motor, with an adjustment angle resolution of 0.9°. After passing through a rack and pinion mechanism, the vertical tilt angle is adjusted. The resolution is 0.341°.

[0041] Specifically, a servo motor is used as the driving component of the attitude adjustment structure. It has a rated voltage of 7.4 V, a rated torque of 2.5 NM, a control angle range of 0~180°, a theoretical minimum control angle resolution of 0.9°, a PWM signal control pulse width of 500~2500 μs, and a pulse dead zone of 5 μs. The servo motor driver board converts the microcontroller control commands into servo motor drive PWM signals. This invention uses the PCA9685 as the chip for generating the servo motor drive signals, with a rated voltage of 7.4 V, a timer resolution of 12 (0~4096) bits, and an IIC drive protocol. A micro vacuum pump combined with a silicone suction cup forms a negative pressure adsorption system to provide suction force. This micro vacuum pump has a rated voltage of 12 V, a theoretical vacuum of -80 kPa, a power of 10 W, and a PWM frequency of 10 kHz. Since the negative pressure needs to be dynamically adjusted, this invention uses a negative pressure sensor to detect the pressure in the pneumatic circuit in real time. The main parameters are a rated voltage of 3.3 V, an accuracy of ±0.1% of span, and an output signal of 0.2~2.7 V.

[0042] Specifically, an Arduino nano is used as the main controller of the end effector, primarily responsible for receiving angle control commands from the AI ​​development board, real-time detection of air pressure within the pneumatic circuit, motor signal output, dynamic adjustment of negative pressure, and detection of harvesting status. A 42-stepper motor with a closed-loop stepper motor driver is used to adjust the horizontal angle of the end effector. To verify the effect of silicone suction cups of different diameters on the holding effect of button mushrooms, this invention tested the holding effects of silicone suction cups with diameters of 13 mm, 20 mm, 25 mm, and 32 mm, ultimately selecting a 25 mm silicone suction cup as the end effector suction cup. The mechanical structure of the end effector needs to work in conjunction with the control system to complete the holding function. A control system is designed based on the harvesting logic of the end effector, mainly including the design of a harvesting posture adjustment module, a communication module, and a suction force adjustment module, realizing functions such as end effector communication control, adjustment of harvesting posture, dynamic adjustment of negative pressure, and detection of harvesting status.

[0043] To ensure stable and efficient operation of the control system, the end effector uses FreeRTOS as its real-time operating system. FreeRTOS reduces the coupling between different functions of the end effector and improves real-time performance. Upon power-up, the system first initializes. After initialization, it creates communication tasks, negative pressure sensor reading tasks, negative pressure adjustment tasks, and picking posture adjustment tasks. The communication task receives control commands from the AI ​​development board and provides feedback on the picking and holding status. The negative pressure sensor reading task reads sensor information in real time, performs filtering, and sends the sensor information to the negative pressure adjustment task via a message queue. The negative pressure adjustment task primarily receives sensor messages, monitors the end effector's holding status in real time, dynamically adjusts the negative pressure based on the holding status using a control algorithm, and simultaneously feeds back the holding status to the communication task. The picking posture adjustment task receives angle values ​​from the communication task and controls the stepper motor and servo motor to perform actions in real time. Its overall control block diagram is shown below. ​ As shown.

[0044] In all examples shown and described herein, any specific values ​​should be interpreted as merely exemplary and not as limitations; therefore, other examples of exemplary embodiments may have different values.

[0045] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0046] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A double-spore mushroom picking robot, characterized by, The picking robot comprises a SCARA mechanical arm, a depth camera, an AI development board and an end effector, the SCARA mechanical arm, the depth camera and the end effector are connected with the AI development board, the SCARA mechanical arm is connected with the end effector, and the execution process of the picking robot comprises: The picking robot enters a working state, acquires an RGB-D image of double-spore mushrooms collected by the depth camera, and sends the RGB-D image to the AI development board, wherein the RGB-D image comprises an RGB image and a depth image; The AI development board adopts an improved Mask R-CNN model algorithm to identify and locate the double-spore mushrooms according to the RGB-D image; The current pose of the SCARA mechanical arm and the picking position information of the double-spore mushrooms are acquired, and corresponding picking trajectory commands are generated and sent to the AI development board, wherein the picking trajectory commands comprise picking trajectory and picking point information; The AI development board controls the SCARA mechanical arm to move to the target position corresponding to the picking trajectory command according to the picking trajectory command, and controls the end effector to pick and grab the double-spore mushrooms corresponding to the target position.

2. The Agaricus bisporus picking robot according to claim 1, characterized in that, The picking robot enters a working state, acquires an RGB-D image of double-spore mushrooms collected by the depth camera, and sends the RGB-D image to the AI development board, comprising: Acquire the RGB image and the depth image collected by the depth camera, and acquire the growth area of the double-spore mushrooms on the RGB image through an instance segmentation network; Identify the picking point of the double-spore mushrooms based on Hough circle detection, and cooperate with the aligned depth image to estimate the growth posture of the double-spore mushrooms to obtain the picking position information of the double-spore mushrooms, wherein the picking point of the double-spore mushrooms is acquired on the RGB image and the growth inclination angle and the deviation angle are acquired on the depth image.

3. The Agaricus bisporus picking robot according to claim 2, characterized in that, Before acquiring the RGB image and the depth image collected by the depth camera, comprising: Acquire the internal parameters and the external parameters of the depth camera for coordinate conversion, wherein the internal parameters comprise the coordinate value of the intersection of the camera optical axis and the imaging plane, the distortion coefficient of the camera lens and the physical magnification of the imaging plane to the image coordinate system, and the external parameters comprise the conversion relationship between the camera coordinate system and the world coordinate system; pixel coordinate system , image coordinate system , camera coordinate system and world coordinate system , wherein the pixel coordinate system is used to describe the position of each pixel in the image, the upper left corner of the image is the origin, the horizontal right is the X axis, and the vertical down is the Y axis, and the position of each pixel is determined by its row number and column number; the image coordinate system is a coordinate system expressed in millimeter units, and the origin is located at the intersection of the optical axis and the image plane; the camera coordinate system is a three-dimensional rectangular coordinate system with the optical center of the depth camera as the origin and the optical axis as the Z axis, the X axis and the Y axis are parallel to the X axis and the Y axis of the image coordinate system respectively, and the Z axis is perpendicular to the image plane and points to the front of the camera; the world coordinate system is used to describe the spatial position of the double-spore mushroom in the real environment, and the world coordinate system is converted into the camera coordinate system through the external parameters of the depth camera, and the corresponding mathematical expression is: （1） wherein K is an obtained camera intrinsic matrix, 、 is a camera focal length, 、 is a pixel coordinate of a projection center, a rotation matrix R and a translation matrix T are an extrinsic matrix, denotes a Z-axis coordinate in a camera coordinate system, 、 and is a coordinate in a world coordinate system.

4. The Agaricus bisporus picking robot according to claim 3, characterized in that, The Mask R-CNN model comprises a feature extraction network, an RPN and a RoiHead three-part network, and the detection process is as follows: Extract the double-spore mushroom image features through the feature extraction network, and fuse different scale feature maps through the FPN network to generate multi-scale information feature maps; Set various scale ratios and morphological rectangular Anchor boxes in the multi-scale information feature maps, and generate preliminary target candidate regions in combination with the RPN network; Align the extracted target candidate regions with the feature maps by using the ROI Align algorithm, and input the aligned candidate region feature maps into the RoiHead network, wherein the RoiHead comprises a target detection branch and a mask prediction branch, the target detection branch is used for locating the double-spore mushroom boundary box position, and the mask prediction branch is used for segmentation through the FCN.

5. The Agaricus bisporus picking robot according to claim 4, characterized in that, The AI development board adopts a Mask R-CNN model algorithm to recognize and locate the agaricus bisporus according to the RGB-D image, and the method comprises the following steps of: Lightweight improvement is made to the Mask R-CNN model, MobileNetV3 is used to replace the backbone network in the Mask R-CNN model, and a BiFPN feature fusion module is introduced; After feature extraction by the MobileNetV3 network, the feature information of Layer1, Layer3, Layer8 and Layer11 is output respectively, the feature information of different layers is integrated by the BiFPN feature fusion module, and then output to the RPN for candidate box extraction, so as to output the prediction result through the RoiHead; The improved Mask R-CNN model recognizes and segments the growth area of the agaricus bisporus to distinguish between independent growth and adhesion growth, and the improved Mask R-CNN model is used to recognize the agaricus bisporus image.

6. The Agaricus bisporus picking robot according to claim 5, characterized in that, The model recognizes and segments the growth area of the agaricus bisporus to distinguish between independent growth and adhesion growth, and the method comprises the following steps: The collected agaricus bisporus original image is input into the agaricus bisporus recognition model to obtain a detection result, and a mask image corresponding to the adhesion area is obtained as class information; The mask image is operated with the agaricus bisporus original image to obtain the agaricus bisporus grown in the adhesion area, and the agaricus bisporus grown in the adhesion area is subjected to median filter processing to smooth the edge profile of the agaricus bisporus; The Canny operator is used to extract the contour information of the agaricus bisporus adhesion area, and the contour information of the adhesion area is subjected to Hough circle detection to separate the agaricus bisporus; The center of the circle is used as the picking point of the agaricus bisporus, and the adhesion segmentation processing is performed on the agaricus bisporus grown in the adhesion area to obtain the detection result of the agaricus bisporus.

7. The Agaricus bisporus picking robot according to claim 6, characterized in that, The picking point of the agaricus bisporus is recognized based on the Hough circle detection, and the growth posture of the agaricus bisporus is estimated by aligning the depth map, and the method comprises the following steps: Finding the picking point: input the RGB image of the agaricus bisporus into the agaricus bisporus recognition model for prediction, extract the target agaricus bisporus recognition mask image, and use the Hough circle detection to recognize the center and radius of the agaricus bisporus, wherein the center is the picking point of the agaricus bisporus, and the radius is the growth radius of the agaricus bisporus; Sampling: taking the picking point as the center and the radius as the reference value, different radius sampling circles are drawn in turn at equal intervals, the RGB image and the depth map are aligned, the depth information on the sampling circle is obtained respectively, and the pixel point with the minimum depth value in the sampling circle is taken as the sampling depth value; Measurement of deflection angle and inclination angle: when the sampling depth values are all greater than the depth value at the center, the deflection angle is the angle between the connecting line of the pixel point corresponding to the sampling depth value and the center and the Y-axis direction of the image, and the inclination angle is the angle between the connecting line of the depth value at the center and the sampling depth value and the XY plane of the image.

8. The Agaricus bisporus picking robot of claim 1, wherein, The end effector comprises a back plate, a connecting piece arranged on the back plate, and a reducer arranged at one end of the back plate through the connecting piece, wherein a gear is arranged below the reducer on the back plate, a steering wheel is arranged on the gear, a rack block is arranged at the other end of the back plate, an arc-shaped rack is arranged on the rack block and engaged with the gear, a buffer suction disc support is arranged on the steering wheel along the length direction of the back plate, a vacuum suction disc is arranged at the end of the buffer suction disc support away from the arc-shaped rack, and a PU air pipe is arranged on the buffer suction disc support.

9. The Agaricus bisporus picking robot according to claim 8, characterized in that, The picking robot further comprises a posture adjusting mechanism connected with the end effector, which converts the rotary motion output by the steering wheel into the motion along the arc around the center through the gear and the arc-shaped rack, and the corresponding posture adjusting process comprises: When the SCARA mechanical arm drives the vacuum suction disc of the end effector to contact the surface of the double-spore mushroom, the error of the contact distance between the vacuum suction disc and the surface of the double-spore mushroom is compensated by the buffer suction disc support; The servo motor is used as the drive motor for the attitude adjustment mechanism, and the gear and rack are considered equivalent to a meshing transmission problem, where the driving gear is the gear mounted on the servo motor, and the driven gear is the rack; the servo motor's rotation range The rotation angle range of the rack is 0~180°. The temperature ranges from 0 to 70 degrees Celsius. The stepping motor is connected with the posture adjusting mechanism through a speed reducer, and the stepping motor is driven to act under the condition of no subdivision, and the deflection angle in the horizontal direction is calculated Adjustment resolution; Using a rudder as a drive motor and calculating the tilt angle in the vertical direction Adjusting resolution.

10. The Agaricus bisporus picking robot according to claim 9, characterized in that, The initialization process of the end effector comprises: Power-on initialization, creating a communication task, a negative pressure sensor reading task, a negative pressure adjusting task and a picking posture adjusting task, wherein the communication task is used to receive the control instructions from the AI development board and feedback the picking suction state; the negative pressure sensor reading task is used to read the sensor information of the picking robot in real time and perform filtering processing, and the sensor information is sent to the negative pressure adjusting task through the message queue; The negative pressure adjusting task is used to receive the sensor information, detect the suction state of the end effector in real time, dynamically adjust the negative pressure size according to the suction state by using the control algorithm, and feedback the suction state to the communication task; the picking posture adjusting task is used to receive the angle value of the communication task, and control the stepping motor and the steering wheel to act in real time.

Images