Factory shiitake mushroom picking robot and vision-based graded picking method

Abstract

A factory shiitake mushroom picking robot, comprising a moving platform, a base (100), a lifting / lowering device (200), a mushroom stick gripping device (300) and a shiitake mushroom picking device (400), wherein the base (100) is connected to the moving platform, the lifting / lowering device (200) is connected to the base, the mushroom stick gripping device (300) is connected to the lifting / lowering device (200), and the shiitake mushroom picking device (400) is connected to the base (100). The mushroom stick gripping device (300) is used for removing mushroom sticks from a mushroom rack (4). Mushroom stick gripping across different layers is achieved by means of the lifting / lowering device (200). The shiitake mushroom picking device (400) comprises a robotic arm (402) and an end effector (403). Image data of the mushroom sticks is collected by three depth cameras, shiitake mushrooms to be picked are identified by means of multi-view target matching, and quality grade classification is carried out, to achieve graded picking. Also disclosed are a vision-based graded picking method, a shiitake mushroom detection and grading method based on multi-view fusion, and a storage medium for executing the steps of the corresponding methods. The technical problems of existing automated shiitake mushroom picking robots, including low picking efficiency, low degree of automation, susceptibility to damaging mushroom caps and mushroom sticks, and the inability to accurately identify mature shiitake mushrooms are solved.

Description

Factory mushroom picking robot and vision-based grading picking method Technical Field

[0001] The present invention relates to the technical field of factory-based shiitake mushroom cultivation, and in particular to a factory-based shiitake mushroom picking robot and a vision-based graded picking method. Background Art

[0002] As we all know, shiitake mushroom is an edible fungus that is rich in nutrients, including high protein, low fat, polysaccharides, multiple amino acids and multiple vitamins.

[0003] Shiitake mushroom cultivation has become factory-based. The factory environment meets the ideal growth conditions for shiitake mushrooms, including temperature, humidity, and light. Numerous spawn racks are installed within the factory. Mushroom sticks are placed on these racks and used to grow shiitake mushrooms. These racks have many layers, each with limited space, and multiple spawn sticks can be placed on each layer. The basic structure of these racks can be found in the utility model patent, "A Shiitake Mushroom Stick Cultivation Rack," with authorization publication number CN215454376U.

[0004] At present, mature shiitake mushrooms on mushroom sticks are mostly picked manually. Manual picking has the problems of high labor intensity, high labor cost, low efficiency, and missed picking.

[0005] There are a few automated picking robots on the market that replace manual harvesting. For reference, see patent applications CN116458388A and CN116671392A. However, automated harvesting is not ideal, characterized by low efficiency, a low degree of automation, damage to the mushroom caps, damage to the mushroom sticks, and inability to accurately identify mature mushrooms. In patent applications CN116458388A and CN116671392A, flexible fingers are used to grip the mushrooms and remove them from the sticks, which can easily damage the caps.

[0006] Summary of the Invention

[0007] The present invention is intended to solve the technical problems of existing automatic picking robots for picking shiitake mushrooms, such as low picking efficiency, low degree of automation, easy damage to the mushroom caps, easy damage to the mushroom sticks, and inability to accurately identify mature shiitake mushrooms. It provides a factory shiitake mushroom picking robot and a vision-based grading picking method.

[0008] The factory mushroom picking robot disclosed in the present invention takes out mushroom sticks from a mushroom stick rack, takes mushroom sticks between different layers through a lifting device, picks the mushrooms that can be picked by an execution terminal, and puts the mushroom sticks back on the mushroom stick rack after picking is completed.

[0009] The present invention provides a factory mushroom picking robot, comprising a mobile platform, a chassis, a lifting device, a mushroom stick picking device and a mushroom picking device, wherein the chassis is connected to the mobile platform, the lifting device is connected to the chassis, the mushroom stick picking device is connected to the lifting device, and the mushroom picking device is connected to the chassis;

[0010] The lifting device includes a first fixed plate, a second fixed plate, a first electric cylinder, a second electric cylinder and a support plate, the first electric cylinder is fixedly connected to the first fixed plate, the second electric cylinder is fixedly connected to the second fixed plate, the first electric cylinder and the second electric cylinder are arranged vertically side by side, the first electric cylinder is provided with a first slider, the second electric cylinder is provided with a second slider, one end of the support plate is fixedly connected to the first slider on the first electric cylinder, and the other end of the support plate is fixedly connected to the second slider on the second electric cylinder; the first fixed plate is fixedly connected to the chassis, and the second fixed plate is fixedly connected to the chassis;

[0011] The mushroom stick picking device includes a first base, a first guide rail assembly, a second guide rail assembly, a Y-axis direction screw rod, a nut, a first drive motor, a first synchronous pulley, a second synchronous pulley, a first synchronous belt, a slide plate, a second base, a third guide rail assembly, an X-axis direction bidirectional screw rod, a slider 1, a slider 2, an L-shaped connecting plate 1, an L-shaped connecting plate 2, a clamping rod 1, a clamping rod 2, a nut seat 1, a nut seat 2, a second drive motor, a third synchronous pulley, a fourth synchronous pulley and a second synchronous belt. The first guide rail assembly and the second guide rail assembly are respectively fixedly connected to the first base The first guide rail assembly and the second guide rail assembly are arranged side by side, the front end of the Y-axis direction screw rod is rotatably connected to the front of the first base through a bearing, the rear end of the Y-axis direction screw rod is rotatably connected to the rear end of the first base through a bearing, the Y-axis direction screw rod is located between the first guide rail assembly and the second guide rail assembly, the first drive motor is connected to the rear end of the first base, the first synchronous pulley is connected to the output shaft of the first drive motor, the second synchronous pulley is connected to the rear end of the Y-axis direction screw rod, the first synchronous belt is connected between the first synchronous pulley and the second synchronous pulley, and the nut is connected to the Y-axis direction screw rod. The first guide rail assembly is connected to the second guide rail assembly, the second guide rail assembly is fixedly connected to the slider, the second guide rail assembly is fixedly connected to the slider, the second guide rail assembly is fixedly connected to the slider, the second guide rail assembly is fixedly connected to the slider, the left end of the bidirectional screw rod in the X-axis direction is rotatably connected to the left part of the second base through a bearing, and the right end of the bidirectional screw rod in the X-axis direction is rotatably connected to the right part of the second base through a bearing. Nut seat 1 and nut seat 2 are respectively connected to the bidirectional screw rod in the X-axis direction, slider 1 is fixedly connected to nut seat 1, and slider Block 2 is fixedly connected to nut seat 2, L-shaped connecting plate 1 is fixedly connected to slider 1, L-shaped connecting plate 2 is fixedly connected to slider 2, the second drive motor is fixedly connected to the left part of the second base, the third synchronous pulley is connected to the output shaft of the second drive motor, the fourth synchronous pulley is connected to the left end of the bidirectional screw rod in the X-axis direction, the second synchronous belt is connected between the third synchronous pulley and the fourth synchronous pulley, the rear end of the clamping rod 1 is fixedly connected to the L-shaped connecting plate 1, and the rear end of the clamping rod 2 is fixedly connected to the L-shaped connecting plate 2; the first base is fixedly connected to the support plate of the lifting device;

[0012] The mushroom picking device includes a lifting mechanism, a mechanical arm and an execution end, the lifting mechanism is fixedly connected to the chassis, the mechanical arm is connected to the lifting mechanism, and the execution end is connected to the free end of the mechanical arm; the execution end includes a support frame, a clamping drive motor, a screw rod, a screw rod nut, a connecting block, a first connecting rod, a first V-shaped connecting rod, a second connecting rod, a second V-shaped connecting rod, a first clamping block and a second clamping block, the clamping drive motor is fixedly connected to the support frame, the screw rod is fixedly connected to the output shaft of the clamping drive motor, the screw rod nut is connected to the screw rod, the connecting block is fixedly connected to the screw rod nut, the rear end of the first connecting rod is connected to the connecting The block is rotatably connected, the front end of the first connecting rod is rotatably connected to the rear end of the first V-shaped connecting rod, the middle part of the first V-shaped connecting rod is rotatably connected to the support frame, the rear end of the second connecting rod is rotatably connected to the connecting block, the front end of the second connecting rod is rotatably connected to the rear end of the second V-shaped connecting rod, the middle part of the second V-shaped connecting rod is rotatably connected to the support frame, the first connecting rod and the second connecting rod are symmetrically arranged, the first V-shaped connecting rod and the second V-shaped connecting rod are symmetrically arranged, the first clamping block is connected to the front end of the first V-shaped connecting rod, and the second clamping block is connected to the front end of the second V-shaped connecting rod; the first clamping block is provided with an arc-shaped groove, and the second clamping block is provided with an arc-shaped groove;

[0013] The support frame at the execution end is fixedly connected to the free end of the robotic arm;

[0014] The lifting mechanism is located between the first electric cylinder and the second electric cylinder of the lifting device.

[0015] Preferably, in the mushroom stick picking device, an inclined portion is provided on the inner side of the first clamping rod and an inclined portion is provided on the inner side of the second clamping rod.

[0016] Preferably, the picking device further comprises a mushroom recognition depth camera, which is connected to the support frame of the execution end.

[0017] The present invention also provides a vision-based grading and picking method using a factory mushroom picking robot. The picking device also includes a mushroom recognition depth camera, which is connected to a support frame of an execution end. The support frame is connected to a first depth camera via a first adjustable bracket, and the support frame is connected to a second depth camera via a second adjustable bracket. The vision-based grading and picking method includes the following steps:

[0018] The first step is to build a multi-view data set for training the mushroom target detection model;

[0019] Step (1) uses a depth camera to collect video data of the mushroom stick; the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed. When the depth camera moves around to the lower left side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the first video data collection of a mushroom stick is completed; then the second video is recorded, the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed. When the camera moves around to the lower right side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the second video data collection of a mushroom stick is completed; for multiple mushroom sticks, multiple videos are collected;

[0020] Step (2) is to obtain a static image of the shiitake mushrooms; extract frames from each mushroom stick video stream to obtain extracted frame images, and filter out a large number of shiitake mushroom target detection data sets from the extracted frame images, wherein the shiitake mushroom target detection data sets include images of the mushroom cap from a top-down perspective and images of the mushroom gills from a top-down perspective;

[0021] Step (3), the dataset is labeled according to six quality categories: white, normal, deformed, unopened, slightly opened, and fully opened;

[0022] The second step is to train a mushroom target detection model based on the YOLOv8 deep learning algorithm;

[0023] Step (1), model training;

[0024] Based on the basic architecture of the YOLOv8 object detection model, the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer were pruned, leaving only the detection head output of the P4 layer. This formed the mushroom object detection model, and the mushroom object detection dataset constructed in the first step was used to train the mushroom object detection model.

[0025] Step (2), matching of multi-view targets; the mushroom recognition depth camera is used to shoot the cap from a top-down perspective, and the first depth camera and the second depth camera are used to shoot the gills from a top-down perspective. The three images taken by the mushroom recognition depth camera, the first depth camera and the second depth camera are combined into a batch. The batch is input into the mushroom target detection model to obtain the mushroom detection results under three perspectives: top-down perspective, right-side upward perspective and left-side upward perspective: det_d, det_ur, det_ul; the rule-based method is used to achieve the matching of the cap and the gills. First, the midline of the width direction of the picture is used as the dividing line, and the midline of det_d is used as the dividing line. The detection frame is divided into two parts, det_dr and det_dl. Let the detection frame in the upper part det_dr correspond to the detection result det_ur of the upward gill image on the right side of the mushroom stick, and let the detection frame in the lower part det_dl correspond to the detection result det_ul of the upward gill image on the left side of the mushroom stick; determine the list of targets to be picked in the top-down cap image. The following four situations are targets to be picked: ① The cap category is white and the gill category is not opened, ② The cap category is white and the gill category is slightly opened, ③ The cap category is normal and the gill category is not opened, ④ The cap category is normal and the gill category is slightly opened;

[0026] The third step is to deploy the mushroom object detection model in the controller;

[0027] In the fourth step, the controller controls the robot arm and performs the end-action motion to perform the picking task;

[0028] Step (1): The mushroom stick picking device moves the mushroom stick to a position close to the execution end of the mushroom picking device. The controller controls the movement of the robotic arm to make the mushroom recognition depth camera reach the photo taking position above the mushroom stick. The mushroom recognition depth camera, the first depth camera, and the second depth camera collect RGB images and depth images of the mushroom stick and send them to the controller.

[0029] Step (2): the RGB image is input into the mushroom target detection model, and the target to be picked is identified through multi-view target matching;

[0030] In step (3), the controller obtains the three-dimensional coordinates of the center point of the cap of the target to be picked, and then converts the three-dimensional coordinates of the center point of the cap into position information in the base coordinate system of the robotic arm through coordinate conversion. The robotic arm action guides the execution end to move to the target to be picked, and the execution end performs a picking action to pick the target mushrooms on the mushroom stick.

[0031] Preferably, in the process of multi-view target matching, for each cap target det_up in the upper part det_dr, a list of detection frames det_ups in a range slightly larger than the width of the cap target det_up in the x-axis direction is collected in det_ur, and then the detection frame with the smallest y-axis coordinate value is selected from det_ups, which is the gill target corresponding to the cap target det_up; for each cap target det_down in the lower part det_dl, a list of detection frames det_downs in a range slightly larger than the width of the cap target det_down in the x-axis direction is collected in det_ul, and then the detection frame with the smallest y-axis coordinate value is selected from det_downs, which is the gill target corresponding to the cap target det_down.

[0032] Preferably, in step (2) of the first step, the extracted frame images are preliminarily screened by an image quality assessment method, and then images containing incomplete mushroom sticks are manually checked and removed, and then image frames extracted within a few seconds at the beginning and end of the original video are selected to finally obtain a shiitake mushroom target detection dataset.

[0033] The present invention also provides a method for detecting and grading shiitake mushrooms based on multi-view fusion, comprising the following steps:

[0034] The first step is to build a multi-view data set for training the mushroom target detection model;

[0035] Step (1) uses a depth camera to collect video data of the mushroom stick; the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed, and when the depth camera moves around to the lower left side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the first video data collection of a mushroom stick is completed; then the second video is recorded, the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed, and when the camera moves around to the lower right side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the second video data collection of a mushroom stick is completed; for multiple mushroom sticks, multiple videos are collected;

[0036] Step (2) is to obtain a static image of the shiitake mushrooms; extract frames from each mushroom stick video stream to obtain extracted frame images, and filter out a large number of shiitake mushroom target detection data sets from the extracted frame images, wherein the shiitake mushroom target detection data sets include images of the mushroom cap from a top-down perspective and images of the mushroom gills from a top-down perspective;

[0037] Step (3), the dataset is labeled according to six quality categories: white, normal, deformed, unopened, slightly opened, and fully opened;

[0038] The second step is to train a mushroom target detection model based on the YOLOv8 deep learning algorithm;

[0039] Step (1), model training;

[0040] Use the YOLOv8 target detection model as the mushroom target detection model, and use the mushroom target detection dataset constructed in the first step above to train the mushroom target detection model;

[0041] Step (2), matching of multi-view targets; the three images of the mushroom sticks taken by the mushroom recognition depth camera, the first depth camera and the second depth camera are combined into a batch, and the batch is input into the mushroom target detection model to obtain the mushroom detection results under three perspectives: top view, right side view and left side view: det_d, det_ur, det_ul; the rule-based method is used to achieve the matching of the cap and the gills. First, the midline of the image in the width direction is used as the dividing line, and the detection frame in det_d is divided into two parts: det_dr and det_ul. det_dl, let the detection frame in the upper part det_dr correspond to the detection result det_ur of the upward-looking gill image on the right side of the mushroom stick, and let the detection frame in the lower part det_dl correspond to the detection result det_ul of the upward-looking gill image on the left side of the mushroom stick; determine the list of targets to be picked in the downward-looking cap image, the following four situations are targets to be picked: ① The cap category is white and the gill category is not opened, ② The cap category is white and the gill category is slightly opened, ③ The cap category is normal and the gill category is not opened, ④ The cap category is normal and the gill category is slightly opened.

[0042] Preferably, in step (2) of the first step, the extracted frame images are preliminarily screened by an image quality assessment method, and then images containing incomplete mushroom sticks are manually checked and removed, and then image frames extracted within a few seconds at the beginning and end of the original video are selected to finally obtain a shiitake mushroom target detection dataset.

[0043] Preferably, in the second step, the mushroom target detection model is formed by removing the detection head of the P3 layer, the P5 feature layer and the detection head of the P5 feature layer on the basis of the basic architecture of the YOLOv8 target detection model, and retaining only the detection head output of the P4 layer.

[0044] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of any one of the above methods is implemented.

[0045] The beneficial effects of the present invention are as follows: (1) the degree of automation is high, the mushroom sticks are automatically taken out from the narrow space of each layer of the mushroom rack, the mature mushrooms on the mushroom sticks are automatically picked off, and the mushroom sticks are automatically put back into the mushroom rack; (2) the picking operation is efficient; (3) the picking operation process will not damage the mushroom caps, nor will it cause damage to the mushroom sticks; (4) the robot's action process is reliable and stable; (5) the execution terminal is small in size, and the action is flexible and reliable; (6) accurate visual recognition is used to automatically pick the mature mushrooms on the mushroom sticks, and the immature mushrooms are retained to continue growing on the mushroom sticks; (7) multi-view target matching associates the category information in the detection results of the mushroom caps and gills of the same target, and comprehensively judges whether the mushrooms meet the picking requirements; (8) the mushroom targets to be picked are divided into quality grades, and graded picking is performed.

[0046] Further features and aspects of the present invention will be clearly described in the following description of specific embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is an axonometric view of a mushroom picking robot in a factory;

[0048] FIG2 is a front view of the factory mushroom picking robot shown in FIG1 ;

[0049] FIG3 is a left side view of the factory mushroom picking robot shown in FIG1 ;

[0050] FIG4 is a right side view of the factory mushroom picking robot shown in FIG1 ;

[0051] FIG5 is a top view of the factory mushroom picking robot shown in FIG1 ;

[0052] FIG6 is a bottom view of the factory mushroom picking robot shown in FIG1 ;

[0053] Figure 7 is an axonometric view of the factory mushroom picking robot from another perspective;

[0054] Figure 8 is an axonometric view of the factory mushroom picking robot from another perspective;

[0055] Figure 9 is an axonometric view of the factory mushroom picking robot from another perspective;

[0056] Figure 10 is an axonometric view of the factory mushroom picking robot from another perspective;

[0057] Figure 11 is an axonometric view of a mushroom stick taking device;

[0058] FIG12 is a front view of the mushroom stick taking device shown in FIG11;

[0059] FIG13 is a top view of the mushroom stick taking device shown in FIG11 ;

[0060] FIG14 is a bottom view of the mushroom stick taking device shown in FIG11;

[0061] FIG15 is a right side view of the mushroom stick taking device shown in FIG11;

[0062] Figure 16 is an axonometric view of a mushroom stick taking device;

[0063] Figure 17 is an axonometric view of a mushroom stick taking device;

[0064] FIG18 is a schematic structural diagram of the mushroom stick taking device, wherein an L-shaped connecting plate 1, a slider 1 and a nut seat 1 are connected, and an L-shaped connecting plate 2, a slider 2 and a nut seat 2 are connected;

[0065] FIG19 is a partial enlarged view of the connection between the execution end and the robotic arm of the mushroom picking device in FIG7;

[0066] Figure 20 is an axonometric view of the execution end;

[0067] FIG21 is a front view of the execution end shown in FIG20;

[0068] FIG22 is a top view of the execution end shown in FIG20;

[0069] FIG23 is a bottom view of the execution end shown in FIG20;

[0070] FIG24 is an axonometric view of the execution terminal shown in FIG20 from another perspective;

[0071] FIG25 is an axonometric view of the execution terminal shown in FIG20 from another perspective;

[0072] FIG26 is a schematic diagram showing a state in which two clamping blocks at the end of the actuator clamp the stem of a shiitake mushroom;

[0073] FIG27 is a schematic diagram showing a state in which two clamping blocks at the end of the actuator clamp the stem of a shiitake mushroom;

[0074] FIG28 is a schematic diagram of a factory mushroom picking robot positioned next to a mushroom rack, removing mushroom sticks from the rack;

[0075] Figure 29 is a side view of Figure 28;

[0076] FIG30 is a partial enlarged view of part of the structure in FIG28;

[0077] FIG31 is a schematic diagram of the structure shown in FIG1 , in which the mushroom stick taking device is lowered to a position close to the execution end of the mushroom picking device;

[0078] FIG32 is a front view of the structure shown in FIG31;

[0079] FIG33 is a side view of the structure shown in FIG31;

[0080] FIG34 is a top view of the structure shown in FIG31;

[0081] FIG35 is a schematic structural diagram of the mushroom stick in FIG31 near the execution end of the mushroom picking device;

[0082] Figure 36 is a schematic diagram of the process in which the depth camera starts shooting from directly above the mushroom stick, and the operator moves the depth camera at a constant speed around the side of the mushroom stick until it moves to the lower side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, and stops recording, as well as images obtained by extracting frames from the captured video;

[0083] Figure 37 shows examples of the six quality categories of shiitake mushrooms;

[0084] Figure 38 is a schematic diagram of the positions of the mushroom recognition depth camera, the first depth camera, and the second depth camera for capturing mushroom stick images, as well as images captured from a top-down perspective and an upward perspective;

[0085] FIG39 is a schematic diagram of the target matching process;

[0086] Figure 40 is the architecture diagram of the mushroom target detection model.

[0087] Explanation of symbols in the figure: 100. Chassis, 200. Lifting device, 201. First fixed plate, 202. Second fixed plate, 203. First electric cylinder, 204. Second electric cylinder, 205. Support plate; 300. Mushroom stick picking device, 301. First base, 302. First guide rail assembly, 303. Second guide rail assembly, 304. Y-axis direction lead screw, 305. First drive motor, 306. First synchronous pulley, 307. Second synchronous pulley, 308. First synchronous belt, 309. Slide plate, 310. Second base, 311. Third guide rail assembly , 312. X-axis bidirectional screw, 313. Slider 1, 314. Slider 2, 315. L-shaped connecting plate 1, 316. L-shaped connecting plate 2, 317. Clamping rod 1, 317-1. Inclined portion, 318. Clamping rod 2, 318-1. Inclined portion, 319. Nut seat 1, 320. Nut seat 2, 321. Second drive motor, 322. Third synchronous pulley, 323. Fourth synchronous pulley, 324. Two synchronous belts; 400. Mushroom picking device, 401. Lifting mechanism, 402. Robotic arm, 403. Execution end, 403-1. Support frame, 403-2. Clamping drive motor, 403-3. Screw, 403-4. Screw nut, 403-5. Connecting block, 403-6. First connecting rod, 403-7. First V-shaped connecting rod, 403-8. Second connecting rod, 403-9. Second V-shaped connecting rod, 403 -10. First clamp, 403-10-1. Arc-shaped groove, 403-11. Second clamp, 403-11-1. Arc-shaped groove, 403-12. Mushroom recognition depth camera; 500. First depth camera, 600. Second depth camera; 1. Collection basket, 2. Mushroom sticks, 2-1. Mushroom, 3. Mushroom, 3-1. Mushroom stem, 3-2. Mushroom cap; 4. Mushroom rack, 5. Mushroom sticks, 6. Mushroom sticks, 6-1. Mushroom. DETAILED DESCRIPTION

[0088] The present invention will be further described in detail below with reference to the accompanying drawings using specific embodiments.

[0089] As shown in Figure 1-10, the factory mushroom picking robot includes a chassis 100, a lifting device 200, a mushroom stick picking device 300, and a mushroom picking device 400. The lifting device 200 is mounted on the chassis 100, the mushroom stick picking device 300 is connected to the lifting device 200, and the mushroom picking device 400 is mounted on the chassis 100. The lifting device 200 can drive the mushroom stick picking device 300 to move vertically. After the mushroom stick picking device 300 removes a mushroom stick from the mushroom rack, it moves downward to a position close to the mushroom picking device 400, driven by the lifting device 200. The mushroom picking device 400 then moves to pick the mushrooms from the mushroom stick. The picked mushrooms are then placed in a collection basket 1 (two collection baskets 1 are shown placed on the chassis 100).

[0090] The lifting device 200 includes a first fixed plate 201, a second fixed plate 202, a first electric cylinder 203, a second electric cylinder 204, and a support plate 205. The first electric cylinder 203 is fixedly connected to the first fixed plate 201, and the second electric cylinder 204 is fixedly connected to the second fixed plate 202. The first electric cylinder 203 and the second electric cylinder 204 are arranged vertically side by side. The first electric cylinder 203 is provided with a first slider, and the second electric cylinder 204 is provided with a second slider. One end of the support plate 205 is fixedly connected to the first slider on the first electric cylinder 203, and the other end of the support plate 205 is fixedly connected to the second slider on the second electric cylinder 204.

[0091] The first fixing plate 201 is fixedly connected to the chassis 100, and the second fixing plate 202 is fixedly connected to the chassis 100, thereby fixing the lifting device 200 to the chassis 100. When the first electric cylinder 203 and the second electric cylinder 204 are actuated simultaneously, the first and second sliders move the support plate 205 upward or downward, causing the support plate 205 to rise and fall in the vertical direction.

[0092] The mushroom stick taking device 300 is fixedly mounted on the support plate 205 , and the support plate 205 is lifted up and down in the vertical direction, thereby driving the mushroom stick taking device 300 to be lifted up and down in the vertical direction.

[0093] As shown in Figures 11-18, the mushroom stick taking device 300 includes a first base 301, a first guide rail assembly 302, a second guide rail assembly 303, a Y-axis direction screw rod 304, a nut, a first drive motor 305, a first synchronous pulley 306, a second synchronous pulley 307, a first synchronous belt 308, a slide plate 309, a second base 310, a third guide rail assembly 311, an X-axis direction bidirectional screw rod 312, a slider 1 313, a slider 2 314, an L-shaped connecting plate 1 315, an L-shaped connecting plate 2 316, a clamping rod 1 317, a clamping rod 2 318, a nut seat 1 319, a nut seat 2 320, a second drive motor 321, a third synchronous pulley 322, a fourth synchronous pulley 323, a second synchronous belt 324, and a third synchronous pulley 325. The first guide rail assembly 302 and the second guide rail assembly 303 are respectively fixedly mounted on the first base 301, and the first guide rail assembly 302 and the second guide rail assembly 303 are arranged side by side. The front end of the Y-axis direction screw rod 304 is rotatably connected to the front of the first base 301 through a bearing, and the rear end of the Y-axis direction screw rod 304 is rotatably connected to the rear end of the first base 301 through a bearing. The Y-axis direction screw rod 304 is located between the first guide rail assembly 302 and the second guide rail assembly 303. The first drive motor 305 is fixedly mounted on the rear end of the first base 301, the first synchronous pulley 306 is connected to the output shaft of the first drive motor 305, the second synchronous pulley 307 is connected to the rear end of the Y-axis direction screw rod 304, and the first synchronous belt 308 is connected to the first Between the synchronous pulley 306 and the second synchronous pulley 307, the nut is connected to the Y-axis direction lead screw 304, the slide 309 is fixedly connected to the nut, one side of the slide 309 is fixedly connected to the slider on the first guide rail assembly 302, and the other side of the slide 309 is fixedly connected to the slider on the second guide rail assembly 303. The second base 310 is fixedly mounted on the slide 309, and the third guide rail assembly 311 is fixedly mounted on the second base 310. The left end of the X-axis direction bidirectional lead screw 312 is rotatably connected to the left part of the second base 310 through a bearing, and the right end of the X-axis direction bidirectional lead screw 312 is rotatably connected to the right part of the second base 310 through a bearing. Nut seat 1 319 and nut seat 2 320 are respectively connected to the X-axis direction bidirectional lead screw 312 Connection, slider 1 313 is fixedly connected to nut seat 1 319, slider 2 314 is fixedly connected to nut seat 2 320, L-shaped connecting plate 1 315 is fixedly connected to slider 1 313, L-shaped connecting plate 2 316 is fixedly connected to slider 2 314, the second drive motor 321 is fixedly connected to the left part of the second base 310, the third synchronous pulley 322 is connected to the output shaft of the second drive motor 321, the fourth synchronous pulley 323 is connected to the left end of the bidirectional screw rod 312 in the X-axis direction, the second synchronous belt 324 is connected between the third synchronous pulley 322 and the fourth synchronous pulley 323, the rear end of the clamping rod 1 317 is fixedly connected to the L-shaped connecting plate 1 315, and the rear end of the clamping rod 2 318 is fixedly connected to the L-shaped connecting plate 2 316.

[0094] The working process of the mushroom stick picking device 300 is mainly that when the second drive motor 321 is working, the third synchronous pulley 322, the second synchronous belt 324, and the fourth synchronous pulley 323 are used to drive the X-axis direction bidirectional screw rod 312 to rotate, and the X-axis direction bidirectional screw rod 312 drives the nut seat 1 319 and the nut seat 2 320 to move closer to each other or move away from each other in the X-axis direction, thereby driving the slider 1 313 and the slider 2 314 to move closer to each other or move away from each other, and the L-shaped connecting plate 1 315 and the L-shaped connecting plate 2 316 move closer to each other or move away from each other. The first drive motor 305 rotates the Y-axis screw rod 304 through the first synchronous pulley 306, the first synchronous belt 308, and the second synchronous pulley 307. The nut connected to the Y-axis screw rod 304 drives the slide 309 to move forward or backward along the Y-axis. The slide 309 then drives the second base 310 to move forward or backward along the Y-axis, ultimately driving the first and second clamping rods 317 and 318 to move forward or backward as a whole. The mushroom stick retrieval device is stable and reliable in removing and placing mushroom sticks without damaging the mushroom sticks. The motor acts as a power source to drive the two clamping rods, and the clamping rods move with high precision. It is suitable for mushroom sticks of various sizes.

[0095] The first base 301 of the mushroom stick picking device 300 is fixedly mounted on the support plate 205 , and the support plate 205 is lifted and lowered in the vertical direction to drive the entire mushroom stick picking device 300 to move upward or downward in the vertical direction.

[0096] The mushroom picking device 400 includes a lifting mechanism 401, a robotic arm 402, and an actuator terminal 403. The lifting mechanism 401 is fixedly mounted on the chassis 100, the robotic arm 402 is connected to the lifting mechanism 401, and the actuator terminal 403 is connected to the free end of the robotic arm 402. The lifting mechanism 401 is located between the first electric cylinder 203 and the second electric cylinder 204 of the lifting mechanism 200. In other words, the mushroom picking device 400 is located between the first electric cylinder 203 and the second electric cylinder 204. The lifting mechanism 401 can drive the entire robotic arm 402 in vertical motion, adjusting the position of the robotic arm 402 and, therefore, the actuator terminal 403.

[0097] As shown in Figure 20-25, the execution end 403 includes a support frame 403-1, a clamping drive motor 403-2, a screw rod 403-3, a screw nut 403-4, a connecting block 403-5, a first connecting rod 403-6, a first V-shaped connecting rod 403-7, a second connecting rod 403-8, a second V-shaped connecting rod 403-9, a first clamping block 403-10, and a second clamping block 403-11. The clamping drive motor 403-2 is fixedly installed on the support frame 403-1, the screw rod 403-3 is fixedly connected to the output shaft of the clamping drive motor 403-2, the screw nut 403-4 is connected and matched with the screw rod 403-3, the connecting block 403-5 is fixedly connected to the screw nut 403-4, and the rear end of the first connecting rod 403-6 is rotatably connected to the connecting block 403-5. The front end of the first connecting rod 403-6 is rotatably connected to the rear end of the first V-shaped connecting rod 403-7, the middle part of the first V-shaped connecting rod 403-7 is rotatably connected to the support frame 403-1, the rear end of the second connecting rod 403-8 is rotatably connected to the connecting block 403-5, the front end of the second connecting rod 403-8 is rotatably connected to the rear end of the second V-shaped connecting rod 403-9, the middle part of the second V-shaped connecting rod 403-9 is rotatably connected to the support frame 403-1, the first connecting rod 403-6 and the second connecting rod 403-8 are symmetrically arranged, the first V-shaped connecting rod 403-7 and the second V-shaped connecting rod 403-9 are symmetrically arranged, the first clamping block 403-10 is fixedly connected to the front end of the first V-shaped connecting rod 403-7, and the second clamping block 403-11 is fixedly connected to the front end of the second V-shaped connecting rod 403-9. The first clamping block 403-10 is provided with an arcuate groove 403-10-1, and the second clamping block 403-11 is provided with an arcuate groove 403-11-1. When the clamping drive motor 403-2 is in operation, it can drive the screw 403-3 to rotate, and the screw nut 403-4 moves accordingly. The screw nut 403-4 drives the connecting block 403-5 to translate forward or backward. When the connecting block 403-5 moves forward, the first clamping block 403-10 and the second clamping block 403-11 move closer together and close. When the connecting block 403-5 moves backward, the first clamping block 403-10 and the second clamping block 403-11 move apart and open (Figure 21 shows the open state).

[0098] 19 , the support frame 403 - 1 is fixed to the free end of the robotic arm 402 by screws, and the first clamping block 403 - 10 and the second clamping block 403 - 11 are on the same horizontal plane.

[0099] Referring to Figures 26 and 27, when connecting block 403-5 moves forward, first clamping block 403-10 and second clamping block 403-11 move together to clamp the stem 3-1 of the shiitake mushroom 3 growing on the mushroom stick. Specifically, arcuate grooves 403-10-1 and arcuate grooves 403-11-1 clamp the stem 3-1. This clamping action does not damage the mushroom caps or the mushroom sticks, and the picking process is reliable and stable. The actuator terminal 40 is compact, flexible, and reliable, allowing for precise adjustment of the opening of the two clamping blocks. Powered by a motor and driven by a screw, the two clamping blocks move with high precision.

[0100] It should be noted that the specific structure of the robotic arm 402 shown in Figures 1-10 is only an example, and the specific structure of the robotic arm can also be any other structure that realizes the function of the robotic arm and drives the execution end 403 to any desired position in space.

[0101] As shown in Figures 28-30, two rows of mushroom sticks are placed on each layer of the mushroom rack 4, and each row consists of 12 mushroom sticks. There is a certain distance between two adjacent mushroom sticks. The above-mentioned factory mushroom picking robot is moved to the side of the mushroom rack 4, and the mushroom stick picking device 300 is facing a mushroom stick in a certain layer on the mushroom rack 4, and then the picking operation is performed. The specific process is that first, the second driving motor 321 of the mushroom stick picking device 300 works to make the slider 1 313 and the slider 2 314 move away from each other, so that the clamping rod 1 317 and the clamping rod 2 318 move away from each other in the X-axis direction, and then the first driving motor 305 works to drive the second base 310 to move forward along the Y-axis direction, thereby making the clamping rod 1 317 and the clamping rod 2 318 move forward along the Y-axis direction (facing the target mushroom stick), and the clamping rod 1 317 and the clamping rod 2 318 extend into the space on both sides of the target mushroom stick respectively ( That is, the target bacteria stick is located between the clamping rod 1 317 and the clamping rod 2 318, and at this time the two clamping rods do not contact the two sides of the bacteria stick), and then the second drive motor 321 works to make the clamping rod 1 317 and the clamping rod 2 318 move closer to each other in the X-axis direction, so that the clamping rod 1 317 and the clamping rod 2 318 are clamped on both sides of the target bacteria stick, thereby clamping the target bacteria stick; then the first drive motor 305 works to drive the second base 310 to move backward along the Y-axis direction, and the clamping rod 1 317 and the clamping rod 2 318 move backward with the target bacteria stick, thereby removing the target bacteria stick 2 from the bacteria rack (refer to the state shown in Figures 30 and 29). The state of the removed bacteria stick 2 is shown in Figures 1, 3, 7, and 9.

[0102] Next, the lifting device 200 moves downward, driving the mushroom stick picking device 300 to move downward with the mushroom stick 2 to a position close to the execution end 403 of the mushroom picking device 400, as shown in Figures 31-35. The multiple mushrooms 2-1 growing on the mushroom stick 2 are within the operating range of the mushroom picking device 400.

[0103] Next, the mushroom picking device 400 picks the mushrooms 2-1 on the mushroom stick 2. Specifically, the robotic arm 402 moves the actuator end 403 to the mushroom 2-1. The first clamping block 403-10 and the second clamping block 403-11 of the actuator end 403 are positioned on both sides of the mushroom 2-1 in the open state. The first clamping block 403-10 and the second clamping block 403-11 then move together to clamp the stem of the mushroom 2-1 (see Figures 26 and 27 for the clamping state). The robotic arm 402 then moves the actuator end 403 as a whole a certain distance to remove the mushroom 2-1 from the mushroom stick 2. It can be seen that when the actuator end 403 clamps the mushrooms, it does not touch the mushroom caps, so it does not damage the mushroom caps. In addition, it does not touch the mushroom stick 2, so it does not damage the mushroom stick. When clamping a mature mushroom, it does not touch other nearby mushrooms, so it does not damage them.

[0104] Next, the robotic arm 402 moves the execution end 403 to the top of the collection basket 1, and then the first clamping block 403-10 and the second clamping block 403-11 of the execution end 403 move away from each other to an open state, and the clamped mushrooms 2-1 are released and fall into the collection basket 1.

[0105] According to the above picking method, the mushroom picking device 400 picks off the other mushrooms on the mushroom stick 2 and puts them into the collecting basket 1. After all the shiitake mushrooms on the mushroom sticks 2 are picked off, the mushroom sticks are placed back to their original positions or other positions on the mushroom rack 4, and the lifting device 200 moves upward with the mushroom stick picking device 300, and the mushroom stick picking device 300 moves the processed mushroom sticks to the original position or other position of the mushroom rack 4, and then the first drive motor 305 works to drive the second base 310 to move forward along the Y-axis direction, thereby causing the clamping rod 1 317 and the clamping rod 2 318 to move forward along the Y-axis direction with the processed mushroom sticks, and the processed mushroom sticks are moved to the corresponding positions on the mushroom rack, and then the clamping rod 1 317 and the clamping rod 2 318 of the mushroom stick picking device 300 move away from each other, so that the clamping force applied to both sides of the processed mushroom sticks disappears, and the processed mushroom sticks are placed on the mushroom rack, and then the first drive motor 305 works to drive the clamping rod 1 317 and the clamping rod 2 318 to move backward.

[0106] Next, the robot continues to pick other mushroom sticks on the mushroom rack 4.

[0107] As can be seen, the above-described picking process is highly automated and highly efficient. The actuator terminal 403 is compact, flexible, and reliable. Driven by the robotic arm 402, the actuator terminal 403 can reach any position within the space to grasp the stem of the target shiitake mushroom.

[0108] It should be noted that a mushroom recognition depth camera 403-12 can be installed on the execution terminal 403, as shown in Figures 35, 19, and 3. The mushroom recognition depth camera 403-12 is fixedly connected to the support frame 403-1. The mushroom recognition depth camera 403-12 captures images of the mushroom sticks and transmits them to the controller. The controller uses machine vision technology to identify the mushrooms on the mushroom sticks and controls the robot arm 402 to move the execution terminal 403 to the mushrooms to be picked. This achieves more accurate movement of the execution terminal to the mushrooms to be picked, more accurately achieving the two clamps clamping the mushroom stems, and then picking the target mushrooms. In order to further optimize and improve, two more depth cameras are set up for a total of three cameras to identify the target mushrooms to be picked on the mushroom sticks. Referring to Figures 2, 5, 7, 8, and 10, the two depth cameras are mounted on the support plate 205 through an adjustable bracket. The first adjustable bracket is connected to the support plate 205, and the first depth camera 500 is connected to the first adjustable bracket (the position and viewing angle of the first depth camera 500 can be adjusted by the first adjustable bracket). The second adjustable bracket is connected to the support plate 205, and the second depth camera 600 is connected to the second adjustable bracket (the position and viewing angle of the second depth camera 600 can be adjusted by the second adjustable bracket); the first depth camera 500 can specifically adopt a realsense D435 depth camera, and the second depth camera 600 can specifically adopt a realsense D435 depth camera; the specific recognition and control methods are as follows:

[0109] The picking method of shiitake mushroom detection and grading based on multi-view fusion mainly includes the following steps:

[0110] The first step is to build a multi-view data set for training the mushroom target detection model.

[0111] Step (1) collects video data of mushroom sticks. In the mushroom cultivation factory, the RGB sensor of the Intel RealSense D435 depth camera is used to collect video stream data. The resolution of the Intel RealSense D435 depth camera is adjusted to 1920*1080, and the frame rate is set to 60fps. Referring to Figure 36, the camera imaging plane is maintained at a distance of 300-400mm from the surface of the mushrooms growing on the mushroom sticks, so that the width direction of the image frame is consistent with the height direction of the mushroom stick cylinder, and the field of view is ensured to include the entire mushroom stick growing the mushrooms. The initial position of the depth camera is directly above the mushroom stick and it takes a downward shot of the mushroom top on the mushroom stick. After the video shooting starts, the handheld depth camera moves around the side of the mushroom stick at a constant speed (moving in the direction of the dotted arrow in Figure 36), keeping the position and size of the mushroom stick constant in the camera's field of view. When the camera moves around to the lower left side of the mushroom stick and takes an upward shot of the clear gills of the mushroom (position 1 in Figure 36), the recording is stopped, and the first video data collection of a mushroom stick is completed. Next, the second video is recorded. The initial position of the depth camera is directly above the mushroom stick and it takes a downward shot of the mushroom top on the mushroom stick. After the video shooting starts, the handheld depth camera moves around the side of the mushroom stick at a constant speed, keeping the position and size of the mushroom stick constant in the camera's field of view. When the camera moves around to the lower right side of the mushroom stick and takes an upward shot of the clear gills of the mushroom (position 2 in Figure 36), the recording is stopped, and the second video data collection of a mushroom stick is completed. In this embodiment, a total of 502 videos were collected for 251 mushroom sticks, and the average length of each video was 8 seconds.

[0112] Step (2) is to obtain a static image of the mushroom. The frame extraction interval is set to 30, and each section of the mushroom stick video stream is extracted to obtain a frame-extracted image. Extracting frames from motion videos to obtain static images is prone to image blurring. In order to screen and obtain clear, high-quality images, the frame-extracted images are preliminarily automatically screened using an image quality assessment method. In this step, the image quality is defined as: Q = αV + βS + γC

[0113] Image quality Q is the quality evaluation index of the reference-free image. In the above formula, α, β, and γ are the coefficients of each term;

[0114] The formula for the gradient variance V is:

[0115] In the gradient variance V formula, and is the average value of the gradient, and N is the total number of pixels in the image;

[0116] The formula for image sharpness S is:

[0117] In the image sharpness S formula, is the Laplace operator, I(x,y) is the grayscale value of the image;

[0118] The formula for contrast C is:

[0119] In the contrast C formula, δ(i,j) is the grayscale value difference between adjacent pixels, P δ (i, j) is the distribution probability of δ.

[0120] Based on the above-mentioned quality evaluation index Q of the reference-free image, a threshold method is used to automatically perform a preliminary screening of the extracted frame images. Then, images containing incomplete mushroom sticks are manually inspected and removed to obtain clear, high-quality static images. On this basis, in order to obtain a targeted image set of the cap and gills, image frames extracted within the first 2 seconds and the last 2 seconds of the original video are selected, and finally a dataset of 820 usable shiitake mushroom target detection images is obtained. The shiitake mushroom target detection dataset mainly includes images of the cap from a top-down perspective and images of the gills from an upward perspective. It should be noted that the first 2 seconds and the last 2 seconds of the original video are only examples and are not intended to limit the scope of protection. In fact, the first few seconds and the last few seconds of the original video are all feasible.

[0121] Step (3), data labeling. In order to achieve supervised learning of the deep learning model, the data set needs to be labeled. In this embodiment, according to the actual production needs of the mushroom cultivation factory, the collected images are labeled according to six quality categories: white, normal, deformed, unopened, slightly open, and fully open. See the mushroom image labeling sample shown in Figure 37. The quality of the cap is from high to low: white, normal, deformed, and the quality of the gills is from high to low: unopened, slightly open, and fully open. The specific process can be to use a semi-supervised assisted annotation tool based on the SAM model. During the actual annotation process, click the central area of ​​a mushroom target on the image to generate a complete mask for the current target. By clicking to increase or decrease the mask area, you can get the precise mask annotation of the target, and then select the corresponding quality category for the current target to complete the annotation of a mushroom. Since a json file containing instance segmentation annotations of all mushroom targets is generated for each image in this process, the annotation file also needs to be converted into a txt file containing only rectangular box annotations. Finally, the dataset is divided into a training set containing 700 images and a test set of 120 images.

[0122] Using a semi-supervised assisted annotation tool based on the SAM model can improve the efficiency of manual annotation. The SAM model is short for Segment Anything Model.

[0123] The second step is to train the mushroom target detection model based on the YOLOv8 deep learning algorithm.

[0124] Step (1), model training and verification.

[0125] Considering the uniformity of the scale of the mushroom targets in this task and the practical deployment scenario with limited computing resources, this embodiment, based on the basic architecture of the YOLOv8 object detection model, removes the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer, retaining only the detection head output of the P4 layer, thereby forming a lightweight single-scale output model (i.e., the mushroom target detection model), as shown in Figure 39. This single-scale output model reduces the number of model parameters and optimizes the model inference speed. The single-scale output model (the mushroom target detection model) is trained using the mushroom target detection dataset constructed in the first step above. During the training process, CIoULoss and BCELoss are used as loss functions for the detection box regression and category prediction branches, respectively, and the AdamW optimizer is used to improve the model convergence speed. After sufficient training, the single-scale output model (the mushroom target detection model) achieves a mAP value of 0.719 in the validation phase, making it suitable for algorithm deployment. It should be noted that the mAP value of 0.719 during validation is an example and is not intended to limit the scope of protection of the technical solution.

[0126] Step (2), multi-view target matching. Because the determination of the quality level of shiitake mushroom targets requires the combination of visual information of the cap and gills, a shiitake mushroom recognition depth camera 403-12 that looks down to shoot the cap and two depth cameras 500 and 600 that look up to shoot the gills are deployed on the robot to simultaneously obtain two-view images of the shiitake mushrooms on the mushroom sticks. Referring to Figure 38, three images are taken for each mushroom stick taken and combined into a batch. The batch is input into the shiitake mushroom target detection model to obtain the shiitake mushroom detection results under three perspectives: looking down, looking up from the right, and looking up from the left: det_d, det_ur, and det_ul. The format of det_d, det_ur, and det_ul is [id, x, y, w, h, conf, cls], where id is the detection frame number, x, y, w, h are the center coordinates of the detection frame and its width and height, conf is the category confidence, and cls is the category of the target.

[0127] The purpose of multi-view object matching is to associate the category information in the cap and gill detection results of the same target, and comprehensively determine whether the mushroom meets the picking requirements. Because the installation position and posture of the mushroom recognition depth camera 403-12, the first depth camera 500, and the second depth camera 600 are controllable, the range of the mushroom stick in the field of view of the three cameras is roughly consistent. A rule-based method can be used to match the cap and gills. Referring to the rightmost image in Figure 38, the detection box in det_d is first divided into two upper and lower parts, det_dr and det_dl, using the midline of the image width as the dividing line. The detection box in the upper part det_dr is set to correspond to the detection result det_ur of the gill image viewed from the right side of the mushroom stick, and the detection box in the lower part det_dl is set to correspond to the detection result det_ul of the gill image viewed from the left side of the mushroom stick. The specific process of target matching can be, referring to Figure 39, taking the above part det_dr as an example, for each cap target det_up, the center point of the cap target det_up is (x_d, y_d), the width is w_d, and the width of the search range F in the x-axis direction is specified to be 1.5 times w_d. Then, in det_ur, a list of detection boxes det_ups is collected whose distance between the coordinate value x_u of the gill target center point and x_d is less than the width of the search range F, that is, in det_ur, a list of detection boxes det_ups whose width in the x-axis direction is greater than that of the cap target det_up is collected. The detection frame list det_ups in a slightly larger range is collected, and then the detection frame with the smallest y-axis coordinate value is selected from det_ups, which is the gill target corresponding to the cap target det_up; similarly, for each cap target det_down in the lower part det_dl, the detection frame list det_downs in the range slightly larger than the width of the cap target det_down in the x-axis direction is collected in det_ul, and then the detection frame with the smallest y-axis coordinate value is selected from det_downs, which is the gill target corresponding to the cap target det_down. Finally, according to actual production needs, the list of targets to be picked in the top-view cap image is determined. Mushrooms with deformed caps or fully open gills do not meet the picking standards; mushrooms with white or normal caps and unopened or slightly open gills meet the picking standards. In other words, the following four situations are targets to be picked: ① White caps and unopened gills, ② White caps and slightly open gills, ③ Normal caps and unopened gills, ④ Normal caps and slightly open gills. They can be divided into high to low quality grades. The first situation is a special grade, the second is a superior grade, the third is a sub-excellent grade, and the fourth is an ordinary grade.

[0128] The third step is to deploy the mushroom target detection model in the controller.

[0129] In the fourth step, the controller controls the robot arm and performs the end action to perform the picking task.

[0130] In step (1), the mushroom stick picking device 300 moves downward with the mushroom stick 2 to a position close to the execution end 403 in the mushroom picking device 400. The controller controls the movement of the robotic arm so that the mushroom recognition depth camera 403-12 reaches the photo taking position above the mushroom stick. The mushroom recognition depth camera 403-12, the first depth camera 500 and the second depth camera 600 collect the RGB image and depth image of the mushroom stick and send them to the controller.

[0131] In step (2), the RGB image is input into the mushroom target detection model, and the target to be picked is identified through multi-view target matching.

[0132] In step (3), the controller obtains the three-dimensional coordinates of the center point of the cap of the target to be picked, and then converts the three-dimensional coordinates of the center point of the cap into position information in the base coordinate system of the robotic arm through coordinate conversion. The robotic arm action guides the execution end to move to the target to be picked, and the execution end performs a picking action to pick the target mushrooms on the mushroom stick.

[0133] Furthermore, regarding the specific structure of the mushroom stick handling device 300, to further optimize the gripping effect of the two clamping rods on the mushroom sticks in the mushroom stick handling device 300, as shown in Figures 11, 17, and 16, a beveled portion 317-1 is provided on the inner side of the first clamping rod 317, and a beveled portion 318-1 is provided on the inner side of the second clamping rod 318. Referring to Figure 35, since the outer side of the mushroom stick is arc-shaped, the beveled portions 317-1 and 318-1 act on both sides of the mushroom stick, enabling a more stable and reliable gripping of the mushroom stick.

[0134] It should be noted that a mobile platform can be installed on the chassis 100. The mobile platform is a conventional product in the prior art. The mobile platform is used to move the entire robot on the ground to a position that is convenient for picking operations.

[0135] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. If inspired by this description, those skilled in the art may, without departing from the spirit of the present invention, employ other component configurations, drive mechanisms, and connection methods, and other structural embodiments and implementations similar to the present invention without inventive design, and without departing from the scope of the present invention, such embodiments shall fall within the scope of protection of the present invention.

Claims

1. An industrial shiitake mushroom picking robot, characterized in that, It includes a mobile platform, a chassis, a lifting device, a mushroom stick picking device and a shiitake mushroom picking device. The chassis is connected to the mobile platform, the lifting device is connected to the chassis, the mushroom stick picking device is connected to the lifting device, and the shiitake mushroom picking device is connected to the chassis; The lifting device includes a first fixing plate, a second fixing plate, a first electric cylinder, a second electric cylinder and a support plate. The first electric cylinder is fixedly connected to the first fixing plate, the second electric cylinder is fixedly connected to the second fixing plate. The first electric cylinder and the second electric cylinder are arranged side by side vertically. The first electric cylinder is provided with a first slider, and the second electric cylinder is provided with a second slider. One end of the support plate is fixedly connected to the first slider on the first electric cylinder, and the other end of the support plate is fixedly connected to the second slider on the second electric cylinder; The first fixing plate is fixedly connected to the chassis, and the second fixing plate is fixedly connected to the chassis; The mushroom stick picking device includes a first base, a first guide rail assembly, a second guide rail assembly, a Y-axis direction lead screw, a nut, a first drive motor, a first synchronous pulley, a second synchronous pulley, a first synchronous belt, a slide plate, a second base, a third guide rail assembly, an X-axis direction bidirectional lead screw, a first slider, a second slider, an L-shaped connecting plate one, an L-shaped connecting plate two, a clamping rod one, a clamping rod two, a nut seat one, a nut seat two, a second drive motor, a third synchronous pulley, a fourth synchronous pulley, and a second synchronous belt. The first guide rail assembly and the second guide rail assembly are respectively fixedly connected to the first base, the first guide rail assembly and the second guide rail assembly are arranged side by side. The front end of the Y-axis direction lead screw is rotationally connected to the front part of the first base through a bearing, and the rear end of the Y-axis direction lead screw is rotationally connected to the rear part of the first base through a bearing. The Y-axis direction lead screw is located between the first guide rail assembly and the second guide rail assembly. The first drive motor is connected to the rear part of the first base. The first synchronous pulley is connected to the output shaft of the first drive motor. The second synchronous pulley is connected to the rear end of the Y-axis direction lead screw. The first synchronous belt is connected between the first synchronous pulley and the second synchronous pulley. The nut is connected to the Y-axis direction lead screw. The slide plate is fixedly connected to the nut. One side of the slide plate is fixedly connected to the slider on the first guide rail assembly, and the other side of the slide plate is fixedly connected to the slider on the second guide rail assembly. The second base is fixedly connected to the slide plate. The third guide rail assembly is fixedly connected to the second base. The left end of the X-axis direction bidirectional lead screw is rotationally connected to the left part of the second base through a bearing, and the right end of the X-axis direction bidirectional lead screw is rotationally connected to the right part of the second base through a bearing. The nut seat one and the nut seat two are respectively connected to the X-axis direction bidirectional lead screw. The first slider is fixedly connected to the nut seat one, and the second slider is fixedly connected to the nut seat two. The L-shaped connecting plate one is fixedly connected to the first slider, and the L-shaped connecting plate two is fixedly connected to the second slider. The second drive motor is fixedly connected to the left part of the second base. The third synchronous pulley is connected to the output shaft of the second drive motor. The fourth synchronous pulley is connected to the left end of the X-axis direction bidirectional lead screw. The second synchronous belt is connected between the third synchronous pulley and the fourth synchronous pulley. The rear end of the clamping rod one is fixedly connected to the L-shaped connecting plate one, and the rear end of the clamping rod two is fixedly connected to the L-shaped connecting plate two. The first base is fixedly connected to the support plate of the lifting device; The shiitake mushroom picking device includes a lifting mechanism, a robotic arm, and an execution end. The lifting mechanism is fixedly connected to the chassis. The robotic arm is connected to the lifting mechanism, and the execution end is connected to the free end of the robotic arm. The execution end includes a support frame, a clamping drive motor, a lead screw, a lead screw nut, a connecting block, a first connecting rod, a first V-shaped connecting rod, a second connecting rod, a second V-shaped connecting rod, a first clamping block, and a second clamping block. The clamping drive motor is fixedly connected to the support frame. The lead screw is fixedly connected to the output shaft of the clamping drive motor. The lead screw nut is connected to the lead screw. The connecting block is fixedly connected to the lead screw nut. The rear end of the first connecting rod is rotatably connected to the connecting block, and the front end of the first connecting rod is rotatably connected to the rear end of the first V-shaped connecting rod. The middle of the first V-shaped connecting rod is rotatably connected to the support frame. The rear end of the second connecting rod is rotatably connected to the connecting block, and the front end of the second connecting rod is rotatably connected to the rear end of the second V-shaped connecting rod. The middle of the second V-shaped connecting rod is rotatably connected to the support frame. The first connecting rod and the second connecting rod are symmetrically arranged. The first V-shaped connecting rod and the second V-shaped connecting rod are symmetrically arranged. The first clamping block is connected to the front end of the first V-shaped connecting rod, and the second clamping block is connected to the front end of the second V-shaped connecting rod. The first clamping block is provided with an arc-shaped groove, and the second clamping block is provided with an arc-shaped groove. The support frame of the execution end is fixedly connected to the free end of the robotic arm. The lifting mechanism is located between the first electric cylinder and the second electric cylinder of the lifting device.

2. The factory shiitake mushroom picking robot according to claim 1, characterized in that, On the inner side of the first clamping rod in the mushroom stick picking device, there is an inclined surface, and on the inner side of the second clamping rod, there is an inclined surface.

3. The factory shiitake mushroom picking robot according to claim 1, characterized in that, The picking device further includes a shiitake mushroom recognition depth camera, which is connected to the support frame of the execution end.

4. A vision-based grading and picking method for the factory shiitake mushroom picking robot described in claim 1, characterized in that, The picking device further includes a shiitake mushroom recognition depth camera, which is connected to the support frame of the execution end. A first depth camera is connected to the support plate through a first adjustable bracket, and a second depth camera is connected to the support plate through a second adjustable bracket. The vision-based grading and picking method includes the following steps: Step 1, construct a multi-view data set for training a shiitake mushroom target detection model. Step (1), use a depth camera to collect mushroom stick video data. The initial position of the depth camera is directly above the mushroom stick and shoots downward. After starting the video shooting, let the depth camera move uniformly around the side of the mushroom stick. When the depth camera moves around to the lower left side of the mushroom stick and shoots up to the gill part of the shiitake mushroom, stop recording. Thus, the first segment of video data of one mushroom stick is collected. Next, record the second segment of video. The initial position of the depth camera is directly above the mushroom stick and shoots downward. After starting the video shooting, let the depth camera move uniformly around the side of the mushroom stick. When the camera moves around to the lower right side of the mushroom stick and shoots up to the gill part of the shiitake mushroom, stop recording. Thus, the second segment of video data of one mushroom stick is collected. For multiple mushroom sticks, multiple segments of video are collected. Step (2), obtain the static images of shiitake mushrooms; extract frames from each video stream of the mushroom sticks to obtain the frame-extracted images, and screen out a large number of shiitake mushroom target detection datasets from the frame-extracted images. The shiitake mushroom target detection datasets include the cap images from a top-down perspective and the gill images from a bottom-up perspective. Step (3), label the dataset according to six quality categories: white surface, normal, deformed, unopened, slightly opened, and fully opened. The second step, train the shiitake mushroom target detection model based on the YOLOv8 deep learning algorithm. Step (1), training of the model. Based on the basic architecture of the YOLOv8 target detection model, the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer are removed, and only the output of the detection head of the P4 layer is retained, thus forming the shiitake mushroom target detection model. The shiitake mushroom target detection dataset constructed in the first step above is used to train the shiitake mushroom target detection model. Step (2), matching of multi-view targets; the shiitake mushroom recognition depth camera is used to take top-down pictures of the caps, and the first depth camera and the second depth camera are used to take bottom-up pictures of the gills. The three images taken by the shiitake mushroom recognition depth camera, the first depth camera, and the second depth camera are combined into a batch. Inputting the batch into the shiitake mushroom target detection model can obtain the shiitake mushroom detection results from three perspectives: top-down, right-side bottom-up, and left-side bottom-up: det_d, det_ur, det_ul; use a rule-based method to achieve the matching of the cap and the gills. First, take the midline in the width direction of the picture as the dividing line, divide the detection boxes in det_d into upper and lower parts det_dr and det_dl, make the detection boxes in the upper part det_dr correspond to the detection results det_ur of the right-side bottom-up gill images of the mushroom stick, and make the detection boxes in the lower part det_dl correspond to the detection results det_ul of the left-side bottom-up gill images of the mushroom stick; determine the list of targets to be picked in the top-down cap image. The following four situations are targets to be picked: ① the cap category is white surface and the gill category is unopened, ② the cap category is white surface and the gill category is slightly opened, ③ the cap category is normal and the gill category is unopened, ④ the cap category is normal and the gill category is slightly opened. The third step, deploy the shiitake mushroom target detection model in the controller. The fourth step, the controller controls the robotic arm and the end effector to perform actions and execute the picking task. Step (1), the mushroom stick handling device moves the mushroom stick to a position close to the end effector in the shiitake mushroom picking device. The controller controls the robotic arm to move so that the shiitake mushroom recognition depth camera reaches the photography position above the mushroom stick. The shiitake mushroom recognition depth camera, the first depth camera, and the second depth camera collect the RGB images and depth images of the mushroom stick and send them to the controller. Step (2), input the RGB image into the shiitake mushroom target detection model, and realize the recognition of the target to be picked through multi-view target matching. Step (3), the controller obtains the three-dimensional coordinates of the center point of the cap of the target to be picked, and then converts the three-dimensional coordinates of the center point of the cap into the position information in the base coordinate system of the robotic arm through coordinate transformation. The robotic arm moves to guide the end effector to move to the target to be picked, and the end effector performs the picking action to pick the target shiitake mushroom on the mushroom stick.

5. The vision-based hierarchical picking method according to claim 4, characterized in that, During the process of multi-view target matching, for each cap target det_up in the upper part det_dr, a list of detection boxes det_ups within a range slightly wider than the width of the cap target det_up in the x-axis direction is collected from det_ur, and then the detection box with the minimum y-axis coordinate value is selected from det_ups, which is the gill target corresponding to the cap target det_up; for each cap target det_down in the lower part det_dl, a list of detection boxes det_downs within a range slightly wider than the width of the cap target det_down in the x-axis direction is collected from det_ul, and then the detection box with the minimum y-axis coordinate value is selected from det_downd, which is the gill target corresponding to the cap target det_down.

6. The vision-based hierarchical picking method according to claim 4, wherein In step (2) of the first step, the extracted frames are preliminarily screened by an image quality assessment method, and then the images containing incomplete mushroom sticks are removed through manual inspection. Then, the image frames extracted within several seconds at the beginning and the end of the original video are selected from them, and finally, a shiitake mushroom target detection dataset is obtained.

7. A method for detecting and grading Lentinula edodes based on multi-view fusion, characterized in that, It includes the following steps: The first step is to construct a multi-view dataset for training a shiitake mushroom target detection model; Step (1), use a depth camera to collect mushroom stick video data; the initial position of the depth camera is directly above the mushroom stick and takes a downward-looking shot. After starting the video shooting, let the depth camera move uniformly around the side of the mushroom stick. When the depth camera moves around to the lower left side of the mushroom stick and shoots up to the gill part of the shiitake mushroom, stop recording. Thus, the first segment of video data of a mushroom stick is collected; next, record the second segment of video. The initial position of the depth camera is directly above the mushroom stick and takes a downward-looking shot. After starting the video shooting, let the depth camera move uniformly around the side of the mushroom stick. When the camera moves around to the lower right side of the mushroom stick and shoots up to the gill part of the shiitake mushroom, stop recording. Thus, the second segment of video data of a mushroom stick is collected; for multiple mushroom sticks, multiple segments of video are collected; Step (2), obtain static images of shiitake mushrooms; Extract frames from each segment of the mushroom stick video stream to obtain extracted frames. A large number of shiitake mushroom target detection datasets are screened out from the extracted frames. The shiitake mushroom target detection dataset contains cap images from a top-down view and gill images from a bottom-up view; Step (3), label the dataset according to six quality categories: white surface, normal, deformed, unopened, slightly opened, and fully opened; The second step is to train a shiitake mushroom target detection model based on the YOLOv8 deep learning algorithm; Step (1), training of the model; Use the YOLOv8 target detection model as the shiitake mushroom target detection model, and train the shiitake mushroom target detection model with the shiitake mushroom target detection dataset constructed in the first step above. Step (2), matching of multi-view targets: Combine three images of the mushroom stick captured by the shiitake mushroom recognition depth camera, the first depth camera, and the second depth camera into a batch, and input the batch into the shiitake mushroom target detection model to obtain the shiitake mushroom detection results from three perspectives: top view, right-side up view, and left-side up view: det_d, det_ur, det_ul; Use a rule-based method to achieve the matching of the mushroom cap and the gills. First, take the midline in the width direction of the picture as the dividing line, divide the detection boxes in det_d into upper and lower parts det_dr and det_dl, make the detection boxes in the upper part det_dr correspond to the detection results det_ur of the gill images of the mushroom stick in the right-side up view, and make the detection boxes in the lower part det_dl correspond to the detection results det_ul of the gill images of the mushroom stick in the left-side up view; Determine the list of targets to be picked in the top-view mushroom cap image. The following four situations are targets to be picked: ① The mushroom cap category is white face and the gill category is not opened, ② The mushroom cap category is white face and the gill category is slightly opened, ③ The mushroom cap category is normal and the gill category is not opened, ④ The mushroom cap category is normal and the gill category is slightly opened.

8. The method for detecting and grading shiitake mushrooms based on multi-perspective fusion according to claim 7, characterized in that, In step (2) of the first step, the extracted images are preliminarily screened by an image quality assessment method, and then the images containing incomplete mushroom sticks are removed by manual inspection. Then, the image frames extracted within a few seconds at the beginning and a few seconds at the end of the original video are selected from them, and finally the shiitake mushroom target detection dataset is obtained.

9. The method for detecting and grading shiitake mushrooms based on multi-view fusion according to claim 7, characterized in that, In the second step, the shiitake mushroom target detection model is formed by pruning the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer on the basis of the basic architecture of the YOLOv8 target detection model, and only retaining the output of the detection head of the P4 layer.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the method according to any one of claims 4-9.

Images