Factory mushroom picking robot and vision-based grading picking method

Abstract

The present invention relates to a factory-based mushroom picking robot and a vision-based graded picking method. These robots address the technical issues of existing automated mushroom picking robots, such as low efficiency, low automation, easy damage to mushroom caps and sticks, and inability to accurately identify mature mushrooms. The robot comprises a mobile platform, a chassis, a lifting device, a mushroom stick removal device, and a mushroom picking device. The mushroom stick removal device is used to remove mushroom sticks from mushroom racks, and the lifting device enables the removal of mushroom sticks from different layers. The mushroom picking device comprises a robotic arm and an actuator terminal. Three depth cameras are used to capture image data of mushroom sticks. Multi-viewpoint target matching is used to identify the mushrooms to be picked and classify them according to quality, enabling graded picking. The present invention is widely used in the field of factory-based mushroom cultivation technology.

Description

Technical Field

[0001] The present invention relates to the technical field of factory-based shiitake mushroom cultivation, and in particular to a factory-based shiitake mushroom picking robot and a vision-based graded picking method. Background Art

[0002] As we all know, shiitake mushroom is an edible fungus that is rich in nutrients, including high protein, low fat, polysaccharides, multiple amino acids and multiple vitamins.

[0003] Shiitake mushroom cultivation has become factory-based. The factory environment meets the ideal growth conditions for shiitake mushrooms, including temperature, humidity, and light. Numerous spawn racks are installed within the factory. Mushroom sticks are placed on these racks and used to grow shiitake mushrooms. These racks have many layers, each with limited space, and multiple spawn sticks can be placed on each layer. The basic structure of these racks can be found in the utility model patent, "A Shiitake Mushroom Stick Cultivation Rack," with authorization publication number CN215454376U.

[0004] At present, mature shiitake mushrooms on mushroom sticks are mostly picked manually. Manual picking has the problems of high labor intensity, high labor cost, low efficiency, and missed picking.

[0005] There are a few automated picking robots on the market that replace manual harvesting. For reference, see patent applications CN116458388A and CN116671392A. However, automated harvesting is not ideal, characterized by low efficiency, a low degree of automation, damage to the mushroom caps, damage to the mushroom sticks, and inability to accurately identify mature mushrooms. In patent applications CN116458388A and CN116671392A, flexible fingers are used to grip the mushrooms and remove them from the sticks, which can easily damage the caps. Summary of the Invention

[0006] The present invention is intended to solve the technical problems of existing automatic picking robots for picking shiitake mushrooms, such as low picking efficiency, low degree of automation, easy damage to the mushroom caps, easy damage to the mushroom sticks, and inability to accurately identify mature shiitake mushrooms. It provides a factory shiitake mushroom picking robot and a vision-based grading picking method.

[0007] The factory mushroom picking robot disclosed in the present invention takes out mushroom sticks from a mushroom stick rack, takes mushroom sticks between different layers through a lifting device, picks the mushrooms that can be picked by an execution terminal, and puts the mushroom sticks back on the mushroom stick rack after picking is completed.

[0008] The present invention provides a factory mushroom picking robot, comprising a mobile platform, a chassis, a lifting device, a mushroom stick picking device and a mushroom picking device, wherein the chassis is connected to the mobile platform, the lifting device is connected to the chassis, the mushroom stick picking device is connected to the lifting device, and the mushroom picking device is connected to the chassis;

[0009] The lifting device includes a first fixed plate, a second fixed plate, a first electric cylinder, a second electric cylinder and a support plate, the first electric cylinder is fixedly connected to the first fixed plate, the second electric cylinder is fixedly connected to the second fixed plate, the first electric cylinder and the second electric cylinder are arranged vertically side by side, the first electric cylinder is provided with a first slider, the second electric cylinder is provided with a second slider, one end of the support plate is fixedly connected to the first slider on the first electric cylinder, and the other end of the support plate is fixedly connected to the second slider on the second electric cylinder; the first fixed plate is fixedly connected to the chassis, and the second fixed plate is fixedly connected to the chassis;

[0010] The mushroom stick picking device includes a first base, a first guide rail assembly, a second guide rail assembly, a Y-axis direction screw rod, a nut, a first drive motor, a first synchronous pulley, a second synchronous pulley, a first synchronous belt, a slide plate, a second base, a third guide rail assembly, an X-axis direction bidirectional screw rod, a slider 1, a slider 2, an L-shaped connecting plate 1, an L-shaped connecting plate 2, a clamping rod 1, a clamping rod 2, a nut seat 1, a nut seat 2, a second drive motor, a third synchronous pulley, a fourth synchronous pulley and a second synchronous belt. The first guide rail assembly and the second guide rail assembly are respectively fixedly connected to the first base The first guide rail assembly and the second guide rail assembly are arranged side by side, the front end of the Y-axis direction screw rod is rotatably connected to the front of the first base through a bearing, the rear end of the Y-axis direction screw rod is rotatably connected to the rear end of the first base through a bearing, the Y-axis direction screw rod is located between the first guide rail assembly and the second guide rail assembly, the first drive motor is connected to the rear end of the first base, the first synchronous pulley is connected to the output shaft of the first drive motor, the second synchronous pulley is connected to the rear end of the Y-axis direction screw rod, the first synchronous belt is connected between the first synchronous pulley and the second synchronous pulley, and the nut is connected to the Y-axis direction screw rod. The first guide rail assembly is connected to the second guide rail assembly, the second guide rail assembly is fixedly connected to the slider, the second guide rail assembly is fixedly connected to the slider, the second guide rail assembly is fixedly connected to the slider, the second guide rail assembly is fixedly connected to the slider, the left end of the bidirectional screw rod in the X-axis direction is rotatably connected to the left part of the second base through a bearing, and the right end of the bidirectional screw rod in the X-axis direction is rotatably connected to the right part of the second base through a bearing. Nut seat 1 and nut seat 2 are respectively connected to the bidirectional screw rod in the X-axis direction, slider 1 is fixedly connected to nut seat 1, and slider Block 2 is fixedly connected to nut seat 2, L-shaped connecting plate 1 is fixedly connected to slider 1, L-shaped connecting plate 2 is fixedly connected to slider 2, the second drive motor is fixedly connected to the left part of the second base, the third synchronous pulley is connected to the output shaft of the second drive motor, the fourth synchronous pulley is connected to the left end of the bidirectional screw rod in the X-axis direction, the second synchronous belt is connected between the third synchronous pulley and the fourth synchronous pulley, the rear end of the clamping rod 1 is fixedly connected to the L-shaped connecting plate 1, and the rear end of the clamping rod 2 is fixedly connected to the L-shaped connecting plate 2; the first base is fixedly connected to the support plate of the lifting device;

[0011] The mushroom picking device includes a lifting mechanism, a mechanical arm and an execution end, the lifting mechanism is fixedly connected to the chassis, the mechanical arm is connected to the lifting mechanism, and the execution end is connected to the free end of the mechanical arm; the execution end includes a support frame, a clamping drive motor, a screw rod, a screw rod nut, a connecting block, a first connecting rod, a first V-shaped connecting rod, a second connecting rod, a second V-shaped connecting rod, a first clamping block and a second clamping block, the clamping drive motor is fixedly connected to the support frame, the screw rod is fixedly connected to the output shaft of the clamping drive motor, the screw rod nut is connected to the screw rod, the connecting block is fixedly connected to the screw rod nut, the rear end of the first connecting rod is connected to the connecting The block is rotatably connected, the front end of the first connecting rod is rotatably connected to the rear end of the first V-shaped connecting rod, the middle part of the first V-shaped connecting rod is rotatably connected to the support frame, the rear end of the second connecting rod is rotatably connected to the connecting block, the front end of the second connecting rod is rotatably connected to the rear end of the second V-shaped connecting rod, the middle part of the second V-shaped connecting rod is rotatably connected to the support frame, the first connecting rod and the second connecting rod are symmetrically arranged, the first V-shaped connecting rod and the second V-shaped connecting rod are symmetrically arranged, the first clamping block is connected to the front end of the first V-shaped connecting rod, and the second clamping block is connected to the front end of the second V-shaped connecting rod; the first clamping block is provided with an arc-shaped groove, and the second clamping block is provided with an arc-shaped groove;

[0012] The support frame at the execution end is fixedly connected to the free end of the robotic arm;

[0013] The lifting mechanism is located between the first electric cylinder and the second electric cylinder of the lifting device.

[0014] Preferably, in the mushroom stick picking device, an inclined portion is provided on the inner side of the first clamping rod and an inclined portion is provided on the inner side of the second clamping rod.

[0015] Preferably, the picking device further comprises a mushroom recognition depth camera, which is connected to the support frame of the execution end.

[0016] The present invention also provides a vision-based grading and picking method using a factory mushroom picking robot. The picking device also includes a mushroom recognition depth camera, which is connected to a support frame of an execution end. The support frame is connected to a first depth camera via a first adjustable bracket, and the support frame is connected to a second depth camera via a second adjustable bracket. The vision-based grading and picking method includes the following steps:

[0017] The first step is to build a multi-view data set for training the mushroom target detection model;

[0018] Step (1) uses a depth camera to collect video data of the mushroom stick; the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed. When the depth camera moves around to the lower left side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the first video data collection of a mushroom stick is completed; then the second video is recorded, the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed. When the camera moves around to the lower right side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the second video data collection of a mushroom stick is completed; for multiple mushroom sticks, multiple videos are collected;

[0019] Step (2) is to obtain a static image of the shiitake mushrooms; extract frames from each mushroom stick video stream to obtain extracted frame images, and filter out a large number of shiitake mushroom target detection data sets from the extracted frame images, wherein the shiitake mushroom target detection data sets include images of the mushroom cap from a top-down perspective and images of the mushroom gills from a top-down perspective;

[0020] Step (3), the dataset is labeled according to six quality categories: white, normal, deformed, unopened, slightly opened, and fully opened;

[0021] The second step is to train a mushroom target detection model based on the YOLOv8 deep learning algorithm;

[0022] Step (1), model training;

[0023] Based on the basic architecture of the YOLOv8 object detection model, the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer were pruned, leaving only the detection head output of the P4 layer. This formed the mushroom object detection model, and the mushroom object detection dataset constructed in the first step was used to train the mushroom object detection model.

[0024] Step (2), matching of multi-view targets; the mushroom recognition depth camera is used to shoot the cap from a top-down perspective, and the first depth camera and the second depth camera are used to shoot the gills from a top-down perspective. The three images taken by the mushroom recognition depth camera, the first depth camera and the second depth camera are combined into a batch. The batch is input into the mushroom target detection model to obtain the mushroom detection results under three perspectives: top-down perspective, right-side upward perspective and left-side upward perspective: det_d, det_ur, det_ul; the rule-based method is used to achieve the matching of the cap and the gills. First, the midline of the width direction of the picture is used as the dividing line, and the midline of det_d is used as the dividing line. The detection frame is divided into two parts, det_dr and det_dl. Let the detection frame in the upper part det_dr correspond to the detection result det_ur of the upward gill image on the right side of the mushroom stick, and let the detection frame in the lower part det_dl correspond to the detection result det_ul of the upward gill image on the left side of the mushroom stick; determine the list of targets to be picked in the top-down cap image. The following four situations are targets to be picked: ① The cap category is white and the gill category is not opened, ② The cap category is white and the gill category is slightly opened, ③ The cap category is normal and the gill category is not opened, ④ The cap category is normal and the gill category is slightly opened;

[0025] The third step is to deploy the mushroom object detection model in the controller;

[0026] In the fourth step, the controller controls the robot arm and performs the end-action motion to perform the picking task;

[0027] Step (1): The mushroom stick picking device moves the mushroom stick to a position close to the execution end of the mushroom picking device. The controller controls the movement of the robotic arm to make the mushroom recognition depth camera reach the photo taking position above the mushroom stick. The mushroom recognition depth camera, the first depth camera, and the second depth camera collect RGB images and depth images of the mushroom stick and send them to the controller.

[0028] Step (2): the RGB image is input into the mushroom target detection model, and the target to be picked is identified through multi-view target matching;

[0029] In step (3), the controller obtains the three-dimensional coordinates of the center point of the cap of the target to be picked, and then converts the three-dimensional coordinates of the center point of the cap into position information in the base coordinate system of the robotic arm through coordinate conversion. The robotic arm action guides the execution end to move to the target to be picked, and the execution end performs a picking action to pick the target mushrooms on the mushroom stick.

[0030] Preferably, in the process of multi-view target matching, for each cap target det_up in the upper part det_dr, a list of detection frames det_ups in a range slightly larger than the width of the cap target det_up in the x-axis direction is collected in det_ur, and then the detection frame with the smallest y-axis coordinate value is selected from det_ups, which is the gill target corresponding to the cap target det_up; for each cap target det_down in the lower part det_dl, a list of detection frames det_downs in a range slightly larger than the width of the cap target det_down in the x-axis direction is collected in det_ul, and then the detection frame with the smallest y-axis coordinate value is selected from det_downs, which is the gill target corresponding to the cap target det_down.

[0031] Preferably, in step (2) of the first step, the extracted frame images are preliminarily screened by an image quality assessment method, and then images containing incomplete mushroom sticks are manually checked and removed, and then image frames extracted within a few seconds at the beginning and end of the original video are selected to finally obtain a shiitake mushroom target detection dataset.

[0032] The present invention also provides a method for detecting and grading shiitake mushrooms based on multi-view fusion, comprising the following steps:

[0033] The first step is to build a multi-view data set for training the mushroom target detection model;

[0034] Step (1) uses a depth camera to collect video data of the mushroom stick; the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed, and when the depth camera moves around to the lower left side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the first video data collection of a mushroom stick is completed; then the second video is recorded, the initial position of the depth camera is directly above the mushroom stick and shoots downward, and after starting video shooting, the depth camera is allowed to move around the side of the mushroom stick at a uniform speed, and when the camera moves around to the lower right side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, the recording is stopped, and the second video data collection of a mushroom stick is completed; for multiple mushroom sticks, multiple videos are collected;

[0035] Step (2) is to obtain a static image of the shiitake mushrooms; extract frames from each mushroom stick video stream to obtain extracted frame images, and filter out a large number of shiitake mushroom target detection data sets from the extracted frame images, wherein the shiitake mushroom target detection data sets include images of the mushroom cap from a top-down perspective and images of the mushroom gills from a top-down perspective;

[0036] Step (3), the dataset is labeled according to six quality categories: white, normal, deformed, unopened, slightly opened, and fully opened;

[0037] The second step is to train a mushroom target detection model based on the YOLOv8 deep learning algorithm;

[0038] Step (1), model training;

[0039] Use the YOLOv8 target detection model as the mushroom target detection model, and use the mushroom target detection dataset constructed in the first step above to train the mushroom target detection model;

[0040] Step (2), matching of multi-view targets; the three images of the mushroom sticks taken by the mushroom recognition depth camera, the first depth camera and the second depth camera are combined into a batch, and the batch is input into the mushroom target detection model to obtain the mushroom detection results under three perspectives: top view, right side view and left side view: det_d, det_ur, det_ul; the rule-based method is used to achieve the matching of the cap and the gills. First, the midline of the image in the width direction is used as the dividing line, and the detection frame in det_d is divided into two parts: det_dr and det_ul. det_dl, let the detection frame in the upper part det_dr correspond to the detection result det_ur of the upward-looking gill image on the right side of the mushroom stick, and let the detection frame in the lower part det_dl correspond to the detection result det_ul of the upward-looking gill image on the left side of the mushroom stick; determine the list of targets to be picked in the downward-looking cap image, the following four situations are targets to be picked: ① The cap category is white and the gill category is not opened, ② The cap category is white and the gill category is slightly opened, ③ The cap category is normal and the gill category is not opened, ④ The cap category is normal and the gill category is slightly opened.

[0041] Preferably, in step (2) of the first step, the extracted frame images are preliminarily screened by an image quality assessment method, and then images containing incomplete mushroom sticks are manually checked and removed, and then image frames extracted within a few seconds at the beginning and end of the original video are selected to finally obtain a shiitake mushroom target detection dataset.

[0042] Preferably, in the second step, the mushroom target detection model is formed by removing the detection head of the P3 layer, the P5 feature layer and the detection head of the P5 feature layer on the basis of the basic architecture of the YOLOv8 target detection model, and retaining only the detection head output of the P4 layer.

[0043] The present invention also provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, each step of any one of the above methods is implemented.

[0044] The beneficial effects of the present invention are as follows: (1) the degree of automation is high, the mushroom sticks are automatically taken out from the narrow space of each layer of the mushroom rack, the mature mushrooms on the mushroom sticks are automatically picked off, and the mushroom sticks are automatically put back into the mushroom rack; (2) the picking operation is efficient; (3) the picking operation process will not damage the mushroom caps, nor will it cause damage to the mushroom sticks; (4) the robot's action process is reliable and stable; (5) the execution terminal is small in size, and the action is flexible and reliable; (6) accurate visual recognition is used to automatically pick the mature mushrooms on the mushroom sticks, and the immature mushrooms are retained to continue growing on the mushroom sticks; (7) multi-view target matching associates the category information in the detection results of the mushroom caps and gills of the same target, and comprehensively judges whether the mushrooms meet the picking requirements; (8) the mushroom targets to be picked are divided into quality grades, and graded picking is performed.

[0045] Further features and aspects of the present invention will be clearly described in the following description of specific embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 This is an axonometric view of the factory's mushroom-picking robot;

[0047] Figure 2 yes Figure 1 The front view of the factory mushroom picking robot is shown;

[0048] Figure 3 yes Figure 1 Left view of the factory's mushroom picking robot;

[0049] Figure 4 yes Figure 1 The right side view of the factory mushroom picking robot is shown;

[0050] Figure 5 yes Figure 1 A top view of the factory's mushroom picking robot is shown;

[0051] Figure 6 yes Figure 1 A bottom view of the factory's mushroom picking robot is shown;

[0052] Figure 7 This is an axonometric view of the factory's mushroom picking robot from another perspective;

[0053] Figure 8 This is an axonometric view of the factory's mushroom picking robot from another perspective;

[0054] Figure 9 This is an axonometric view of the factory's mushroom picking robot from another perspective;

[0055] Figure 10 This is an axonometric view of the factory's mushroom picking robot from another perspective;

[0056] Figure 11 This is the axonometric drawing of the mushroom stick picking device;

[0057] Figure 12 yes Figure 11 A front view of the mushroom stick picking device shown;

[0058] Figure 13 yes Figure 11 A top view of the mushroom stick picking device shown;

[0059] Figure 14 yes Figure 11 A bottom view of the mushroom stick taking device shown;

[0060] Figure 15 yes Figure 11 A right side view of the mushroom stick taking device shown;

[0061] Figure 16 This is an axonometric view of the mushroom stick picking device;

[0062] Figure 17 This is an axonometric view of the mushroom stick picking device;

[0063] Figure 18 This is a structural diagram of the mushroom stick picking device, in which the L-shaped connecting plate 1, the slider 1 and the nut seat 1 are connected, and the L-shaped connecting plate 2, the slider 2 and the nut seat 2 are connected;

[0064] Figure 19 yes Figure 7 A partial enlarged view of the connection between the actuator end and the robotic arm of the shiitake mushroom picking device;

[0065] Figure 20 It is an axonometric view of the execution end;

[0066] Figure 21 yes Figure 20 A front view of the execution end shown;

[0067] Figure 22 yes Figure 20 a top view of the execution end shown;

[0068] Figure 23 yes Figure 20 a bottom view of the execution end shown;

[0069] Figure 24 yes Figure 20 An axonometric drawing showing another perspective of the execution end;

[0070] Figure 25 yes Figure 20 An axonometric drawing showing another perspective of the execution end;

[0071] Figure 26This is a schematic diagram of the state where two clamps at the end of the execution clamp the mushroom stem;

[0072] Figure 27 This is a schematic diagram of the state where two clamps at the end of the execution clamp the mushroom stem;

[0073] Figure 28 This is a schematic diagram of a factory mushroom picking robot positioned next to a mushroom rack, taking out mushroom sticks from the rack.

[0074] Figure 29 yes Figure 28 Side view of

[0075] Figure 30 yes Figure 28 A partial enlarged view of the middle structure;

[0076] Figure 31 yes Figure 1 In the structure shown, the mushroom stick picking device is lowered to a position close to the execution end of the mushroom picking device;

[0077] Figure 32 yes Figure 31 a front view of the structure shown;

[0078] Figure 33 yes Figure 31 a side view of the structure shown;

[0079] Figure 34 yes Figure 31 a top view of the structure shown;

[0080] Figure 35 yes Figure 31 A schematic diagram of the structure of the middle mushroom stick close to the execution end of the mushroom picking device;

[0081] Figure 36 This is a diagram showing the process where the depth camera starts shooting from directly above the mushroom stick, and the operator moves the depth camera around the side of the mushroom stick at a constant speed until it moves to the side and below the mushroom stick and shoots upwards to the gills of the shiitake mushroom, and stops recording. The diagram also shows images obtained by extracting frames from the captured video.

[0082] Figure 37 Here are six examples of quality categories for shiitake mushrooms;

[0083] Figure 38 It is a schematic diagram of the positions of the mushroom recognition depth camera, the first depth camera, and the second depth camera to capture mushroom stick images, as well as the images captured from the top-down perspective and the images captured from the bottom-up perspective;

[0084] Figure 39 It is a schematic diagram of the target matching process;

[0085] Figure 40 This is the architecture diagram of the mushroom target detection model.

[0086] Explanation of symbols in the figure:

[0087] 100. Chassis, 200. Lifting device, 201. First fixed plate, 202. Second fixed plate, 203. First electric cylinder, 204. Second electric cylinder, 205. Support plate; 300. Mushroom stick picking device, 301. First base, 302. First guide rail assembly, 303. Second guide rail assembly, 304. Y-axis direction screw, 305. First drive motor, 306. First synchronous pulley, 307. Second synchronous pulley, 308. First Synchronous belt, 309. Slide plate, 310. Second base, 311. Third guide rail assembly, 312. X-axis bidirectional screw, 313. Slider 1, 314. Slider 2, 315. L-shaped connecting plate 1, 316. L-shaped connecting plate 2, 317. Clamping rod 1, 317-1. Inclined portion, 318. Clamping rod 2, 318-1. Inclined portion, 319. Nut seat 1, 320. Nut seat 2, 321. Second drive motor, 322. Third synchronous Step pulley, 323. Fourth synchronous pulley, 324. Second synchronous belt; 400. Mushroom picking device, 401. Lifting mechanism, 402. Robotic arm, 403. Actuator, 403-1. Support frame, 403-2. Clamping drive motor, 403-3. Screw, 403-4. Screw nut, 403-5. Connecting block, 403-6. First connecting rod, 403-7. First V-shaped connecting rod, 403-8. Second connecting rod, 403-9. Second V-shaped connecting rod, 403-10. First clamping block, 403-10-1. Arc-shaped groove, 403-11. Second clamping block, 403-11-1. Arc-shaped groove, 403-12. Mushroom recognition depth camera; 500. First depth camera, 600. Second depth camera; 1. Collection basket, 2. Mushroom sticks, 2-1. Mushrooms, 3. Mushrooms, 3-1. Mushroom stems, 3-2. Mushroom caps; 4. Mushroom rack, 5. Mushroom sticks, 6. Mushroom sticks, 6-1. Mushrooms. DETAILED DESCRIPTION

[0088] The present invention will be further described in detail below with reference to the accompanying drawings using specific embodiments.

[0089] like Figure 1-10As shown, the factory mushroom picking robot includes a chassis 100, a lifting device 200, a mushroom stick picking device 300, and a mushroom picking device 400. The lifting device 200 is installed on the chassis 100, the mushroom stick picking device 300 is connected to the lifting device 200, and the mushroom picking device 400 is installed on the chassis 100. The lifting device 200 can drive the mushroom stick picking device 300 to move in the vertical direction. After the mushroom stick picking device 300 takes out a mushroom stick from the mushroom rack, it moves downward to a position close to the mushroom picking device 400 under the drive of the lifting device 200. The mushroom picking device 400 then moves to pick the mushrooms from the mushroom sticks, and the picked mushrooms are placed in a collection basket 1 (two collection baskets 1 are shown in the figure placed on the chassis 100).

[0090] The lifting device 200 includes a first fixed plate 201, a second fixed plate 202, a first electric cylinder 203, a second electric cylinder 204, and a support plate 205. The first electric cylinder 203 is fixedly connected to the first fixed plate 201, and the second electric cylinder 204 is fixedly connected to the second fixed plate 202. The first electric cylinder 203 and the second electric cylinder 204 are arranged vertically side by side. The first electric cylinder 203 is provided with a first slider, and the second electric cylinder 204 is provided with a second slider. One end of the support plate 205 is fixedly connected to the first slider on the first electric cylinder 203, and the other end of the support plate 205 is fixedly connected to the second slider on the second electric cylinder 204.

[0091] The first fixing plate 201 is fixedly connected to the chassis 100, and the second fixing plate 202 is fixedly connected to the chassis 100, thereby fixing the lifting device 200 to the chassis 100. When the first electric cylinder 203 and the second electric cylinder 204 are actuated simultaneously, the first and second sliders move the support plate 205 upward or downward, causing the support plate 205 to rise and fall in the vertical direction.

[0092] The mushroom stick taking device 300 is fixedly mounted on the support plate 205 , and the support plate 205 is lifted up and down in the vertical direction, thereby driving the mushroom stick taking device 300 to be lifted up and down in the vertical direction.

[0093] like Figure 11-18As shown, the mushroom stick taking device 300 includes a first base 301, a first guide rail assembly 302, a second guide rail assembly 303, a Y-axis direction screw rod 304, a nut, a first drive motor 305, a first synchronous pulley 306, a second synchronous pulley 307, a first synchronous belt 308, a slide plate 309, a second base 310, a third guide rail assembly 311, an X-axis direction bidirectional screw rod 312, a slider 1 313, a slider 2 314, an L-shaped connecting plate 1 315, an L-shaped connecting plate 2 316, a clamping rod 1 317, a clamping rod 2 318, a nut seat 1 319, a nut seat 2 320, a second drive motor 321, a third synchronous pulley 322, a fourth synchronous pulley 323, a second synchronous belt 324, and the first guide rail assembly 3 02. The second guide rail assembly 303 is fixedly mounted on the first base 301, the first guide rail assembly 302 and the second guide rail assembly 303 are arranged side by side, the front end of the Y-axis direction screw rod 304 is rotatably connected to the front of the first base 301 through a bearing, and the rear end of the Y-axis direction screw rod 304 is rotatably connected to the rear end of the first base 301 through a bearing, the Y-axis direction screw rod 304 is located between the first guide rail assembly 302 and the second guide rail assembly 303, the first drive motor 305 is fixedly mounted on the rear end of the first base 301, the first synchronous pulley 306 is connected to the output shaft of the first drive motor 305, the second synchronous pulley 307 is connected to the rear end of the Y-axis direction screw rod 304, and the first synchronous belt 308 is connected to the first synchronous pulley Between 306 and the second synchronous pulley 307, the nut is connected to the Y-axis direction screw rod 304, the slide 309 is fixedly connected to the nut, one side of the slide 309 is fixedly connected to the slider on the first guide rail assembly 302, and the other side of the slide 309 is fixedly connected to the slider on the second guide rail assembly 303. The second base 310 is fixedly mounted on the slide 309, and the third guide rail assembly 311 is fixedly mounted on the second base 310. The left end of the X-axis direction bidirectional screw rod 312 is rotatably connected to the left part of the second base 310 through a bearing, and the right end of the X-axis direction bidirectional screw rod 312 is rotatably connected to the right part of the second base 310 through a bearing. The nut seat 1 319 and the nut seat 2 320 are respectively connected to the X-axis direction bidirectional screw rod 312 , slider 1 313 is fixedly connected to nut seat 1 319, slider 2 314 is fixedly connected to nut seat 2 320, L-shaped connecting plate 1 315 is fixedly connected to slider 1 313, L-shaped connecting plate 2 316 is fixedly connected to slider 2 314, the second drive motor 321 is fixedly connected to the left part of the second base 310, the third synchronous pulley 322 is connected to the output shaft of the second drive motor 321, the fourth synchronous pulley 323 is connected to the left end of the bidirectional screw rod 312 in the X-axis direction, the second synchronous belt 324 is connected between the third synchronous pulley 322 and the fourth synchronous pulley 323, the rear end of the clamping rod 1 317 is fixedly connected to the L-shaped connecting plate 1 315, and the rear end of the clamping rod 2 318 is fixedly connected to the L-shaped connecting plate 2 316.

[0094] The working process of the mushroom stick picking device 300 is mainly that when the second drive motor 321 is working, the third synchronous pulley 322, the second synchronous belt 324, and the fourth synchronous pulley 323 are used to drive the X-axis direction bidirectional screw rod 312 to rotate, and the X-axis direction bidirectional screw rod 312 drives the nut seat 1 319 and the nut seat 2 320 to move closer to each other or move away from each other in the X-axis direction, thereby driving the slider 1 313 and the slider 2 314 to move closer to each other or move away from each other, and the L-shaped connecting plate 1 315 and the L-shaped connecting plate 2 316 move closer to each other or move away from each other. The first drive motor 305 rotates the Y-axis screw rod 304 through the first synchronous pulley 306, the first synchronous belt 308, and the second synchronous pulley 307. The nut connected to the Y-axis screw rod 304 drives the slide 309 to move forward or backward along the Y-axis. The slide 309 then drives the second base 310 to move forward or backward along the Y-axis, ultimately driving the first and second clamping rods 317 and 318 to move forward or backward as a whole. The mushroom stick retrieval device is stable and reliable in removing and placing mushroom sticks without damaging the mushroom sticks. The motor acts as a power source to drive the two clamping rods, and the clamping rods move with high precision. It is suitable for mushroom sticks of various sizes.

[0095] The first base 301 of the mushroom stick picking device 300 is fixedly mounted on the support plate 205 , and the support plate 205 is lifted and lowered in the vertical direction to drive the entire mushroom stick picking device 300 to move upward or downward in the vertical direction.

[0096] The mushroom picking device 400 includes a lifting mechanism 401, a robotic arm 402, and an actuator terminal 403. The lifting mechanism 401 is fixedly mounted on the chassis 100, the robotic arm 402 is connected to the lifting mechanism 401, and the actuator terminal 403 is connected to the free end of the robotic arm 402. The lifting mechanism 401 is located between the first electric cylinder 203 and the second electric cylinder 204 of the lifting mechanism 200. In other words, the mushroom picking device 400 is located between the first electric cylinder 203 and the second electric cylinder 204. The lifting mechanism 401 can drive the entire robotic arm 402 in vertical motion, adjusting the position of the robotic arm 402 and, therefore, the actuator terminal 403.

[0097] like Figure 20-25As shown, the execution end 403 includes a support frame 403-1, a clamping drive motor 403-2, a screw rod 403-3, a screw nut 403-4, a connecting block 403-5, a first connecting rod 403-6, a first V-shaped connecting rod 403-7, a second connecting rod 403-8, a second V-shaped connecting rod 403-9, a first clamping block 403-10, and a second clamping block 403-11. The clamping drive motor 403-2 is fixedly mounted on the support frame 403-1, the screw rod 403-3 is fixedly connected to the output shaft of the clamping drive motor 403-2, the screw nut 403-4 is connected and matched with the screw rod 403-3, the connecting block 403-5 is fixedly connected to the screw nut 403-4, the rear end of the first connecting rod 403-6 is rotatably connected to the connecting block 403-5, and the first connecting rod 403-6 is fixedly connected to the connecting block 403-5. The front end of the rod 403-6 is rotatably connected to the rear end of the first V-shaped link 403-7, the middle part of the first V-shaped link 403-7 is rotatably connected to the support frame 403-1, the rear end of the second link 403-8 is rotatably connected to the connecting block 403-5, the front end of the second link 403-8 is rotatably connected to the rear end of the second V-shaped link 403-9, the middle part of the second V-shaped link 403-9 is rotatably connected to the support frame 403-1, the first link 403-6 and the second link 403-8 are symmetrically arranged, the first V-shaped link 403-7 and the second V-shaped link 403-9 are symmetrically arranged, the first clamping block 403-10 is fixedly connected to the front end of the first V-shaped link 403-7, and the second clamping block 403-11 is fixedly connected to the front end of the second V-shaped link 403-9. The first clamping block 403-10 is provided with an arcuate groove 403-10-1, and the second clamping block 403-11 is provided with an arcuate groove 403-11-1. When the clamping drive motor 403-2 is working, it can drive the screw rod 403-3 to rotate, and the screw nut 403-4 moves accordingly. The screw nut 403-4 drives the connecting block 403-5 to move forward or backward. When the connecting block 403-5 moves forward, the first clamping block 403-10 and the second clamping block 403-11 can be moved closer to each other and closed. When the connecting block 403-5 moves backward, the first clamping block 403-10 and the second clamping block 403-11 can be separated from each other and opened. Figure 21 Shown in the open state).

[0098] refer to Figure 19 , the support frame 403-1 is fixed to the free end of the robot arm 402 by screws, and the first clamping block 403-10 and the second clamping block 403-11 are on the same horizontal plane.

[0099] refer to Figure 26 and 27When connecting block 403-5 moves forward, first clamping block 403-10 and second clamping block 403-11 move together to clamp the stem 3-1 of the shiitake mushroom 3 growing on the mushroom stick. Specifically, arcuate grooves 403-10-1 and arcuate grooves 403-11-1 clamp the stem 3-1. This clamping action does not damage the mushroom caps or the mushroom sticks, and the picking process is reliable and stable. The actuator terminal 40 is compact, flexible, and reliable, allowing for precise adjustment of the opening of the two clamping blocks. Powered by a motor and driven by a screw, the two clamping blocks move with high precision.

[0100] It should be noted that Figure 1-10 The specific structure of the robotic arm 402 shown in the figure is only an example. The specific structure of the robotic arm can also be any other structure that realizes the function of the robotic arm and drives the execution end 403 to any desired position in space.

[0101] like Figures 28-30 As shown, two rows of mushroom sticks are placed on each layer of the mushroom rack 4, and each row consists of 12 mushroom sticks. There is a certain distance between two adjacent mushroom sticks. The above-mentioned factory mushroom picking robot is moved to the side of the mushroom rack 4, and the mushroom stick picking device 300 is facing a mushroom stick in a certain layer on the mushroom rack 4, and then the picking operation is performed. The specific process is that first, the second drive motor 321 of the mushroom stick picking device 300 is operated to make the slider 1 313 and the slider 2 314 move away from each other, so that the clamping rod 1 317 and the clamping rod 2 318 are away from each other in the X-axis direction, and then the first drive motor 305 is operated to drive the second base 310 to move forward along the Y-axis direction, thereby making the clamping rod 1 317 and the clamping rod 2 318 move forward along the Y-axis direction (facing the target mushroom stick), and the clamping rod 1 317 and the clamping rod 2 318 are moved forward along the Y-axis direction. 18 are respectively extended into the space on both sides of the target bacteria stick (that is, the target bacteria stick is located between the clamping rod 1 317 and the clamping rod 2 318, and at this time the two clamping rods do not touch the two sides of the bacteria stick), and then the second driving motor 321 works to make the clamping rod 1 317 and the clamping rod 2 318 move closer to each other in the X-axis direction, so that the clamping rod 1 317 and the clamping rod 2 318 are clamped on both sides of the target bacteria stick, thereby clamping the target bacteria stick; then the first driving motor 305 works to drive the second base 310 to move backward along the Y-axis direction, and the clamping rod 1 317 and the clamping rod 2 318 move backward with the target bacteria stick, thereby taking the target bacteria stick 2 out of the bacteria rack (reference Figure 30 、 29 The state of the taken-out mushroom stick 2 is as shown in FIG. Figure 1 、 3 , 7, and 9.

[0102] Next, the lifting device 200 moves to drive the mushroom stick picking device 300 to move downward, and the mushroom stick picking device 300 moves downward with the mushroom stick 2 to a position close to the execution end 403 of the mushroom picking device 400. Figures 31-35 As shown, a plurality of shiitake mushrooms 2-1 grown on the mushroom stick 2 are within the operating range of the shiitake mushroom picking device 400.

[0103] Next, the mushroom picking device 400 performs a picking operation on the mushroom 2-1 on the mushroom stick 2. Specifically, the robot arm 402 moves the execution end 403 to the mushroom 2-1, and the first clamping block 403-10 and the second clamping block 403-11 of the execution end 403 are located on both sides of the mushroom 2-1 in the open state. Then, the first clamping block 403-10 and the second clamping block 403-11 move closer to each other to clamp the stem of the mushroom 2-1 (the clamping state can be referred to Figure 26 、 27 ); then, robotic arm 402 and actuator terminal 403 move together a certain distance to remove mushroom 2-1 from mushroom stick 2. As can be seen, when actuator terminal 403 grips the mushroom, it does not touch the cap, thus preventing damage to the cap; it also does not touch mushroom stick 2, thus preventing damage to the mushroom. When gripping a mature mushroom, it does not touch or damage other nearby mushrooms.

[0104] Next, the robotic arm 402 moves the execution end 403 to the top of the collection basket 1, and then the first clamping block 403-10 and the second clamping block 403-11 of the execution end 403 move away from each other to an open state, and the clamped mushrooms 2-1 are released and fall into the collection basket 1.

[0105] According to the above picking method, the mushroom picking device 400 picks off the other mushrooms on the mushroom stick 2 and puts them into the collecting basket 1. After all the shiitake mushrooms on the mushroom sticks 2 are picked off, the mushroom sticks are placed back to their original positions or other positions on the mushroom rack 4, and the lifting device 200 moves upward with the mushroom stick picking device 300, and the mushroom stick picking device 300 moves the processed mushroom sticks to the original position or other position of the mushroom rack 4, and then the first drive motor 305 works to drive the second base 310 to move forward along the Y-axis direction, thereby causing the clamping rod 1 317 and the clamping rod 2 318 to move forward along the Y-axis direction with the processed mushroom sticks, and the processed mushroom sticks are moved to the corresponding positions on the mushroom rack, and then the clamping rod 1 317 and the clamping rod 2 318 of the mushroom stick picking device 300 move away from each other, so that the clamping force applied to both sides of the processed mushroom sticks disappears, and the processed mushroom sticks are placed on the mushroom rack, and then the first drive motor 305 works to drive the clamping rod 1 317 and the clamping rod 2 318 to move backward.

[0106] Next, the robot continues to pick other mushroom sticks on the mushroom rack 4.

[0107] As can be seen, the above-described picking process is highly automated and highly efficient. The actuator terminal 403 is compact, flexible, and reliable. Driven by the robotic arm 402, the actuator terminal 403 can reach any position within the space to grasp the stem of the target shiitake mushroom.

[0108] It should be noted that a mushroom recognition depth camera 403-12 may be installed on the execution terminal 403, such as Figure 35 、 19 As shown in Figure 3, the mushroom recognition depth camera 403-12 is fixedly connected to the support frame 403-1. The mushroom recognition depth camera 403-12 collects images of the mushroom sticks and transmits them to the controller. The controller uses machine vision technology to identify the mushrooms on the mushroom sticks and controls the robot arm 402 to move the execution end 403 to the mushrooms to be picked. This achieves a more accurate movement of the execution end to the mushrooms to be picked, and a more accurate implementation of the two clamps clamping the mushroom stems, thereby picking off the target mushrooms. In order to further optimize and improve, two more depth cameras are set up, a total of three cameras, to identify the target mushrooms to be picked on the mushroom sticks. Figure 2 、 5 , 7, 8, 10, two depth cameras are mounted on the support plate 205 by an adjustable bracket, the first adjustable bracket is connected to the support plate 205, the first depth camera 500 is connected to the first adjustable bracket (the position and viewing angle of the first depth camera 500 can be adjusted by the first adjustable bracket), the second adjustable bracket is connected to the support plate 205, and the second depth camera 600 is connected to the second adjustable bracket (the position and viewing angle of the second depth camera 600 can be adjusted by the second adjustable bracket); the first depth camera 500 can specifically adopt a realsense D435 depth camera, and the second depth camera 600 can specifically adopt a realsense D435 depth camera; the specific recognition and control methods are as follows:

[0109] The picking method of shiitake mushroom detection and grading based on multi-view fusion mainly includes the following steps:

[0110] The first step is to build a multi-view data set for training the mushroom target detection model.

[0111] Step (1): Collect video data of mushroom sticks. In the mushroom cultivation factory, use the RGB sensor of the Intel RealSense D435 depth camera to collect video stream data. Adjust the resolution of the Intel RealSense D435 depth camera to 1920*1080 and the frame rate to 60fps. Figure 36, keep the camera imaging plane 300-400mm away from the surface of the mushroom growing on the mushroom stick, make the width of the image frame consistent with the height of the mushroom stick cylinder, and ensure that the field of view can include the entire mushroom stick with the mushroom growing on it. The initial position of the depth camera is just above the mushroom stick and looks down at the top of the mushroom on the mushroom stick. After starting the video shooting, the handheld depth camera moves around the side of the mushroom stick at a constant speed (according to Figure 36 Move in the direction of the dotted arrow in the middle), keep the position and size of the mushroom stick constant in the camera's field of view, and when the camera moves around to the lower left side of the mushroom stick and shoots upward to capture the clear gills of the shiitake mushroom ( Figure 36 The first segment of video data collection for a mushroom stick is now completed. Next, the second segment of video is recorded. The initial position of the depth camera is directly above the mushroom stick and it shoots the mushroom top on the mushroom stick from below. After the video shooting starts, the handheld depth camera moves around the side of the mushroom stick at a constant speed, keeping the position and size of the mushroom stick in the camera's field of view constant. When the camera moves around to the lower right side of the mushroom stick and shoots the clear gills of the mushroom upwards ( Figure 36 The second video data collection of a mushroom stick is completed. In this embodiment, a total of 502 videos are collected for 251 mushroom sticks, and the average length of each video is 8 seconds.

[0112] Step (2) is to obtain a static image of the mushroom. The frame extraction interval is set to 30, and each section of the mushroom stick video stream is frame extracted to obtain a frame-extracted image. Extracting frames from motion videos to obtain static images is prone to image blurring. In order to screen and obtain clear, high-quality images, the frame-extracted images are preliminarily automatically screened using an image quality assessment method. In this step, the image quality is defined as:

[0113] Q=αV+βS+γC

[0114] Image quality Q is the quality evaluation index of the reference-free image. In the above formula, α, β, and γ are the coefficients of each term;

[0115] The formula for the gradient variance V is:

[0116]

[0117] In the gradient variance V formula, and is the average value of the gradient, and N is the total number of pixels in the image;

[0118] The formula for image sharpness S is:

[0119]

[0120] In the image sharpness S formula, is the Laplace operator, I(x,y) is the grayscale value of the image;

[0121] The formula for contrast C is:

[0122]

[0123] In the contrast C formula, δ(i,h) is the grayscale value difference between adjacent pixels, P δ (i,j) is the distribution probability of δ.

[0124] Based on the above-mentioned quality evaluation index Q of the reference-free image, a threshold method is used to automatically perform a preliminary screening of the extracted frame images. Then, images containing incomplete mushroom sticks are manually inspected and removed to obtain clear, high-quality static images. On this basis, in order to obtain a targeted image set of the cap and gills, image frames extracted within the first 2 seconds and the last 2 seconds of the original video are selected, and finally a dataset of 820 usable shiitake mushroom target detection images is obtained. The shiitake mushroom target detection dataset mainly includes images of the cap from a top-down perspective and images of the gills from an upward perspective. It should be noted that the first 2 seconds and the last 2 seconds of the original video are only examples and are not intended to limit the scope of protection. In fact, the first few seconds and the last few seconds of the original video are all feasible.

[0125] Step (3), data annotation. In order to achieve supervised learning of deep learning models, the data set needs to be annotated. In this embodiment, according to the actual production needs of the mushroom cultivation factory, the collected images are annotated according to six quality categories: white, normal, deformed, unopened, slightly opened and fully opened. Figure 37 The example of shiitake mushroom image annotation shown in the figure has cap quality ranked from high to low as: white, normal, and deformed, and gill quality ranked from high to low as: unopened, slightly open, and fully open. The specific process can be performed using a semi-supervised auxiliary annotation tool based on the SAM model. During the actual annotation process, click the center area of ​​a shiitake mushroom target in the image to generate a complete mask for the current target. By clicking to increase or decrease the mask area, you can get an accurate mask annotation of the target. Then, select the corresponding quality category for the current target to complete the annotation of a shiitake mushroom. Since this process generates a json file containing instance segmentation annotations of all shiitake mushroom targets for each image, it is also necessary to convert the annotation file into a txt file containing only rectangular box annotations. Finally, the dataset is divided into a training set containing 700 images and a test set containing 120 images.

[0126] Using a semi-supervised assisted annotation tool based on the SAM model can improve the efficiency of manual annotation. The SAM model is short for Segment Anything Model.

[0127] The second step is to train the mushroom target detection model based on the YOLOv8 deep learning algorithm.

[0128] Step (1), model training and verification.

[0129] Considering the uniformity of the scale of the mushroom targets in this task and the actual deployment scenario with limited computing resources, this embodiment removes the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer based on the basic architecture of the YOLOv8 target detection model, and only retains the detection head output of the P4 layer, thereby forming a lightweight single-scale output model (i.e., the mushroom target detection model). Figure 39 As shown. The single-scale output model reduces the number of model parameters and optimizes the model inference speed. The mushroom target detection dataset constructed in the first step above is used to train the single-scale output model (mushroom target detection model). During the training process: CIoULoss and BCELoss are used as loss functions for the detection box regression and category prediction branches respectively, and the AdamW optimizer is used to improve the model convergence speed. After sufficient training, the mAP value of the single-scale output model (mushroom target detection model) reached 0.719 in the verification stage. At this time, the single-scale output model can be used for algorithm deployment. It should be noted that the mAP value of 0.719 during verification is an example and is not intended to limit the scope of protection of the technical solution.

[0130] Step (2), multi-view target matching. Because the determination of the quality level of shiitake mushroom targets requires the combination of visual information of the cap and gills, a shiitake mushroom recognition depth camera 403-12 that looks down to shoot the cap and two depth cameras 500 and 600 that look up to shoot the gills are deployed on the robot to simultaneously obtain two-view images of the shiitake mushrooms on the mushroom stick. Figure 38 For each mushroom stick, three images are taken and combined into a batch. This batch is then fed into the mushroom object detection model to obtain the mushroom detection results for three viewpoints: top view, right side upward view, and left side upward view: det_d, det_ur, and det_ul. The format of det_d, det_ur, and det_ul is [id, x, y, w, h, conf, cls], where id is the detection box number, x, y, w, h are the center coordinates of the detection box and its width and height, conf is the category confidence, and cls is the target category.

[0131] The purpose of multi-view target matching is to associate the category information in the cap and gill detection results of the same target, and comprehensively judge whether the mushroom meets the picking requirements. Since the installation position and posture of the mushroom recognition depth camera 403-12, the first depth camera 500 and the second depth camera 600 are controllable, the range of the mushroom stick in the field of view of the three cameras is roughly the same, and a rule-based method can be used to achieve cap and gill matching. Figure 38 In the rightmost figure, the detection frame in det_d is first divided into two parts, det_dr and det_dl, using the midline of the image in the width direction as the dividing line. The detection frame in the upper part det_dr is made to correspond to the detection result det_ur of the gill image on the right side of the mushroom stick when viewed from above, and the detection frame in the lower part det_dl is made to correspond to the detection result det_ul of the gill image on the left side of the mushroom stick when viewed from above. The specific process of target matching can be as follows: Figure 39 , take the above part det_dr as an example, for each cap target det_up, the center point of the cap target det_up is (x_d, y_d), the width is w_d, and the width of the search range F in the x-axis direction is specified to be 1.5 times w_d. Then, in det_ur, a list of detection boxes det_ups is collected whose distance between the coordinate values ​​of the gill target center point x_u and x_d is less than the width of the search range F, that is, in det_ur, a list of detection boxes in the range slightly larger than the width of the cap target det_up in the x-axis direction is collected. The detection box list det_ups is collected, and then the detection box with the smallest y-axis coordinate value is selected from det_ups, which is the gill target corresponding to the cap target det_up; similarly, for each cap target det_down in the lower part det_dl, the detection box list det_downs within a range slightly larger than the width of the cap target det_down in the x-axis direction is collected in det_ul, and then the detection box with the smallest y-axis coordinate value is selected from det_downs, which is the gill target corresponding to the cap target det_down. Finally, according to actual production needs, the list of targets to be picked in the top-view cap image is determined. Mushrooms with deformed caps or fully open gills do not meet the picking standards; mushrooms with white or normal caps and unopened or slightly open gills meet the picking standards. In other words, the following four situations are targets to be picked: ① White caps and unopened gills, ② White caps and slightly open gills, ③ Normal caps and unopened gills, ④ Normal caps and slightly open gills. They can be divided into high to low quality grades. The first situation is a special grade, the second is a superior grade, the third is a sub-excellent grade, and the fourth is an ordinary grade.

[0132] The third step is to deploy the mushroom target detection model in the controller.

[0133] In the fourth step, the controller controls the robot arm and performs the end action to perform the picking task.

[0134] In step (1), the mushroom stick picking device 300 moves downward with the mushroom stick 2 to a position close to the execution end 403 in the mushroom picking device 400. The controller controls the movement of the robotic arm so that the mushroom recognition depth camera 403-12 reaches the photo taking position above the mushroom stick. The mushroom recognition depth camera 403-12, the first depth camera 500 and the second depth camera 600 collect the RGB image and depth image of the mushroom stick and send them to the controller.

[0135] In step (2), the RGB image is input into the mushroom target detection model, and the target to be picked is identified through multi-view target matching.

[0136] In step (3), the controller obtains the three-dimensional coordinates of the center point of the cap of the target to be picked, and then converts the three-dimensional coordinates of the center point of the cap into position information in the base coordinate system of the robotic arm through coordinate conversion. The robotic arm action guides the execution end to move to the target to be picked, and the execution end performs a picking action to pick the target mushrooms on the mushroom stick.

[0137] In addition, regarding the specific structure of the mushroom stick picking device 300, in order to further optimize the clamping effect of the two clamping rods on the mushroom sticks in the mushroom stick picking device 300, as shown in FIG. Figure 11 、 17 As shown in FIG16 , the inner side of the clamping rod 1 317 is provided with an inclined portion 317-1, and the inner side of the clamping rod 2 318 is provided with an inclined portion 318-1. Figure 35 Since the outer side of the mushroom stick is arc-shaped, the inclined surface portion 317-1 and the inclined surface portion 318-1 act on both sides of the mushroom stick, which can clamp the mushroom stick more stably and reliably.

[0138] It should be noted that a mobile platform can be installed on the chassis 100. The mobile platform is a conventional product in the prior art. The mobile platform is used to move the entire robot on the ground to a position that is convenient for picking operations.

[0139] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. If inspired by this description, those skilled in the art may, without departing from the spirit of the present invention, employ other component configurations, drive mechanisms, and connection methods, and other structural embodiments and implementations similar to the present invention without inventive design, and without departing from the scope of the present invention, such embodiments shall fall within the scope of protection of the present invention.

Claims

1. A vision-based grading picking method for mushroom picking robots in factories, characterized in that: The factory mushroom picking robot includes a mobile platform, a chassis, a lifting device, a mushroom stick picking device and a mushroom picking device, wherein the chassis is connected to the mobile platform, the lifting device is connected to the chassis, the mushroom stick picking device is connected to the lifting device, and the mushroom picking device is connected to the chassis. The mushroom picking device includes a robotic arm, an execution terminal and a mushroom recognition depth camera, the mushroom recognition depth camera is connected to a support frame of the execution terminal, the lifting device includes a support plate, a first depth camera is connected to the support plate through a first adjustable bracket, and a second depth camera is connected to the support plate through a second adjustable bracket; the vision-based graded picking method includes the following steps: The first step is to build a multi-view data set for training the mushroom target detection model; Step (1), using the third depth camera to collect video data of the mushroom stick; the initial position of the third depth camera is directly above the mushroom stick and shoots downward, after starting video shooting, let the third depth camera move around the side of the mushroom stick at a uniform speed, when the third depth camera moves around to the lower left side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, stop recording, and thus complete the first video data collection of a mushroom stick; then record the second video, the initial position of the third depth camera is directly above the mushroom stick and shoots downward, after starting video shooting, let the third depth camera move around the side of the mushroom stick at a uniform speed, when the camera moves around to the lower right side of the mushroom stick and shoots upward to the gills of the shiitake mushroom, stop recording, and thus complete the second video data collection of a mushroom stick; for multiple mushroom sticks, multiple videos are collected; Step (2) is to obtain a static image of the shiitake mushrooms; extract frames from each mushroom stick video stream to obtain extracted frame images, and filter out a large number of shiitake mushroom target detection data sets from the extracted frame images, wherein the shiitake mushroom target detection data sets include images of the mushroom cap from a top-down perspective and images of the mushroom gills from a top-down perspective; Step (3), the dataset is labeled according to six quality categories: white, normal, deformed, unopened, slightly opened, and fully opened; The second step is to train a mushroom target detection model based on the YOLOv8 deep learning algorithm; Step (1), model training; Based on the basic architecture of the YOLOv8 object detection model, the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer were pruned, leaving only the detection head output of the P4 layer. This formed the mushroom object detection model, and the mushroom object detection dataset constructed in the first step was used to train the mushroom object detection model. Step (2), matching of multi-view targets; the mushroom recognition depth camera is used to shoot the cap from a top-down perspective, and the first depth camera and the second depth camera are used to shoot the gills from a top-down perspective. The three images taken by the mushroom recognition depth camera, the first depth camera and the second depth camera are combined into a batch. The batch is input into the mushroom target detection model to obtain the mushroom detection results under three perspectives: top-down perspective, right-side upward perspective and left-side upward perspective: det_d, det_ur, det_ul; the rule-based method is used to achieve the matching of the cap and the gills. First, the midline of the width direction of the picture is used as the dividing line, and the midline of det_d is used as the dividing line. The detection frame is divided into two parts, det_dr and det_dl. Let the detection frame in the upper part det_dr correspond to the detection result det_ur of the upward gill image on the right side of the mushroom stick, and let the detection frame in the lower part det_dl correspond to the detection result det_ul of the upward gill image on the left side of the mushroom stick; determine the list of targets to be picked in the top-down cap image. The following four situations are targets to be picked: ① The cap category is white and the gill category is not opened, ② The cap category is white and the gill category is slightly opened, ③ The cap category is normal and the gill category is not opened, ④ The cap category is normal and the gill category is slightly opened; The third step is to deploy the mushroom object detection model in the controller; In the fourth step, the controller controls the robot arm and performs the end-action motion to perform the picking task; Step (1): The mushroom stick picking device moves the mushroom stick to a position close to the execution end of the mushroom picking device. The controller controls the movement of the robotic arm to make the mushroom recognition depth camera reach the photo taking position above the mushroom stick. The mushroom recognition depth camera, the first depth camera, and the second depth camera collect RGB images and depth images of the mushroom stick and send them to the controller. Step (2): the RGB image is input into the mushroom target detection model, and the target to be picked is identified through multi-view target matching; In step (3), the controller obtains the three-dimensional coordinates of the center point of the cap of the target to be picked, and then converts the three-dimensional coordinates of the center point of the cap into position information in the base coordinate system of the robotic arm through coordinate conversion. The robotic arm action guides the execution end to move to the target to be picked, and the execution end performs a picking action to pick the target mushrooms on the mushroom stick.

2. The visual-based grading picking method according to claim 1, characterized in that: In the process of multi-view target matching, for each cap target det_up in the upper part det_dr, a list of detection frames det_ups in the range slightly larger than the width of the cap target det_up in the x-axis direction is collected in det_ur, and then the detection frame with the smallest y-axis coordinate value is selected from det_ups, which is the gill target corresponding to the cap target det_up; for each cap target det_down in the lower part det_dl, a list of detection frames det_downs in the range slightly larger than the width of the cap target det_down in the x-axis direction is collected in det_ul, and then the detection frame with the smallest y-axis coordinate value is selected from det_downs, which is the gill target corresponding to the cap target det_down.

3. The visual-based grading picking method according to claim 1, characterized in that: In step (2) of the first step, the extracted frame images are preliminarily screened by an image quality assessment method, and then images containing incomplete mushroom sticks are manually checked and removed. Then, image frames extracted within a few seconds at the beginning and end of the original video are selected to finally obtain a shiitake mushroom target detection dataset.

4. A method for detecting and grading shiitake mushrooms based on multi-view fusion, characterized in that: The following steps are involved: The first step is to build a multi-view data set for training the mushroom target detection model; Step (1), using the third depth camera to collect video data of the mushroom stick; the initial position of the third depth camera is directly above the mushroom stick and shoots downward, after starting video shooting, let the third depth camera move around the side of the mushroom stick at a uniform speed, when the third depth camera moves around to the lower left side of the mushroom stick and shoots upward to the gills of the shiitake mushrooms, stop recording, and thus complete the first video data collection of a mushroom stick; then record the second video, the initial position of the third depth camera is directly above the mushroom stick and shoots downward, after starting video shooting, let the third depth camera move around the side of the mushroom stick at a uniform speed, when the camera moves around to the lower right side of the mushroom stick and shoots upward to the gills of the shiitake mushrooms, stop recording, and thus complete the second video data collection of a mushroom stick; for multiple mushroom sticks, multiple videos are collected; Step (2), obtaining a static image of the mushroom; Each mushroom stick video stream is framed to obtain framed images. A large number of mushroom target detection datasets are screened from the framed images. The mushroom target detection dataset includes images of the mushroom cap from a top-down perspective and images of the mushroom gills from a top-down perspective. Step (3), the dataset is labeled according to six quality categories: white, normal, deformed, unopened, slightly opened, and fully opened; The second step is to train a mushroom target detection model based on the YOLOv8 deep learning algorithm; Step (1), model training; Use the YOLOv8 target detection model as the mushroom target detection model, and use the mushroom target detection dataset constructed in the first step above to train the mushroom target detection model; Step (2), matching of multi-view targets; the three images of the mushroom sticks taken by the mushroom recognition depth camera, the first depth camera and the second depth camera are combined into a batch, and the batch is input into the mushroom target detection model to obtain the mushroom detection results under three perspectives: top view, right side view and left side view: det_d, det_ur, det_ul; the rule-based method is used to achieve the matching of the cap and the gills. First, the midline of the image in the width direction is used as the dividing line, and the detection frame in det_d is divided into two parts: det_dr and det_ul. det_dl, let the detection frame in the upper part det_dr correspond to the detection result det_ur of the upward-looking gill image on the right side of the mushroom stick, and let the detection frame in the lower part det_dl correspond to the detection result det_ul of the upward-looking gill image on the left side of the mushroom stick; determine the list of targets to be picked in the downward-looking cap image, the following four situations are targets to be picked: ① The cap category is white and the gill category is not opened, ② The cap category is white and the gill category is slightly opened, ③ The cap category is normal and the gill category is not opened, ④ The cap category is normal and the gill category is slightly opened.

5. The method for detecting and grading shiitake mushrooms based on multi-view fusion according to claim 4, characterized in that: In step (2) of the first step, the extracted frame images are preliminarily screened by an image quality assessment method, and then images containing incomplete mushroom sticks are manually checked and removed. Then, image frames extracted within a few seconds at the beginning and end of the original video are selected to finally obtain a shiitake mushroom target detection dataset.

6. The method for detecting and grading shiitake mushrooms based on multi-view fusion according to claim 4, characterized in that: In the second step, the mushroom target detection model is formed by removing the detection head of the P3 layer, the P5 feature layer, and the detection head of the P5 feature layer on the basis of the basic architecture of the YOLOv8 target detection model, and only retaining the detection head output of the P4 layer.

7. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 6 is implemented.

Images