A flexible strawberry gripping device and a contact information sensing method

Abstract

This invention relates to a flexible strawberry gripping device and a contact information sensing method, belonging to the technical field of gripping devices and contact information sensing methods. It includes: an end effector and two visual-tactile sensing grippers, each comprising a fixed base, a camera, a fixed cylinder, a lens, and an elastic element. The two fixed bases are respectively fixed to two connecting bases; the two cameras are respectively fixed to the end faces of the two fixed bases; one end of each of the two fixed cylinders is fixed to the adjacent end faces of the two fixed bases; one side of each of the two lenses is sealed to the other end of each of the two fixed cylinders; and one side of each of the two elastic elements is fixed to the other side of each of the two lenses. This invention utilizes the two connecting bases of the end effector to move the two fixed bases closer together or further apart. Since the lenses and elastic elements are fixed to the adjacent ends of the two fixed cylinders fixed to the two fixed bases, the elastic elements can be used to flexibly grip the strawberry, preventing damage.

Description

Technical Field

[0001] This invention relates to the technical field of clamping devices and contact information sensing methods, and particularly to a flexible strawberry clamping device. Background Technology

[0002] Strawberries contain many important dietary components, including vitamins, minerals, folic acid, and fiber, and are a rich source of phytochemicals, primarily polyphenols. While beneficial to human health, strawberries also play a vital role as an economic crop, widely consumed fresh or processed into jams and juices. However, due to their exposed and soft fruit, strawberries are highly susceptible to damage during harvesting. Currently, most strawberry harvesting is still done manually. However, with increasing strawberry yields and labor costs, efficient and rapid automated mechanized harvesting will gradually replace manual labor. Although many teams have developed rigid grippers for strawberry harvesting, these grippers have blind spots in force control and gripping area perception, posing a significant risk of damage during strawberry handling. Therefore, two visual-tactile sensing grippers and tactile feedback are crucial for harvesting strawberries or similar fruits that are exposed, soft, and easily damaged.

[0003] Strawberries are typical soft and fragile fruits, and excessive force should not be applied when grasping them to avoid crushing or breaking them. Furthermore, the surface of a strawberry is smooth and sensitive; even slight pressure can leave indentations or cause damage. Therefore, robots need a gentle grasping method when handling strawberries, typically using flexible grippers or a force control system with sensing capabilities. Additionally, each strawberry varies in size and shape, and its surface has irregular and small bumps, so the robot needs to be able to adapt to different shapes. Moreover, strawberries grow outdoors or in greenhouses, and environmental factors such as light and humidity can affect the vision sensors or grasping accuracy; therefore, the grasping system needs to have a certain degree of environmental adaptability. Summary of the Invention

[0004] In summary, the key characteristics of strawberry grasping lie in the soft and fragile nature of the fruit, requiring robots to possess high flexibility, intelligent force control, and a precise visual perception system. Therefore, this invention aims to provide a flexible strawberry gripping device and contact information sensing method based on visual-tactile sensing. By applying visual-tactile sensing technology to the design of the strawberry gripper, and leveraging the flexibility of its contact surface elastomer and its high-resolution information extraction and feedback capabilities, the device can perceive the tactile information of the soft and fragile strawberry in real time—that is, the contact area between the flexible gripper and the strawberry, and the contact normal force. This enables stable and damage-free strawberry gripping, meeting the structural design and functional requirements of the aforementioned strawberry grasping robot, and providing a theoretical basis for flexible gripping devices in strawberry harvesting robots.

[0005] The purpose of this invention is to provide a flexible strawberry gripping device, comprising: an end effector and a visual-tactile sensing gripper. The end effector has two connecting seats that can be brought close together or moved away from each other. The visual-tactile sensing gripper consists of two oppositely arranged grippers, each including a fixed base, a camera, a fixed cylinder, a lens, and an elastic element. The two fixed bases are respectively fixed to the two connecting seats. The two cameras are respectively fixed to the end faces of the two fixed bases that are close to each other. The two fixed cylinders are respectively fitted around the outer periphery of the two cameras with a gap, and one end of each is fixed to the end face of the two fixed bases that are close to each other. One side of each of the two lenses is respectively sealed to the other end of the two fixed cylinders. One side of each of the two elastic elements is respectively fixed to the other side of the two lenses.

[0006] The beneficial effects of this invention are: by using the two connecting seats of the end effector to drive the two fixed seats to move closer or further apart, and since the other ends of the two fixed cylinders fixed on the two fixed seats are fixed to each other with lenses and elastic elements, the elastic elements can be used to flexibly clamp the strawberries and prevent them from breaking.

[0007] Based on the above technical solution, the present invention can be further improved as follows.

[0008] Furthermore, both of the aforementioned visual-tactile sensing grippers also include a light source plate, LED beads, and a light diffuser. There are multiple light source plates, which are distributed at intervals along the circumference of the fixing cylinder on the outer periphery of the camera. The surfaces of the multiple light source plates are arranged at an angle α with the mirror surface of the lens and are all fixed inside the fixing cylinder. There are multiple LED beads, which are respectively fixed on the multiple light source plates. There are multiple light diffusers, and one side of each diffuser is attached to the side of the multiple light source plates on which the LED beads are installed.

[0009] The further beneficial effect of adopting the above is that by distributing multiple light diffusers on one side of the corresponding LEDs of multiple light source boards, multiple independent point light sources can be converted into approximate surface light sources, providing a good field of view for camera observation.

[0010] Furthermore, the included angle α is 100 to 120°.

[0011] Furthermore, the multiple LEDs on the multiple light source boards are of different colors.

[0012] The further beneficial effect of adopting the above is that the multiple LEDs on the multiple light source boards can be of various colors such as red, yellow, and blue, thereby providing a multi-color uniformly distributed light source inside the fixed tube.

[0013] Furthermore, the elastic element comprises, in the direction from away from the lens to towards the lens, a silicone layer, a black dot matrix layer, a photosensitive layer, and a contact layer connected in sequence.

[0014] The further beneficial effects of the above-mentioned method are as follows: the elastic element is divided into four layers. The first layer, from the direction away from the lens to the direction closer to the lens, is mainly made of food-grade liquid transparent silicone (Shore hardness of 5 after curing). The second layer is a 10×10 black marking matrix, which is obtained by pressing the black silicone with a dot matrix needle. The third layer is a photosensitive layer made of spherical aluminum metal powder (diameter of about 1-3μm) to better receive the light signal corresponding to the deformation of the elastic body. The fourth layer, which is the contact layer with the object, is an extremely thin layer of silicone to protect the photosensitive layer without losing the deformation accuracy of the photosensitive layer. The overall thickness is mainly provided by the first layer of transparent silicone to provide deformation depth, ensuring the high resolution of the sensor in subsequent image processing. This gripper designed based on visual-tactile sensing technology can extract contact information such as the contact area and normal force distribution estimation when gripping strawberries, thus achieving stable gripping of strawberries.

[0015] Furthermore, the end effector includes a base, a servo motor, a gear, two racks, and two connecting seats. The top of the base is provided with a fixed groove and two sliding grooves. The two sliding grooves are distributed on both sides of the fixed groove and are connected to the fixed groove. The servo motor is fixed to the bottom of the fixed groove, and its output shaft is arranged along the height direction of the base. The gear is sleeved on the output shaft of the servo motor. The two racks slide in the two sliding grooves respectively and are meshed with the gear so that they move closer to or further away from each other under the drive of the gear. The two connecting seats are respectively fixed on the two racks.

[0016] In addition, a method for sensing strawberry contact information is provided, including the aforementioned flexible strawberry clamping device, the specific steps of which are as follows:

[0017] S1. Hold the strawberry stem and adjust the position of the strawberry fruit so that the strawberry fruit is between the two elastic parts;

[0018] S2. Adjust the distance between the two connecting seats in the end effector and control the stroke of the two elastic elements to clamp the strawberry;

[0019] S3. During the clamping process, the camera inside the visual-tactile sensor gripper acquires continuous deformation images of the elastic element. At the same time, based on this series of images, the contact area between the visual-tactile sensor gripper and the strawberry and the contact normal force are displayed through the information perception model.

[0020] S4. Input the real-time contact normal force on the strawberry into the feedback algorithm, and adjust the stroke of the visual-tactile sensor gripper based on the experimentally obtained strawberry force threshold.

[0021] S5. When the visual-tactile sensor gripper is in a stable state, release the strawberry stem. After releasing, the strawberry in the visual-tactile sensor gripper is in a stable state, and the strawberry is not obviously damaged after the visual-tactile sensor gripper is released, thus achieving the purpose.

[0022] Furthermore, between S3 and S2, the camera is manually focused to acquire a clear image; the light intensity of multiple light source plates is adjusted by changing the light source voltage to obtain appropriate brightness for the internal cavity. Too high a brightness will make it difficult for the camera to capture changes in the pixel value of the image, while too low a brightness will result in insufficient light and too much noise in the image captured by the camera.

[0023] Furthermore, the information perception model adopts the AFE-Unet3+ neural network model, and the training process is as follows:

[0024] S1. Connect the visual-tactile sensor gripper to the pressure head of the press (ZQ-990B multi-functional electric tensile testing machine: maximum force 100N; force resolution 0.001N; displacement resolution 0.01mm). Simultaneously, place an acrylic test mold on the table. Then, control the pressure head to move downwards and use the camera inside the visual-tactile sensor gripper to acquire contact images of the elastic element when it contacts different molds and when the pressure head applies different forces.

[0025] S2. After acquiring the original contact image, use a script to collect contact area labels;

[0026] S3. After obtaining the binary contact tag, the normal force true value is obtained by reading the force reading of the press at the contact moment. Since the sensor and the mold are in parallel contact in the experiment, the pixel-level force distribution tag can be obtained by dividing the force true value by the number of non-zero pixels of the contact tag in equation (1) to obtain the unit force value. Then, the value is assigned to each non-zero pixel in equation (2) to obtain the complete pixel-level force distribution tag.

[0027]

[0028] S4. Collect the contact images, pixel-level force distribution labels, and normal force ground truth values ​​at the same time as a set of data, and use the AFE-Unet3+ neural network model to train the collected datasets to generate an information perception model.

[0029] Furthermore, the empirical threshold test for the stress on strawberries includes the following specific procedures:

[0030] S1. First, replace the elastic element of the visual-tactile sensor gripper with an auxiliary finger that has the same silicone elastic element attached. The auxiliary finger is connected to the pressure head of the force gauge (range 2N; scale division value 0.001; accuracy ±1%).

[0031] S2. Then, holding the strawberry stem, place the strawberry between the two elastic parts, control one of the elastic parts in the visual-tactile sensor gripper to slowly apply pressure to the strawberry, observe the reading of the force gauge, and after reaching a certain value, maintain this state, release the stem, read the reading of the force gauge, so that the strawberry is in a suspended state without support.

[0032] S3. By applying different forces to strawberries, the damage to strawberries under different force ranges was obtained, and it was determined that strawberries can be in a stable suspended state without support and without any damage within the force range, thus completing the strawberry force experience threshold test. Attached Figure Description

[0033] Figure 1 This is a three-dimensional structural diagram of a flexible strawberry clamping device according to the present invention;

[0034] Figure 2 This is a schematic diagram of the internal structure of a flexible strawberry clamping device according to the present invention;

[0035] Figure 3 This is a schematic diagram of the assembly structure of the visual-tactile sensing gripper in a flexible strawberry gripping device of the present invention.

[0036] Figure 4 This is a schematic diagram of the disassembled structure of the visual-tactile sensing gripper in a flexible strawberry gripping device of the present invention.

[0037] Figure 5 This is a three-dimensional structural diagram of the strawberry force experience threshold test platform in the strawberry contact information sensing method of the present invention;

[0038] Figure 6 This is a schematic diagram of the information perception model acquisition process in the strawberry contact information perception method of the present invention;

[0039] Figure 7 This is a flowchart of the strawberry force empirical threshold test in the strawberry contact information sensing method of the present invention.

[0040] Figure 8 This is a flowchart of a flexible strawberry clamping device according to the present invention.

[0041] The attached diagram lists the components represented by each number as follows:

[0042] 1. End effector; 11. Connector; 12. Base; 121. Slide; 13. Servo motor; 14. Gear; 15. Rack; 2. Vision and tactile sensor gripper; 21. Mounting base; 22. Camera; 23. Mounting cylinder; 24. Lens; 25. Elastic element; 26. Light source board; 27. Lamp bead; 28. Light diffuser; 3. Press. Detailed Implementation

[0043] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0044] like Figure 1 and Figure 4As shown, a flexible strawberry gripping device includes: an end effector 1 and a visual-tactile sensing gripper 2. The end effector 1 has two connecting seats 11 that can be brought close to or moved away from each other. The visual-tactile sensing gripper 2 consists of two oppositely arranged grippers, each including a fixed seat 21, a camera 22, a fixed cylinder 23, a lens 24, and an elastic element 25. The two fixed seats 21 are respectively fixed to the two connecting seats 11. The two cameras 22 are respectively fixed to the end faces of the two fixed seats 21 that are close to each other. The two fixed cylinders 23 are respectively fitted around the outer periphery of the two cameras 22 with gaps, and one end of each is fixed to the end faces of the two fixed seats 21 that are close to each other. One side of each of the two lenses 24 is respectively sealed to the other end of the two fixed cylinders 23. One side of each of the two elastic elements 25 is respectively fixed to the other side of the two lenses 24.

[0045] like Figure 1 and Figure 4 As shown, in some specific embodiments, both visual-tactile sensing grippers 2 may further include a light source plate 26, LED beads 27, and a light diffuser 28. There are multiple light source plates 26, which are distributed at intervals along the circumference of the fixing cylinder 23 on the outer periphery of the camera 22. The plate surfaces of the multiple light source plates 26 are arranged at an angle α with the mirror surface of the lens 24 and are all fixed inside the fixing cylinder 23. There are multiple LED beads 27, which are respectively fixed on the multiple light source plates 26. There are multiple light diffusers 28, and one side of the diffuser is attached to the side of the multiple light source plates 26 on which the LED beads 27 are installed.

[0046] Specifically, the included angle α can be 100 to 120°.

[0047] Specifically, the multiple LEDs 27 on the multiple light source boards 26 are of different colors.

[0048] In some specific embodiments, the elastic element 25 may include a silicone layer, a black dot matrix layer, a photosensitive layer, and a contact layer connected in sequence from the direction away from the lens 24 to the direction closer to the lens 24.

[0049] like Figure 1 and Figure 2 In some specific embodiments, the end effector 1 includes a base 12, a servo motor 13, a gear 14, two racks 15, and two connecting seats 11. The top of the base 12 is provided with a fixed groove and two sliding grooves 121. The two sliding grooves 121 are distributed on both sides of the fixed groove and are in communication with the fixed groove. The servo motor 13 is fixed to the bottom of the fixed groove and its output shaft is arranged along the height direction of the base 12. The gear 14 is sleeved on the output shaft of the servo motor 13. The two racks 15 slide in the two sliding grooves 121 respectively and are meshed with the gear 14 so that they move closer to each other or further away from each other under the drive of the gear 14. The two connecting seats 11 are respectively fixed on the two racks 15.

[0050] In addition, a method for sensing strawberry contact information is provided, including a flexible strawberry gripping device, the specific steps of which are as follows:

[0051] S1. Hold the strawberry stem and adjust the position of the strawberry fruit so that the strawberry fruit is between the two elastic parts 25;

[0052] S2. Adjust the distance between the two connecting seats 11 in the end effector 1 to control the stroke of the two elastic elements 25 to clamp the strawberry;

[0053] S3. During the clamping process, the camera 22 inside the visual-tactile sensing gripper 2 acquires continuous deformation images of the elastic element 25. At the same time, for this series of images, the contact area between the visual-tactile sensing gripper 2 and the strawberry and the contact normal force are displayed through the information perception model.

[0054] S4. Input the real-time contact normal force on the strawberry into the feedback algorithm, and adjust the stroke of the visual-tactile sensor gripper 2 according to the experimentally obtained strawberry force experience threshold.

[0055] S5. When the visual-tactile sensor gripper 2 is in a stable state, release the strawberry stem. After releasing, the strawberry in the visual-tactile sensor gripper 2 is in a stable state, and the strawberry is not obviously damaged after the visual-tactile sensor gripper 2 is released, thus achieving the purpose.

[0056] In some specific embodiments, the camera is manually focused between S3 and S2 to enable it to acquire a clear image.

[0057] In some specific embodiments, the information perception model adopts the AFE-Unet3+ neural network model, and the training process is as follows:

[0058] S1. Connect the visual-tactile sensor gripper 2 to the pressure head of the press machine 3, and place an acrylic experimental mold on the table. Then control the pressure head to move downwards and use the camera 22 inside the visual-tactile sensor gripper 2 to obtain contact images of the elastic element 25 when it contacts different molds and when the pressure head applies different forces.

[0059] S2. After acquiring the original contact image, use a script to collect contact area labels;

[0060] S3. After obtaining the binary contact tag, the true value of the normal force is obtained by reading the force reading of the press 3 at the contact moment. Since the sensor and the mold are in parallel contact in the experiment, obtaining the pixel-level force distribution tag only requires dividing the true force value by the number of non-zero pixels of the contact tag to obtain the unit force value, and then assigning this value to each non-zero pixel as in formula 2 to obtain the complete pixel-level force distribution tag.

[0061]

[0062] S4. Collect the contact images, pixel-level force distribution labels, and normal force ground truth values ​​at the same time as a set of data, and use the AFE-Unet3+ neural network model to train the collected datasets to generate an information perception model.

[0063] In some specific embodiments, the strawberry stress experience threshold test includes the following operations:

[0064] S1. First, replace the elastic element 25 of the visual-tactile sensor gripper 2 with an auxiliary finger that has the same silicone elastic element 25 attached. The auxiliary finger is connected to the pressure head of the force gauge.

[0065] S2. Then, holding the strawberry stem, place the strawberry fruit between the two elastic elements 25, control one of the elastic elements 25 in the visual-tactile sensor gripper 2 to slowly apply pressure to the strawberry, observe the reading of the force gauge, and after reaching a certain value, maintain this state, release the stem, read the reading of the force gauge, so that the strawberry is in a suspended state without support.

[0066] S3. By applying different forces to strawberries, the damage to strawberries under different force ranges was obtained, and it was determined that strawberries can be in a stable suspended state without support and without any damage within the force range, thus completing the strawberry force experience threshold test.

[0067] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A flexible strawberry clamping device, characterized in that, include: An end effector (1) having two connecting seats (11) that can be brought close to or moved away from each other; A visual-tactile sensing gripper (2), comprising two grippers arranged opposite each other, each including a fixed base (21), a camera (22), a fixed cylinder (23), a lens (24), and an elastic element (25). The two fixed bases (21) are respectively fixed on the two connecting bases (11); the two cameras (22) are respectively fixed on the end faces of the two fixed bases (21) that are close to each other; the two fixed cylinders (23) are respectively fitted with gaps around the outer periphery of the two cameras (22) and one end of each is respectively fixed on the end faces of the two fixed bases (21) that are close to each other; one side of each of the two lenses (24) is respectively sealed to the other end of each of the two fixed cylinders (23); one side of each of the two elastic elements (25) is respectively fixed to the other side of each of the two lenses (24). Both of the aforementioned visual-tactile sensing grippers (2) also include a light source plate (26), an LED bead (27), and a light diffuser (28). There are multiple light source plates (26) and they are distributed at intervals along the circumference of the fixing cylinder (23) on the outer periphery of the camera (22). The plate surfaces of the multiple light source plates (26) are arranged at an angle α with the mirror surface of the lens (24) and are all fixed inside the fixing cylinder (23). There are multiple LED beads (27) and they are respectively fixed on the multiple light source plates (26). There are multiple light diffusers (28) and one side of each light diffuser is attached to the side of the multiple light source plates (26) on which the LED beads (27) are installed.

2. The strawberry flexible clamping device according to claim 1, characterized in that, The included angle α is 100 to 120°.

3. The strawberry flexible clamping device according to claim 1, characterized in that, The multiple LED beads (27) on the multiple light source plates (26) are of different colors.

4. The strawberry flexible clamping device according to claim 1, characterized in that, The elastic element (25) comprises, in the direction away from the lens (24) and towards the lens (24), a silicone layer, a black dot matrix layer, a photosensitive layer, and a contact layer connected in sequence.

5. The strawberry flexible clamping device according to claim 1, characterized in that, The end effector (1) includes a base (12), a servo motor (13), a gear (14), two racks (15), and two connecting seats (11). The top of the base (12) is provided with a fixed groove and two sliding grooves (121). The two sliding grooves (121) are distributed on both sides of the fixed groove and are connected to the fixed groove. The servo motor (13) is fixed to the bottom of the fixed groove and its output shaft is arranged along the height direction of the base (12). The gear (14) is sleeved on the output shaft of the servo motor (13). The two racks (15) slide in the two sliding grooves (121) respectively and are meshed with the gear (14) so ​​that they move closer to each other or further away from each other under the drive of the gear (14). The two connecting seats (11) are fixed on the two racks (15) respectively.

6. A method for sensing strawberry contact information, characterized in that, The specific steps of using the strawberry flexible clamping device according to any one of claims 1-5 are as follows: S1. Hold the strawberry stem and adjust the position of the strawberry fruit so that the strawberry fruit is between the two elastic elements (25); S2. Adjust the distance between the two connecting seats (11) in the end effector (1) and control the stroke of the two elastic elements (25) to clamp the strawberry; S3. During the clamping process, the continuous deformation images of the elastic element (25) are obtained by the camera (22) inside the visual-tactile sensing gripper (2). At the same time, for this series of images, the contact area between the visual-tactile sensing gripper (2) and the strawberry and the contact normal force are displayed by the information perception model. S4. Input the real-time contact normal force on the strawberry into the feedback algorithm, and adjust the stroke of the visual-tactile sensor gripper (2) according to the experimentally obtained strawberry force experience threshold. S5. When the visual-tactile sensor gripper (2) is in a stable state, release the strawberry stem. After releasing, the strawberry in the visual-tactile sensor gripper (2) is in a stable state, and the strawberry is not obviously damaged after the visual-tactile sensor gripper (2) is released, thus achieving the goal.

7. The strawberry contact information sensing method according to claim 6, characterized in that, Between S3 and S2, manually focus the camera to obtain a clear image.

8. The strawberry contact information sensing method according to claim 6, characterized in that, The information perception model uses the AFE-Unet3+ neural network model, and the training process is as follows: S1. Connect the visual-tactile sensor gripper (2) to the pressure head of the press (3), place an acrylic experimental mold on the table, and then control the pressure head to move downward. Use the camera (22) inside the visual-tactile sensor gripper (2) to obtain contact images of the elastic element (25) when it contacts different molds and when the pressure head applies different forces. S2. After acquiring the original contact image, use a script to collect contact area labels; S3. After obtaining the binary contact tag, the normal force true value is obtained by reading the force reading of the press (3) at the contact moment. Since the sensor and the mold are in parallel contact in the experiment, the pixel-level force distribution tag can be obtained by dividing the force true value by the number of non-zero pixels of the contact tag in equation (1) to obtain the unit force value. Then, the value is assigned to each non-zero pixel in equation (2) to obtain the complete pixel-level force distribution tag. S4. Collect the contact images, pixel-level force distribution labels, and normal force ground truth values ​​at the same time as a set of data, and use the AFE-Unet3+ neural network model to train the collected datasets to generate an information perception model.

9. A strawberry contact information sensing method according to claim 6, characterized in that, The empirical threshold test for stress on strawberries includes the following specific procedures: S1. First, replace the elastic element (25) of the visual-tactile sensor gripper (2) with an auxiliary finger attached with the same silicone elastic element (25). The auxiliary finger is connected to the pressure head of the force gauge. S2. Then, holding the strawberry stem, place the strawberry fruit between the two elastic elements (25), control one of the elastic elements (25) in the visual-tactile sensor gripper (2) to slowly apply pressure to the strawberry, observe the reading of the force gauge, and after reaching a certain value, maintain this state, release the fruit stem, read the reading of the force gauge, so that the strawberry is in a suspended state without support. S3. By applying different forces to strawberries, the damage to strawberries under different force ranges was obtained, and it was determined that strawberries can be in a stable suspended state without support and without any damage within the force range, thus completing the strawberry force experience threshold test.

Images