Object sorting method based on multi-modal perception and hardness tactile detection and arm-hand system

Abstract

An object sorting method and arm-hand system based on multi-mode perception and hardness tactile detection, wherein the method comprises: 1, constructing a training set of a fruit hardness classification network, and training the fruit hardness classification network; 2, hand-eye calibration is performed on a depth camera, and the target position perceived by the depth camera is mapped to the motion trajectory of the mechanical arm; 3, the YOLOv8 model is trained; 4, under the joint assistance of the depth camera and the trained YOLOv8 model, the mechanical arm uses a mechanical gripper to grasp the target fruit, and tactile information of a first flexible tactile sensor is transmitted to the trained fruit hardness classification network to obtain the hardness of the target fruit; 5, the fruit maturity is classified and sorted to a specified position. The present application uses visual guidance to grasp the arm-hand system, realizes full tactile maturity detection of the target fruit, improves the detection efficiency, and has good universality in life and production scenes.

Description

Technical Field

[0001] This invention relates to the field of object sorting technology, and in particular to an object sorting method and arm system based on multimodal sensing and hardness tactile detection. Background Technology

[0002] Hardness is a key indicator for evaluating material quality and manufacturing processes. Accurate hardness identification helps manufacturers ensure product consistency and reliability, thereby improving product quality and market competitiveness. Hardness data can provide valuable information about material behavior and properties, such as abrasion resistance, fatigue resistance, and deformation characteristics. This information is crucial for predicting material performance under different environmental conditions. Fruit hardness identification is an important task in the agricultural and food industries, primarily used to assess fruit ripeness, quality, and edibility.

[0003] my country is a major fruit and vegetable producer, playing a vital role in the national economy. However, approximately 30% of fruit is wasted annually due to various reasons, with a significant proportion of waste occurring during harvesting, storage, and transportation. Currently, the main technologies for fruit maturity testing rely on farmers' experience or manual methods using tools such as sugar and acidity analyzers and firmness testers to determine the fruit's maturity level according to relevant standards. Furthermore, fruit that has undergone destructive testing cannot be consumed or sold. This process results in high labor costs, low efficiency, subjectivity and instability in fruit maturity extraction and testing, and an inability to conduct large-scale real-time testing. Existing methods for predicting and classifying fruit maturity aim to address these issues, such as machine learning-based approaches. However, these methods often require specific environments, complex equipment, and extensive data processing. There is still a considerable gap between these methods and achieving truly intelligent, automated detection and maturity grading.

[0004] Sensors can be used for interaction and detection of objects. However, current development of flexible sensors has limitations such as area and sensing range. This limits the types, sizes, and weights of fruits that can be measured, especially planar fingertip flexible tactile sensors, which struggle to achieve full contact with fruits of special shapes. Summary of the Invention

[0005] This invention provides an object sorting method and arm system based on multimodal sensing and hardness tactile detection to solve the technical problems mentioned in the background art.

[0006] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0007] This invention provides an object sorting method based on multimodal sensing and hardness tactile detection, comprising the following steps:

[0008] S1. First, use the fruit maturity detection module to construct a training set for the fruit firmness classification network, and use the training set to train the fruit firmness classification network to obtain the trained fruit firmness classification network.

[0009] S2. Perform hand-eye calibration on the depth camera and map the target position perceived by the depth camera onto the movement trajectory of the robotic arm.

[0010] S3. Train the YOLOv8 model in the host computer to obtain the trained YOLOv8 model;

[0011] S4. With the joint assistance of the depth camera and the trained YOLOv8 model, the robotic arm uses the mechanical gripper at the end of the robotic arm to grasp the target fruit. The first flexible tactile sensor on the mechanical gripper senses the hardness information of the target fruit and transmits the obtained tactile information to the trained fruit hardness classification network to obtain the hardness of the target fruit.

[0012] S5. Based on the fruit types and corresponding hardness grade boundaries identified by the trained YOLOv8 model, the target fruits are classified according to their ripeness and then sorted to designated locations by a robotic arm.

[0013] Furthermore, the training set for constructing the fruit firmness classification network using the fruit ripeness detection module in step S1 specifically includes the following steps:

[0014] S11. Randomly select multiple detection points in the planar area at the top of the dual-axis moving platform. Place and clamp the test object with the corresponding hardness level on the top of the dual-axis moving platform in the fruit maturity detection module.

[0015] S12. Rotate the handwheel on the side of the test frame to drive the push-pull force gauge to move vertically until the detection end of the push-pull force gauge contacts the object being tested and the pressure reaches the set value to simulate the initial grasping state; the second flexible tactile sensor on the fruit maturity detection module is pressed and outputs the pressure data of multiple sensing units in the second flexible tactile sensor at the detection point.

[0016] S13. Then, adjust the position of the object being measured and the detection end of the push-pull force gauge through the dual-axis moving platform until the detection end of the push-pull force gauge is directly above another detection point. After the adjustment is completed, continue to drive the push-pull force gauge to move down a fixed stroke to obtain the pressure data of the second flexible tactile sensor at the detection point.

[0017] S14, repeat S13 until the pressure data of all detection points in the planar region are obtained, thus obtaining the training set of the fruit hardness classification network.

[0018] Furthermore, step S2 specifically includes the following steps:

[0019] S21. Fix the depth camera outside the robotic arm and keep it stationary, while keeping the relative position of the calibration pattern and the end of the robotic arm fixed.

[0020] S22. Move the robotic arm and use a depth camera to take multiple pictures of the calibration pattern, recording the homogeneous transformation matrix of the calibration pattern relative to the depth camera each time. And the homogeneous transformation matrix of the end effector of the robotic arm relative to the robotic arm base.

[0021] S23. Then, based on the homogeneous transformation matrix of the calibration image relative to the robotic arm base... The transformation matrix of the depth camera relative to the robotic arm base remains unchanged, and is solved using the following equation. This completes the hand-eye calibration process of the depth camera and maps the target position perceived by the depth camera onto the movement trajectory of the robotic arm.

[0022]

[0023] in, and This represents the homogeneous transformation matrix of the end effector of the robotic arm relative to the robotic arm base at three different positions after the robotic arm has been moved. and This represents the homogeneous transformation matrix of the calibration pattern relative to the depth camera at three different positions after the robotic arm is moved.

[0024] Furthermore, step S4 specifically includes the following steps:

[0025] S41. The depth camera transmits visual information to the trained YOLOv8 model in the host computer control unit. The pose estimator in the trained YOLOv8 model determines the grasping point of the target fruit based on the visual information transmitted from the depth camera.

[0026] S42. The host computer control unit generates control information based on the grasping point of the target fruit and transmits the control information to the robotic arm. The robotic arm uses the mechanical gripper at the end of the robotic arm to grasp the target fruit.

[0027] S43. The first flexible tactile sensor on the mechanical gripper senses the hardness information of the target fruit and transmits the obtained tactile information to the trained fruit hardness classification network to obtain the hardness of the target fruit.

[0028] Furthermore, the object sorting method further includes the following steps:

[0029] S6. Repeat S4 to S5 until the ripeness information of all fruits is obtained, and sort all fruits into designated locations according to the ripeness information of all fruits.

[0030] In another aspect, this invention provides an arm-and-hand system based on multimodal sensing and hardness tactile detection, which uses the above-described object sorting method for sorting. The arm-and-hand system specifically includes:

[0031] A robotic arm is mounted in the object sorting area via a robotic arm base.

[0032] Mechanical grippers, installed at the end of a robotic arm, are used to grasp fruit.

[0033] The first flexible tactile sensor is fixedly installed on the fingertip of the mechanical gripper via a base;

[0034] Depth cameras are mounted around the perimeter of the robotic arm to acquire real-time visual information about the fruit;

[0035] The host computer control unit is electrically connected to the robotic arm, the mechanical gripper, the first flexible tactile sensor, and the depth camera to control the robotic arm and the mechanical gripper. The host computer control unit has a built-in fruit hardness classification network and a YOLOv8 model.

[0036] Furthermore, the robotic arm is selected as a six-degree-of-freedom robotic arm.

[0037] Furthermore, the mechanical gripper is an articulated adaptive electric gripper, which is equipped with a force control and a programmable control unit.

[0038] Furthermore, the first flexible tactile sensor is selected as a flexible fingertip tactile sensor based on a 4×4 array of barometer sensors.

[0039] Furthermore, the fruit hardness classification network includes a 1-D convolutional layer with ReLU activation, a Dropou layer, and an average pooling layer connected in sequence.

[0040] The beneficial effects of this invention are:

[0041] 1. The object sorting method provided by the present invention utilizes the vision guidance of a host computer to guide a robotic arm to grasp and achieve full tactile ripeness detection and sorting of target fruits; it improves detection efficiency and has good universality in life and production scenarios, and can be applied to a variety of different objects, including but not limited to various ball products.

[0042] 2. The object sorting method in this invention utilizes a fruit hardness classification network, which can accurately and quickly estimate the hardness of the target fruit, and has good recognition accuracy and speed.

[0043] 3. In another aspect, the present invention also provides an arm-hand system with a high degree of intelligence and tactile feedback, which reduces the labor intensity of operators. Attached Figure Description

[0044] Figure 1 This is a flowchart of the object sorting method in this invention;

[0045] Figure 2 This is a schematic diagram of the arm-hand system in this invention;

[0046] Figure 3 This is a schematic diagram of the fruit ripeness detection module in this invention;

[0047] Figure 4 This is a schematic diagram of the fruit hardness classification network in this invention;

[0048] Figure 5 This is a schematic diagram of hand-eye calibration in this invention. Detailed Implementation

[0049] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Preferred embodiments of the invention are shown in the drawings. However, the invention can be implemented in many other different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the invention.

[0050] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to that other component.

[0051] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0052] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0053] It should also be noted that in the embodiments of this application, the same reference numerals are used to represent the same component or part. For the same part in the embodiments of this application, the reference numerals may only be used to mark one part or component as an example. It should be understood that the reference numerals are also applicable to other identical parts or components.

[0054] Reference Figure 1 This application provides an object sorting method based on multimodal sensing and hardness tactile detection, including the following steps:

[0055] S1. First, use the fruit maturity detection module to construct a training set for the fruit firmness classification network, and use the training set to train the fruit firmness classification network to obtain the trained fruit firmness classification network.

[0056] Specifically, refer to Figure 3 The fruit maturity detection module includes a test frame with a handwheel mounted on the side, a digital push-pull force gauge mounted on the test frame, and a second flexible tactile sensor. The test frame base is equipped with a dual-axis moving platform in the X and Y directions to support the second flexible tactile sensor, allowing for left-right and forward-backward movement for easy and accurate clamping. The test frame base has good stability and is suitable for indoor tabletop testing. The digital push-pull force gauge can be selected from NK / HP series push-pull force gauges to form a professional small testing machine. In this embodiment, the second flexible tactile sensor is a flexible fingertip tactile sensor using a 4×4 array of MS5805-02BA01 high-resolution barometer sensor based on FPGA.

[0057] S2. Perform hand-eye calibration on the depth camera and map the target position perceived by the depth camera onto the motion trajectory of the robotic arm; the relative pose and positional relationship between the end effector of the robotic arm (hand) and the depth camera (eye) can be determined through hand-eye calibration.

[0058] S3. Train the YOLOv8 model in the host computer to obtain the trained YOLOv8 model; the YOLOv8 model can be used to detect and identify different kinds of fruits in the fruit classification task. By training a YOLOv8 model, it can learn to accurately locate and identify fruits from the images acquired by the depth camera, and output the category and location information of each fruit;

[0059] S4. With the joint assistance of the depth camera and the trained YOLOv8 model, the robotic arm uses the mechanical gripper at the end of the robotic arm to grasp the target fruit. The first flexible tactile sensor on the mechanical gripper senses the hardness information of the target fruit and transmits the obtained tactile information to the trained fruit hardness classification network to obtain the hardness of the target fruit.

[0060] S5. Based on the fruit types and corresponding hardness grade boundaries identified by the trained YOLOv8 model, the target fruits are classified according to their ripeness and then sorted to designated locations by a robotic arm.

[0061] The object sorting method provided by this invention utilizes a host computer's vision-guided robotic arm to automatically detect and sort target fruits based on their ripeness. This improves detection efficiency and has good versatility in everyday life and production scenarios, applicable to a variety of objects, including but not limited to various spherical products.

[0062] In some embodiments, the training process of the fruit hardness classification network in S1 involves model selection, i.e., adjusting the hyperparameters of the classifier architecture. In this embodiment, the selected hyperparameters are: number of convolutional layers from 2 to 4 (filter candidates: (8, 8), (16, 16), (16, 32), (4, 8, 16), (8, 8, 8, 8, 8, 16 32), (4, 8, 16, 32)); kernel size: Ks = {8, 12, 16}; dropout percentage 20%. The object sorting method of this invention utilizes a fruit hardness classification network, which can accurately and quickly estimate the hardness of the target fruit, possessing good recognition accuracy and speed.

[0063] In some embodiments, the YOLOv8 model introduces TaskAlignedAssigner, enabling it to more intelligently assign anchor boxes to appropriate targets. This differs from traditional anchor box assignment strategies, which are typically based on fixed sizes and proportions. TaskAlignedAssigner provides a dynamic assignment mechanism, helping the model predict actual target boxes more accurately. While optimizing the anchor box strategy, the YOLOv8 model also introduces more refined feature learning strategies for targets of different sizes. By adjusting the learning focus for targets of different sizes, it ensures good detection performance for both small and large targets. This is particularly important for arm-and-hand systems that detect the freshness of fruit, as these systems need to accurately identify fruit of various sizes.

[0064] In some embodiments, the step S1 of constructing a training set for a fruit firmness classification network using a fruit maturity detection module specifically includes the following steps:

[0065] S11. Randomly select multiple detection points in the planar area at the top of the dual-axis moving platform. The test object with the corresponding hardness level is placed and clamped on the top of the dual-axis moving platform in the fruit ripeness detection module. In this embodiment, the number of detection points is 10.

[0066] S12. Rotate the handwheel on the side of the test frame to drive the push-pull force gauge to move vertically until the detection end of the push-pull force gauge contacts the object being tested and the pressure reaches the set value. In this embodiment, the set value is 2N to simulate the initial grasping state. The second flexible tactile sensor on the fruit maturity detection module is pressed and outputs the pressure data of multiple sensing units in the second flexible tactile sensor at the detection point. That is, a 1*16 vector will be collected for each test.

[0067] S13. Then, the position of the object being measured and the detection end of the push-pull force gauge are adjusted by the dual-axis moving platform until the detection end of the push-pull force gauge is directly above another detection point. After the adjustment is completed, the push-pull force gauge is driven to move downward by a fixed stroke. In this embodiment, the fixed stroke is 0.95mm, which is the minimum unit stroke of the mechanical gripper, to obtain the pressure data of the second flexible tactile sensor at the detection point.

[0068] S14, repeat S13 until the pressure data of all 10 detection points in the planar region are obtained, thus obtaining the training set of the fruit hardness classification network.

[0069] Reference Figure 5 In some embodiments, step S2 specifically includes the following steps:

[0070] S21. Fix the depth camera outside the robotic arm and keep it stationary, while keeping the relative position of the calibration pattern and the end of the robotic arm fixed.

[0071] S22. Move the robotic arm and use a depth camera to take multiple pictures of the calibration pattern, recording the homogeneous transformation matrix of the calibration pattern relative to the depth camera each time. And the homogeneous transformation matrix of the end effector of the robotic arm relative to the robotic arm base.

[0072] S23. Then, based on the homogeneous transformation matrix of the calibration image relative to the robotic arm base... The transformation matrix of the depth camera relative to the robotic arm base remains unchanged, and is solved using the following equation. This completes the hand-eye calibration process of the depth camera and maps the target position perceived by the depth camera onto the movement trajectory of the robotic arm.

[0073]

[0074] in, and This represents the homogeneous transformation matrix of the end effector of the robotic arm relative to the robotic arm base at three different positions after the robotic arm has been moved. and This represents the homogeneous transformation matrix of the calibration pattern relative to the depth camera at three different positions after the robotic arm is moved.

[0075] In this embodiment, the hand-eye calibration method uses external eye calibration, a key technology for calibrating robotic vision arm-hand systems. This involves installing a camera (depth camera) outside the robotic arm to obtain real-time image information of the area outside the hand. This method helps the robotic arm perceive and understand its surroundings, thereby achieving more precise operation and control.

[0076] Eye-to-hand calibration typically involves determining the position and orientation of the depth camera and the robotic arm base, and exchanging information with the robotic arm's control system to achieve object grasping, positioning, or manipulation. This calibration method has broad application prospects in industrial production, logistics and distribution, and service robots, and can improve the robot's autonomy and flexibility, as well as its work efficiency and accuracy.

[0077] In some embodiments, S4 specifically includes the following steps:

[0078] S41. The depth camera transmits visual information to the trained YOLOv8 model in the host computer control unit. The pose estimator in the trained YOLOv8 model determines the grasping point of the target fruit based on the visual information transmitted from the depth camera.

[0079] S42. The host computer control unit generates control information based on the grasping point of the target fruit and transmits the control information to the robotic arm. The robotic arm uses the mechanical gripper at the end of the robotic arm to grasp the target fruit.

[0080] S43. The first flexible tactile sensor on the mechanical gripper senses the hardness information of the target fruit and transmits the obtained tactile information to the trained fruit hardness classification network to obtain the hardness of the target fruit.

[0081] In some embodiments, the fruit firmness classification network estimates firmness using a firmness value estimation method, specifically implemented as follows:

[0082] First, the fruit to be tested is clamped by mechanical grippers and a set pressure is applied to the fruit. The hardness value of the fruit is estimated by the pressure on the first flexible tactile sensor after the fruit passes through the mechanical grippers for a fixed stroke.

[0083] In this embodiment, the present invention uses a hardness estimation method to estimate the hardness of fruit. The hardness estimation method refers to the Vickers hardness (symbol HV) measurement method. Vickers hardness is the most widely used test method that can be tested with any test force. In the Vickers hardness test, a diamond or hard spherical steel ball is applied to the surface of the test material with a specific load, and then the diagonal length of the resulting indentation is measured.

[0084] Referring to the Vickers hardness calculation formula, we can obtain the following relationship between fruit hardness P, contact area S, and applied pressure N:

[0085] P∝N / S

[0086] The results obtained by the hardness estimation method are mainly used for classifying fruit maturity. Only the relative hardness relationship is needed, not the precise hardness value.

[0087] Fruit firmness is one of the important indicators for measuring fruit maturity and storage quality. During the ripening and senescence process, fruit firmness gradually decreases. By measuring fruit firmness, we can understand the degree of ripeness or the extent of post-ripening softening, thereby determining the characteristics of fruit quality changes and providing correct guidance for fruit and vegetable storage.

[0088] In some embodiments, the object sorting method further includes the following steps:

[0089] S6. Repeat S4 to S5 until the ripeness information of all fruits is obtained, and sort all fruits into designated locations according to the ripeness information of all fruits.

[0090] Reference Figure 2 In another aspect, the present invention provides an arm-and-hand system based on multimodal sensing and hardness tactile detection, which uses the above-described object sorting method for sorting. The arm-and-hand system specifically includes:

[0091] A robotic arm is mounted in the object sorting area via a robotic arm base.

[0092] Mechanical grippers, installed at the end of a robotic arm, are used to grasp fruit.

[0093] The first flexible tactile sensor is fixedly installed on the fingertip of the mechanical gripper via a base;

[0094] A depth camera, mounted around the periphery of the robotic arm, acquires real-time visual information about the fruit. This depth camera is used to obtain image and depth information of the target fruit (or group of fruit). It is a special type of camera capable of capturing the depth information of every pixel in the scene, not just color information. This depth information provides crucial information such as the distance, shape, and size of objects in the scene, enabling the host computer to better understand and perceive the surrounding environment.

[0095] The host computer control unit is electrically connected to the robotic arm, the mechanical gripper, the first flexible tactile sensor, and the depth camera to control the robotic arm and the mechanical gripper. The host computer control unit has a built-in fruit hardness classification network and a YOLOv8 model.

[0096] The host computer control unit controls the movement of the robotic arm and the gripping and releasing of the target fruit by issuing commands from the host computer, and monitors the information returned by the first flexible tactile sensor and the robotic arm to achieve intelligent control.

[0097] In some embodiments, the robotic arm is a six-degree-of-freedom (DOF) robotic arm. As the support component of the arm-hand system, the six-DOF robotic arm primarily performs end-effector movement and object sorting actions. The six-DOF robotic arm is a lightweight and highly adaptable collaborative industrial robot characterized by a series of joint connections, enabling it to move flexibly and position precisely in multiple dimensions. It is ideally suited for optimizing lightweight collaborative processes such as pick-and-place operations.

[0098] In some embodiments, the mechanical gripper is a two-finger mechanical gripper, specifically an articulated adaptive electric gripper. Articulated adaptive electric grippers are suitable for collaborative robots, and their structural design is adapted to stably grasp objects of different shapes. The gripper linkage mechanism on the articulated adaptive electric gripper supports envelope adaptive grasping, making it more suitable for round, spherical, or irregularly shaped objects. It is highly adaptable to fruit grasping scenarios, improving grasping stability and achieving stable gripping and release of fruit.

[0099] The articulated adaptive electric gripper is equipped with force control and a programmable control unit. This effectively prevents damage to fruit from excessive gripping. The bottom of the articulated adaptive electric gripper connects to the end effector of a six-degree-of-freedom robotic arm via a flange, forming the overall framework of the arm-hand system hardware.

[0100] In some embodiments, the first flexible tactile sensor is a flexible fingertip tactile sensor using a 4×4 array of an MS5805-02BA01 high-resolution barometer sensor based on an FPGA. The flexible fingertip tactile sensor exhibits good performance, small error, and a wide measurement range. Its low-power design allows it to operate for extended periods with minimal power. Deployed at the fingertips of the mechanical grippers, the flexible fingertip tactile sensor ensures full contact with the fruit, acquiring tactile information and transmitting it to a fruit hardness classification network for sorting based on fruit ripeness.

[0101] The independently developed flexible fingertip tactile sensor is fixed to the fingertip of the two-finger mechanical gripper via an independently developed base. The independently developed interface of the flexible fingertip tactile sensor is simple and has good software and hardware compatibility, which can be adapted to different robot and manipulator structures to achieve tactile perception.

[0102] The tactile information acquired by the flexible fingertip tactile sensor mainly includes pressure magnitude information transmitted to the pressure gauge unit through flexible silicone and contact position information acquired through the spatial relationship of the array.

[0103] In some embodiments, the fruit firmness classification network includes a 1-D convolutional layer with ReLU activation, a Dropout layer, and an average pooling layer connected in sequence. The 1-D convolutional layer performs convolution operations to extract features from lower to higher layers. The Dropout layer prevents overfitting by making other hidden units unreliable. The average pooling layer performs feature selection, reducing the number of features and thus reducing the number of parameters. The network structure is as follows: Figure 3 As shown.

[0104] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An object sorting method based on multimodal sensing and hardness tactile detection, characterized in that, Includes the following steps: S1. First, use the fruit maturity detection module to construct a training set for the fruit firmness classification network, and use the training set to train the fruit firmness classification network to obtain the trained fruit firmness classification network. S2. Perform hand-eye calibration on the depth camera and map the target position perceived by the depth camera onto the movement trajectory of the robotic arm. S3. Train the YOLOv8 model in the host computer to obtain the trained YOLOv8 model; S4. With the joint assistance of the depth camera and the trained YOLOv8 model, the robotic arm uses the mechanical gripper at the end of the robotic arm to grasp the target fruit. The first flexible tactile sensor on the mechanical gripper senses the hardness information of the target fruit and transmits the obtained tactile information to the trained fruit hardness classification network to obtain the hardness of the target fruit. S5. Based on the fruit types and corresponding hardness grade boundaries identified by the trained YOLOv8 model, the target fruit ripeness is classified and sorted to the designated location by a robotic arm. The training set for constructing the fruit firmness classification network using the fruit maturity detection module in step S1 specifically includes the following steps: S11. Randomly select multiple detection points in the planar area at the top of the dual-axis moving platform. Place and clamp the test object with the corresponding hardness level on the top of the dual-axis moving platform in the fruit maturity detection module. S12. Rotate the handwheel on the side of the test frame to drive the push-pull force gauge to move vertically until the detection end of the push-pull force gauge contacts the object being tested and the pressure reaches the set value to simulate the initial grasping state; the second flexible tactile sensor on the fruit maturity detection module is pressed and outputs the pressure data of multiple sensing units in the second flexible tactile sensor at the detection point. S13. Then, adjust the position of the object being measured and the detection end of the push-pull force gauge through the dual-axis moving platform until the detection end of the push-pull force gauge is directly above another detection point. After the adjustment is completed, continue to drive the push-pull force gauge to move down a fixed stroke to obtain the pressure data of the second flexible tactile sensor at the detection point. S14, repeat S13 until the pressure data of all detection points in the planar region are obtained, thus obtaining the training set of the fruit hardness classification network; S2 specifically includes the following steps: S21. Fix the depth camera outside the robotic arm and keep it stationary, while keeping the relative position of the calibration pattern and the end of the robotic arm fixed. S22. Move the robotic arm and use a depth camera to take multiple pictures of the calibration pattern, recording the homogeneous transformation matrix of the calibration pattern relative to the depth camera each time. And the homogeneous transformation matrix of the end effector of the robotic arm relative to the robotic arm base. ; S23. Then, based on the homogeneous transformation matrix of the calibration image relative to the robotic arm base... The transformation matrix of the depth camera relative to the robotic arm base remains unchanged, and is solved using the following equation. This completes the hand-eye calibration process of the depth camera and maps the target position perceived by the depth camera onto the movement trajectory of the robotic arm. in, , and This represents the homogeneous transformation matrix of the end effector of the robotic arm relative to the robotic arm base at three different positions after the robotic arm has been moved. , and This represents the homogeneous transformation matrix of the calibration pattern relative to the depth camera at three different positions after the robotic arm is moved.

2. The object sorting method according to claim 1, characterized in that, S4 specifically includes the following steps: S41. The depth camera transmits visual information to the trained YOLOv8 model in the host computer control unit. The pose estimator in the trained YOLOv8 model determines the grasping point of the target fruit based on the visual information transmitted from the depth camera. S42. The host computer control unit generates control information based on the grasping point of the target fruit and transmits the control information to the robotic arm. The robotic arm uses the mechanical gripper at the end of the robotic arm to grasp the target fruit. S43. The first flexible tactile sensor on the mechanical gripper senses the hardness information of the target fruit and transmits the obtained tactile information to the trained fruit hardness classification network to obtain the hardness of the target fruit.

3. The object sorting method according to claim 1, characterized in that, It also includes the following steps: S6. Repeat S4 to S5 until the ripeness information of all fruits is obtained, and sort all fruits into designated locations according to the ripeness information of all fruits.

4. An arm-hand system, characterized in that, The object sorting system using the object sorting method according to any one of claims 1 to 3 specifically includes: A robotic arm is mounted in the object sorting area via a robotic arm base. Mechanical grippers, installed at the end of a robotic arm, are used to grasp fruit. The first flexible tactile sensor is fixedly installed on the fingertip of the mechanical gripper via a base; Depth cameras are mounted around the perimeter of the robotic arm to acquire real-time visual information about the fruit; The host computer control unit is electrically connected to the robotic arm, the mechanical gripper, the first flexible tactile sensor, and the depth camera to control the robotic arm and the mechanical gripper. The host computer control unit has a built-in fruit hardness classification network and a YOLOv8 model.

5. The arm-hand system according to claim 4, characterized in that, The robotic arm is a six-degree-of-freedom robotic arm.

6. The arm-hand system according to claim 4, characterized in that, The mechanical gripper is an articulated adaptive electric gripper, which is equipped with force control and a programmable control unit.

7. The arm-hand system according to claim 4, characterized in that, The first flexible tactile sensor is a flexible fingertip tactile sensor based on a 4×4 array of barometer sensors.

8. The arm-hand system according to any one of claims 4 to 7, characterized in that, The fruit firmness classification network comprises a 1-D convolutional layer with ReLU activation, a Dropou layer, and an average pooling layer connected in sequence.

Images