Robot intelligent fruit and vegetable sorting system and sorting method

Abstract

The invention provides a robot intelligent fruit and vegetable sorting system and method. The system comprises a camera, a mechanical arm and a central processing module. Wherein the camera is used for acquiring field fruit and vegetable images and sending the field fruit and vegetable images to the central processing module; the camera and the mechanical arm are combined to form a hand-eye calibration module; the central processing module obtains fruit and vegetable mask geometric features by using a pre-trained fruit and vegetable feature extraction model according to the field fruit and vegetable image, and obtains an actual placement posture of a target fruit and vegetable based on the features; obtaining an actual placement position of a target fruit and vegetable based on the hand-eye calibration module; and according to the actual placing posture of the target fruit and vegetable and the actual placing position of the target fruit and vegetable, a control instruction is generated, the mechanical arm is controlled to move to the target fruit and vegetable, grabbing operation is executed, and intelligent sorting of the fruit and vegetable is completed.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, specifically to a robotic intelligent fruit and vegetable sorting system and sorting method. Background Technology

[0002] With the continuous growth of market demand, the optimization of automated fruit and vegetable picking systems has become increasingly important. How to build a fruit and vegetable sorting system that improves the efficiency, accuracy, and stability of order fulfillment, reduces operating costs and manpower, and meets the ever-changing market demands has become a challenge that needs to be addressed.

[0003] The integrated intelligent sorting technology combining robotics and AI is maturing and has become a crucial technological means to address pain points in the fruit and vegetable sorting and logistics industry, demonstrating a high degree of maturity for industrial application. Therefore, the application of robotic intelligent sorting systems is an effective way to solve current challenges for enterprises in terms of competitiveness, efficiency, and cost, and it has significant practical value in improving mechanization and intelligence levels.

[0004] However, in practice, as the fruit and vegetable placement crates are stacked, significant demands are placed on the vertical operating range of the robotic arm, requiring a match between the placement height and the robotic arm's reach. Therefore, existing intelligent sorting technologies still face the following technical challenges when dealing with the intelligent sorting of fruits and vegetables: 1. Limited working space of robotic arms: The working range of traditional 6-DOF robotic arms is generally 0.6-1.4m, and the vertical movement range is even more limited. In actual fruit and vegetable sorting scenarios, the frames need to be stacked in multiple layers, with a total height of up to 1.5 meters, which makes it impossible for the robotic arm to cover the entire working area. Especially when dealing with the top or bottom fruit and vegetable frames, there are problems of not being able to reach or having difficulty grasping, which seriously affects sorting efficiency and automation level. 2. Insufficient positioning accuracy: Traditional hand-eye calibration methods are based on the assumption of rigid objects and do not fully consider the special properties of fruits and vegetables, including: ① the irregular geometric shape of the surface of fruits and vegetables, ② the natural deformation of fruits and vegetables under the action of gravity, ③ the optical interference such as reflection and shadow that may exist on the surface of fruits and vegetables. These factors lead to large calibration errors. 3. Weak posture recognition capability: Existing vision systems mostly use simple target detection methods, which can only recognize the two-dimensional position of fruits and vegetables. They are not capable of recognizing the three-dimensional posture of fruits and vegetables (such as tilt angle, orientation, etc.), which makes it impossible for the robotic arm to adjust the grasping angle and direction according to the actual posture of the fruits and vegetables. The grasping failure rate is relatively high, especially for irregularly placed fruits and vegetables, the grasping success rate is even lower. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, this invention provides a robotic intelligent fruit and vegetable sorting system and sorting method.

[0006] According to a first aspect of the present invention, a robotic intelligent fruit and vegetable sorting system is provided, comprising: a camera, a robotic arm, and a central processing module; in: The camera is used to acquire images of fruits and vegetables on site and send them to the central processing module; The camera and the robotic arm are combined to form a hand-eye calibration module; The central processing module obtains the geometric features of the fruit and vegetable mask based on the on-site fruit and vegetable images and using a pre-trained fruit and vegetable feature extraction model, and obtains the actual placement posture of the target fruit and vegetable based on the features; obtains the actual placement position of the target fruit and vegetable based on the hand-eye calibration module; and generates control commands based on the actual placement posture and actual placement position of the target fruit and vegetable to control the robotic arm to move to the target fruit and vegetable and perform a grasping operation to complete the intelligent sorting of the fruit and vegetable.

[0007] Preferably, the robotic arm includes: a column, a sorting mechanism mounted on the column and extending outward, and a sorting mechanism disposed at the end of the sorting mechanism and perpendicular to the sorting mechanism; wherein: The sorting mechanism performs multi-degree-of-freedom movements (up, down, forward, backward, and rotation) via control commands from the central processing module, transporting the sorting mechanism to the target position. The sorting mechanism is used to grab and place the target fruits and vegetables.

[0008] Preferably, the sorting mechanism adopts a crank-type robotic arm structure, including: a rotating platform, a lifting mechanism, a robotic arm rotation shaft, and an end effector push rod mechanism; wherein: The rotating platform is located at the bottom of the column and is driven to rotate by a motor. The lifting mechanism is installed in the column and connected to the rotating shaft of the robotic arm, driving the robotic arm to move up and down through up and down motion. The robotic arm's rotation axis is connected to the end effector push rod mechanism to control the planar rotation of the robotic arm; The end-effector mechanism enables the sorting mechanism to move up and down, thereby allowing for fine-tuning of its vertical position during sorting operations. The sorting mechanism is connected to the end of the end push rod mechanism. The direction of the sorting mechanism is changed by a rotary motor, and the opening and closing operation or adsorption and release operation of the sorting mechanism is controlled by a drive motor to realize the grasping and placement of target fruits and vegetables.

[0009] Preferably, the sorting mechanism includes flexible grippers and suction cups, wherein the flexible grippers are used to grip fragile fruits and vegetables, and the suction cups are used to adsorb hard fruits and vegetables.

[0010] Preferably, the central processing module uses a pre-trained fruit and vegetable feature extraction model to obtain the geometric features of the fruit and vegetable mask, including: Acquire sufficient fruit and vegetable image samples from real-world scenarios, and use image enhancement techniques to expand the fruit and vegetable image samples to obtain a sample dataset; The deep learning-based segmentation architecture model was trained using the sample dataset to obtain a fruit and vegetable feature extraction model. The fruit and vegetable feature extraction model was optimized. Images of the fruits and vegetables to be sorted and their storage boxes are obtained and used as input to the optimized fruit and vegetable segmentation model. Feature detection is performed on the fruits and vegetables in the images to obtain the geometric features of the fruit and vegetable mask.

[0011] Preferably, the central processing module obtains the actual placement posture of the target fruits and vegetables based on the features, including: Based on the obtained geometric features of the fruit and vegetable mask, the contour lines of the features are extracted, the minimum bounding rectangle of the contour lines is calculated, and the coordinates of the center point, the direction of the long side, and the direction of the short side of the bounding rectangle are obtained to obtain the required contour. The principal component analysis method is used to analyze the pixels within the contour, calculate the principal axis and secondary axis of the fruits and vegetables, and obtain the tilt angle θ of the fruits and vegetables in the image plane. Based on the area of ​​the geometric features of the fruit and vegetable mask and the area of ​​its circumscribed rectangle, the fullness ratio R=S is calculated. mask / S rect S mask Let S be the area of ​​the mask. rect The area of ​​the circumscribed rectangle; Based on the fullness ratio R and the convex hull feature of the contour, the shape category of the fruit and vegetable is determined, and the optimal gripping point position is determined. Using the tilt angle θ, the main axis direction, and the shape category of the fruit and vegetable, the actual placement posture parameters of the target fruit and vegetable are generated, including the rotation angle of the fruit and vegetable in the plane, the orientation of the major axis, and the grasping priority.

[0012] Preferably, the central processing module obtains the actual placement position of the target fruits and vegetables based on the hand-eye calibration module, including: The camera's intrinsic parameters are calibrated to obtain the camera's intrinsic parameters; The extrinsic parameters of the camera and robotic arm are calibrated to obtain the extrinsic parameter matrix; The camera captures images of real-world fruits and vegetables, and the camera's intrinsic parameters are used to transform the pixel coordinates in the fruit and vegetable images to the camera coordinate system. Using the aforementioned extrinsic parameter matrix, the coordinates in the camera coordinate system are transformed to the robotic arm coordinate system, thus completing the positioning of fruits and vegetables in reality.

[0013] Preferably, the central processing module generates control commands based on the actual placement posture and position of the target fruits and vegetables, controlling the robotic arm to move to the target fruits and vegetables and perform a grasping operation, including: Based on the actual placement of the target fruits and vegetables, the optimal height is calculated, and up-and-down movement control commands are generated to control the operation of the conveyor belt of the robotic arm so that the end of the robotic arm can move up and down to the appropriate height. Based on the actual placement posture and position of the target fruits and vegetables, the rotation angle of each degree of freedom motor is calculated using planar spatial coordinates, and the optimal gripping direction is calculated. Then, forward and backward motion and rotational motion control commands are generated to control the multi-degree-of-freedom motors of the robotic arm to drive the telescopic mechanism and rotation mechanism to achieve forward and backward motion and planar motion of the robotic arm end to reach the target position and perform the task of gripping or placing vegetables.

[0014] Preferably, calculating the optimal height based on the actual placement of the target fruits and vegetables includes: Based on the actual placement of the target fruits and vegetables, obtain the three-dimensional coordinates (x, y, y) of the vegetables in the robotic arm coordinate system. target y target , z target ); Based on the geometric characteristics of the fruit and vegetable mask, calculate the vertical height h of the vegetables. veg The height h veg The distance is determined by the top edge of the fruit and vegetable outline and the bottom of the storage box; Obtain the thickness parameter d of the end effector sorting mechanism of the robotic arm. gripper and preset safety margin d safe ; Calculate the target height h at the end of the robotic arm target =z target +h veg +d gripper +d safe ; For the target height h target Perform constraint checks to ensure h min ≤h target ≤h max , where h min and h max These are the minimum and maximum positions allowed for the robotic arm. If the position exceeds the range, it will be adjusted to the boundary value, and an alarm message will be generated. The target height h targetThe motion commands are converted into those for the lifting mechanism and the push rod mechanism, and then controlled in a closed loop via encoder feedback.

[0015] Preferably, the step of calculating the rotation angle of each degree of freedom motor using planar spatial coordinates based on the actual placement posture and position of the target fruits and vegetables, and calculating the optimal grasping direction, includes: Establish a kinematic model of the crank arm and obtain the link length parameters L1 and L2, where L1 is the link length of the lifting mechanism and L2 is the length of the rotating arm; Based on the planar coordinates (x, y) of the target fruit and vegetable in the robot arm's coordinate system target y target The following inverse kinematics solution is then performed: Calculate the target's position angle θ1 in the robot arm's coordinate system as θ1 = arctan(y target / x target ); The rotation angle θ2 of the rotating arm is calculated using the law of cosines: θ2 = arccos((d² - L1² - L2²) / (2 × L1 × L2)), where d is the distance from the target point to the column. ; Calculate the rotation angle θ3 of the rotating mechanism: θ3 = θ1 - arccos((L2² - d² - L1²) / (2 × d × L1)); Based on the actual placement parameters of the target fruits and vegetables, the optimal grasping direction is calculated as follows: Obtain the principal axis direction angle α of the fruits and vegetables; Calculate the angle β that the end gripping mechanism should rotate so that the gripping direction of the jaws is aligned with or perpendicular to the main axis of the fruit and vegetable; for long and strip-shaped fruits and vegetables, the gripping direction is perpendicular to the main axis, and β = α - θ1; ​​for spherical or block-shaped fruits and vegetables, the gripping direction is parallel to the main axis, and β = α - θ1 + 90°. Boundary constraints are applied to the rotation angles to ensure that the angles of each joint are within the allowable range, and that each rotational joint moves simultaneously to improve motion efficiency.

[0016] According to a second aspect of the present invention, a sorting method using a robotic intelligent fruit and vegetable sorting system is provided, comprising: Initialize the robotic arm position; The camera captures images of fruits and vegetables on site, and a pre-trained fruit and vegetable feature extraction model is used to obtain the geometric features of the fruit and vegetable mask. Based on these features, the actual placement posture of the target fruits and vegetables is obtained. Using a camera and a robotic arm, target fruits and vegetables are located based on hand-eye calibration to obtain their actual placement positions. Based on the actual placement posture and location of the target fruits and vegetables, control commands are generated to control the robotic arm to move to the target fruits and vegetables and perform a grasping operation, thereby completing the intelligent sorting of fruits and vegetables.

[0017] By adopting the above technical solution, the present invention has at least one of the following beneficial effects compared with the prior art: This invention employs a column-cranked robotic arm structure. Through a lifting mechanism providing a coarse-range motion and a pusher mechanism enabling a wide-range vertical movement on the column, combined with the crank arm structure, it achieves flexible operation in the horizontal plane, effectively solving the technical problem of limited vertical working space in traditional 6-DOF robotic arms. This structure allows the robotic arm's vertical movement range to reach 0.2-1.5 meters, compared to the 0.6-1.4 meters of the traditional 6-DOF robotic arm, significantly increasing the working space. This allows it to adapt to multi-layered stacked vegetable crates, meeting the needs of multi-layered operations in actual production, and significantly improving the system's applicability and automation level. Simultaneously, the crank arm structure simplifies the complexity of the robotic arm, reduces the number of joints, and lowers equipment costs and maintenance difficulty.

[0018] This invention utilizes deep geometric feature analysis of fruit and vegetable masks, including contour extraction, principal component analysis, and fullness calculation, to accurately identify the actual placement posture of fruits and vegetables in three-dimensional space, effectively solving the technical problem of weak posture recognition capability in traditional methods. This method not only acquires the position information of fruits and vegetables but also their posture parameters such as tilt angle and major axis orientation, enabling the robotic arm to adjust the grasping angle and direction according to the actual posture of the fruits and vegetables, achieving more precise grasping.

[0019] This invention establishes a precise mapping relationship from image pixel coordinates to the robotic arm coordinate system through camera intrinsic parameter calibration and camera-robotic arm extrinsic parameter calibration. Combined with geometric feature analysis of fruit and vegetable masks, it achieves accurate positioning of the actual location of the fruits and vegetables. High-precision positioning ensures that the robotic arm can accurately reach the target position, avoiding problems such as grasping deviation, missing grasp, or collisions caused by inaccurate positioning. This improves the reliability and stability of the system and reduces damage to the fruits and vegetables.

[0020] This invention optimizes the robotic arm's motion trajectory by establishing a kinematic model and employing inverse kinematics and geometric optimization methods to calculate the optimal vertical height and gripping direction in the horizontal plane, thereby significantly improving sorting efficiency. The optimal height calculation considers the actual height of the fruits and vegetables, the thickness of the end effector, and safety margins, ensuring that the gripping process avoids collisions with the fruits and vegetables or excessive height leading to unstable gripping. The optimal gripping direction calculation determines the gripping angle of the jaws based on the spindle direction of the fruits and vegetables, matching the direction of the gripping force with the shape characteristics of the fruits and vegetables, thus improving gripping stability.

[0021] This invention organically combines deep learning for fruit and vegetable recognition, machine vision for hand-eye calibration, and robotic arm motion control technologies to construct a complete intelligent fruit and vegetable sorting system. This system boasts a high level of automation and intelligence, capable of autonomously completing the entire process from fruit and vegetable recognition, posture estimation, localization, motion planning to grasping and execution, without human intervention. Attached Figure Description

[0022] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a working architecture diagram of a robotic intelligent fruit and vegetable sorting system in a preferred embodiment of the present invention.

[0023] Figure 2 This is a schematic diagram of the robotic arm structure in a preferred embodiment of the present invention.

[0024] Figure 3 This is a schematic diagram of the flexible gripper structure in a specific application example of the present invention.

[0025] In the diagram, 1 is a rotating platform used to drive the overall rotation of the robotic arm; 2 is a lifting mechanism used to drive the robotic arm up and down (for coarse adjustment during sorting); 3 is the robotic arm rotation axis used for the rotational degrees of freedom of the robotic arm; 4 is an end effector push rod mechanism used to control the up and down movement of the sorting mechanism (for fine adjustment during sorting); 5 is a sorting mechanism used for grasping fruits and vegetables; 6 is a rotary motor used to adjust the grasping posture of the sorting mechanism; 7 is a drive motor used to drive the opening and closing of the gripper; and 8 is a flexible gripper used for grasping and placing fruits and vegetables. Detailed Implementation

[0026] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

[0027] Existing intelligent sorting technologies, when faced with the intelligent sorting of fruits and vegetables, typically suffer from problems such as limited working space for robotic arms, insufficient positioning accuracy, and weak posture recognition capabilities. To address these issues, one embodiment of the present invention provides a robotic intelligent fruit and vegetable sorting system. This system organically combines deep learning for fruit and vegetable recognition, machine vision-based hand-eye calibration, and robotic arm motion control technologies, achieving a high level of automation and intelligence. It can autonomously complete the entire process from fruit and vegetable recognition, posture estimation, localization, motion planning to grasping and execution.

[0028] Specifically, such as Figure 1As shown, the robotic intelligent fruit and vegetable sorting system provided in this embodiment may include: a camera, a robotic arm, and a central processing module; in: The camera is used to acquire images of the fruits and vegetables on site and send them to the central processing module; A camera and a robotic arm are combined to form a hand-eye calibration module; The central processing module obtains the geometric features of the fruit and vegetable mask based on the on-site fruit and vegetable images and using a pre-trained fruit and vegetable feature extraction model. Based on the features, it obtains the actual placement posture of the target fruit and vegetable. Based on the hand-eye calibration module, it obtains the actual placement position of the target fruit and vegetable. Based on the actual placement posture and actual placement position of the target fruit and vegetable, it generates control commands to control the robotic arm to move to the target fruit and vegetable and perform the grasping operation to complete the intelligent sorting of the fruit and vegetable.

[0029] In some preferred embodiments, the robotic arm includes: a column, a sorting mechanism mounted on the column and extending outward, and a sorting mechanism disposed at the end of the sorting mechanism and perpendicular to the sorting mechanism; wherein: The sorting mechanism moves in multiple degrees of freedom, including up and down, forward and backward, and rotation, according to the control commands of the central processing module, and transports the sorting mechanism to the target position. The sorting mechanism is used to grab the target fruits and vegetables.

[0030] In some preferred embodiments, such as Figure 2 As shown, the sorting mechanism adopts a crank-type robotic arm structure, including: a rotating platform 1, a lifting mechanism 2, a robotic arm rotation shaft 3, and an end effector push rod mechanism 4; wherein: The rotating platform 1 is located at the bottom of the column and is driven to rotate by a motor. The lifting mechanism 2 is installed in the column and connected to the rotating shaft 3 of the robotic arm, which drives the robotic arm to move up and down through the up and down motion. The robotic arm's rotation axis 3 is connected to the end effector push rod mechanism 4 to control the planar rotation of the robotic arm; The end push rod mechanism 4 enables the sorting mechanism 5 to move up and down, thereby achieving fine adjustment of its up and down position during sorting operations; The sorting mechanism 5 is connected to the end of the end push rod mechanism 4. The direction of the sorting mechanism is changed by the rotary motor 6, and the opening and closing operation or adsorption and release operation of the sorting mechanism is controlled by the drive motor 7 to realize the grabbing and placement of target fruits and vegetables.

[0031] In some preferred embodiments, the sorting mechanism 5 includes flexible grippers 8 and suction cups. The flexible grippers are used to grasp fragile fruits and vegetables, while the suction cups are used to adsorb hard fruits and vegetables. It should be noted that the sorting mechanism 5 is not limited to these two forms, and other gripping devices can be set according to the actual application scenario.

[0032] In some preferred embodiments, the flexible gripper and the suction cup can be freely switched between each other via a rotating ring or a detachable interface, depending on the actual application scenario.

[0033] In some preferred embodiments, the central processing module, using a pre-trained fruit and vegetable feature extraction model to obtain the geometric features of the fruit and vegetable mask, may further include: We obtained a sufficient number of fruit and vegetable image samples in real-world scenarios, and then used image enhancement techniques to expand the fruit and vegetable image samples to obtain a sample dataset. A fruit and vegetable feature extraction model was obtained by training a deep learning-based segmentation architecture model using a sample dataset. Optimize the fruit and vegetable feature extraction model; Images of the fruits and vegetables to be sorted and their storage boxes are obtained and used as input to the optimized fruit and vegetable segmentation model. Feature detection is performed on the fruits and vegetables in the images to obtain the geometric features of the fruit and vegetable mask.

[0034] In some preferred embodiments, the central processing module, which obtains the actual placement posture of the target fruits and vegetables based on features, may further include: Based on the obtained geometric features of the fruit and vegetable mask, the contour lines of the features are extracted, the minimum bounding rectangle of the contour lines is calculated, and the coordinates of the center point, the direction of the long side, and the direction of the short side of the bounding rectangle are obtained to obtain the required contour. Principal component analysis is used to analyze the pixels within the contour, calculate the principal axis and secondary axis of the fruits and vegetables, and obtain the tilt angle θ of the fruits and vegetables in the image plane. Based on the geometric features of the fruit and vegetable mask, the area and the area of ​​its circumscribed rectangle are used to calculate the fullness ratio R=S. mask / S rect S mask Let S be the area of ​​the mask. rect The area of ​​the circumscribed rectangle; Based on the fullness ratio R and the convex hull feature of the contour, the shape category of fruits and vegetables is determined, and the optimal gripping point position is determined. Using the tilt angle θ, the main axis direction, and the shape category of the fruits and vegetables, the actual placement posture parameters of the target fruits and vegetables are generated, including the rotation angle of the fruits and vegetables in the plane, the orientation of the major axis, and the grasping priority.

[0035] In some preferred embodiments, the central processing module, based on the hand-eye calibration module obtaining the actual placement position of the target fruits and vegetables, may further include: The camera's intrinsic parameters are calibrated to obtain the camera's intrinsic parameters; The extrinsic parameters of the camera and robotic arm are calibrated to obtain the extrinsic parameter matrix; The camera captures images of real-world fruits and vegetables, and the camera's intrinsic parameters are used to transform the pixel coordinates in the fruit and vegetable images to the camera coordinate system. By using the extrinsic parameter matrix, the coordinates in the camera coordinate system are transformed to the robotic arm coordinate system, thus completing the positioning of fruits and vegetables in reality.

[0036] In some preferred embodiments, the central processing module generates control commands based on the actual placement posture and position of the target fruits and vegetables, controls the robotic arm to move to the target fruits and vegetables, and performs a grasping operation. It may further include: Based on the actual placement of the target fruits and vegetables, the optimal height is calculated, and up-and-down movement control commands are generated to control the operation of the conveyor belt of the robotic arm so that the end of the robotic arm can move up and down to the appropriate height. Based on the actual placement posture and position of the target fruits and vegetables, the rotation angle of each degree of freedom motor is calculated using planar spatial coordinates, and the optimal gripping direction is calculated. This generates forward and backward motion and rotational motion control commands, which control the multi-degree-of-freedom motors of the robotic arm to drive the telescopic and rotational mechanisms to achieve forward and backward motion and planar motion of the robotic arm end to reach the target position and perform the task of gripping or placing the vegetables.

[0037] In some preferred embodiments, calculating the optimal height based on the actual placement of the target fruits and vegetables may further include: Based on the actual placement of the target fruits and vegetables, obtain the three-dimensional coordinates (x, y, y) of the vegetables in the robotic arm coordinate system. target y target , z target ); Calculate the vertical height h of the vegetables based on the geometric characteristics of the fruit and vegetable mask. veg The height h veg The distance is determined by the top edge of the fruit and vegetable outline and the bottom of the storage box; Obtain the thickness parameter d of the end effector sorting mechanism of the robotic arm. gripper and preset safety margin d safe The safety margin is adaptively adjusted according to the type of fruit and vegetable. For example, it is set to 20-30mm for fragile fruits and vegetables and 10-15mm for hard fruits and vegetables. Calculate the target height h at the end of the robotic arm target =z target +h veg +d gripper +d safe ; For the target height h target Perform constraint checks to ensure h min ≤h target ≤h max , where h minand h max These are the minimum and maximum positions allowed for the robotic arm. If the position exceeds the range, it will be adjusted to the boundary value, and an alarm message will be generated. Target height h target The motion commands are converted into those for the lifting mechanism and the push rod mechanism, and then controlled in a closed loop via encoder feedback.

[0038] In some preferred embodiments, based on the actual placement posture and position of the target fruits and vegetables, the rotation angle of each degree of freedom motor is calculated using planar spatial coordinates, and the optimal gripping direction is calculated. This may further include: Establish a kinematic model of the crank arm and obtain the link length parameters L1 and L2, where L1 is the link length of the lifting mechanism and L2 is the length of the rotating arm; Based on the planar coordinates (x, y) of the target fruit and vegetable in the robot arm's coordinate system target y target The following inverse kinematics solution is then performed: Calculate the target's position angle θ1 in the robot arm's coordinate system as θ1 = arctan(y target / x target ); The rotation angle θ2 of the rotating arm is calculated using the law of cosines: θ2 = arccos((d² - L1² - L2²) / (2 × L1 × L2)), where d is the distance from the target point to the column. ; Calculate the rotation angle θ3 of the rotating mechanism: θ3 = θ1 - arccos((L2² - d² - L1²) / (2 × d × L1)); Based on the actual placement parameters of the target fruits and vegetables, the optimal grasping direction is calculated as follows: Obtain the main axis direction angle α of the fruits and vegetables (relative to the x-axis of the robotic arm coordinate system); Calculate the angle β that the end gripping mechanism should rotate to align or be perpendicular to the main axis of the fruit and vegetable. β is determined based on the shape of the fruit and vegetable. For long, narrow fruits and vegetables, the gripping direction is perpendicular to the main axis, and β = α - θ1. For spherical or blocky fruits and vegetables, the gripping direction is parallel to the main axis, and β = α - θ1 + 90°. Boundary constraints are imposed on the rotation angles to ensure that the angles of each joint are within the allowable range, and that all rotational joints move simultaneously to improve motion efficiency.

[0039] Based on the robotic intelligent fruit and vegetable sorting system provided in the above embodiments of the present invention, an embodiment of the present invention also provides a sorting method for the robotic intelligent fruit and vegetable sorting system, comprising the following steps: Initialize the robotic arm position; The system uses a camera to capture images of fruits and vegetables on-site, and uses a pre-trained fruit and vegetable feature extraction model to obtain the geometric features of the fruit and vegetable mask. Based on these features, the system obtains the actual placement posture of the target fruits and vegetables. Using a camera and a robotic arm, target fruits and vegetables are located based on hand-eye calibration to obtain their actual placement positions. Based on the actual placement posture and location of the target fruits and vegetables, control commands are generated to control the robotic arm to move to the target fruits and vegetables and perform the gripping operation, thus completing the intelligent sorting of fruits and vegetables.

[0040] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the system to implement the steps and flow of the method. That is, the embodiments in the system can be understood as preferred examples of the method, and will not be elaborated here.

[0041] The following example further illustrates the specific application of the system provided in the above embodiments of the present invention in fruit and vegetable sorting tasks.

[0042] In this specific application example, the recommended models of the central processing module and camera are shown in Table 1, and the recommended models of the robotic arm are shown in Table 2.

[0043] Table 1 Table 2 Furthermore, the system provided by the above embodiments of the present invention enables real-time fruit and vegetable sorting tasks, including sorting task planning and control and the capture of fruit and vegetable information.

[0044] The information of the 6-DOF robotic arm for fruit and vegetable grasping tasks is shown in Table 3.

[0045] Table 3 As the frames are stacked, there are significant requirements for the vertical operating range, making it impossible to guarantee that the placement height and the robotic arm can be matched.

[0046] This specific application example uses a column-mounted crank-type robotic arm as the sorting and operating mechanism to perform fruit and vegetable grabbing tasks. Its structure is as follows: Figure 3As shown. To meet the adaptive loading requirements of vegetables, a telescopic device is added to the end of the robotic arm; and a rotating device is added to the end of the telescopic device to allow the sorting mechanism to adjust the gripping posture. The belt conveyor and end effector control the up-and-down movement, while the column and the second degree of freedom motor control the rotation to achieve planar movement; a push rod-like structure is added to the end for up-and-down movement, facilitating the up-and-down adjustment of the sorting mechanism. After the camera provides a suitable target position, the belt conveyor is first run to achieve up-and-down movement to the appropriate height. Then, the rotation angle of each degree of freedom motor is calculated based on the planar spatial coordinates, driving the robotic arm to reach the target position and perform the task of gripping / placing vegetables.

[0047] Different gripping designs should be considered based on the inherent properties (geometry, hardness, etc.) of different types of vegetables: suction cups can be considered for hard vegetables (such as potatoes); pneumatic / electrically controlled flexible grippers can be considered for leafy vegetables and fragile fruits and vegetables. Based on the actual placement of the target vegetables (taking into account the geometric features of the detected vegetable masks), the optimal gripping method is estimated, and then the gripping direction of the end-of-line sorting mechanism is adjusted to ultimately achieve intelligent sorting for different types of fruits and vegetables.

[0048] The robotic intelligent fruit and vegetable sorting system and method provided in the above embodiments of the present invention adopts a column-cranked robotic arm structure. Through a lifting mechanism for coarse-range movement and a pusher mechanism for large-range vertical movement on the column, combined with the cranked arm structure, it achieves flexible operation in the horizontal plane, effectively solving the technical problem of limited vertical working space in traditional 6-DOF robotic arms. This structure allows the robotic arm's vertical movement range to reach 0.2-1.5 meters, compared to the 0.6-1.4 meters of the traditional 6-DOF robotic arm, significantly improving the working space. This enables the system to adapt to multi-layered stacked vegetable crates, meeting the operational needs of multi-layered stacking in actual production, and significantly improving the system's applicability and automation level. Simultaneously, the cranked arm structure simplifies the complexity of the robotic arm, reduces the number of joints, and lowers equipment costs and maintenance difficulty. By performing deep geometric feature analysis on the fruit and vegetable masks, including contour extraction, principal component analysis, and fullness calculation, the actual placement posture of the fruits and vegetables in three-dimensional space can be accurately identified, effectively solving the technical problem of weak posture recognition capability in traditional methods. This method not only acquires the location information of fruits and vegetables but also their tilt angle, major axis orientation, and other posture parameters. This allows the robotic arm to adjust its gripping angle and direction according to the actual posture of the fruits and vegetables, achieving more precise gripping. Through camera intrinsic parameter calibration and camera-robotic arm extrinsic parameter calibration, a precise mapping relationship from image pixel coordinates to the robotic arm coordinate system is established. Combined with geometric feature analysis of the fruit and vegetable masks, accurate positioning of the actual location of the fruits and vegetables is achieved. High-precision positioning ensures that the robotic arm can accurately reach the target position, avoiding problems such as gripping deviation, missing grip, or collision caused by inaccurate positioning, improving the reliability and stability of the system, and reducing damage to fruits and vegetables. By establishing a kinematic model of the robotic arm and using inverse kinematics and geometric optimization methods, the optimal vertical height and gripping direction in the horizontal plane are calculated, optimizing the robotic arm's motion trajectory and effectively improving sorting efficiency. The optimal height calculation takes into account the actual height of the fruits and vegetables, the thickness of the end effector, and the safety margin, ensuring that the fruits and vegetables will not collide during gripping or that the gripping will be too high, causing instability. The optimal gripping direction calculation determines the gripping angle of the jaws based on the spindle direction of the fruits and vegetables, so that the direction of the gripping force matches the shape characteristics of the fruits and vegetables, improving the stability of gripping.

[0049] Any matters not covered in the above embodiments of the present invention are well-known in the art.

[0050] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A robotic intelligent fruit and vegetable sorting system, characterized in that, include: Camera, robotic arm, and central processing module; in: The camera is used to acquire images of fruits and vegetables on site and send them to the central processing module; The camera and the robotic arm are combined to form a hand-eye calibration module; The central processing module obtains the geometric features of the fruit and vegetable mask based on the on-site fruit and vegetable images and using a pre-trained fruit and vegetable feature extraction model, and obtains the actual placement posture of the target fruit and vegetables based on the features. The actual placement position of the target fruits and vegetables is obtained based on the hand-eye calibration module; according to the actual placement posture and actual placement position of the target fruits and vegetables, control commands are generated to control the robotic arm to move to the target fruits and vegetables and perform a grasping operation to complete the intelligent sorting of fruits and vegetables.

2. The robotic intelligent fruit and vegetable sorting system according to claim 1, characterized in that, The robotic arm includes: a column, a sorting mechanism mounted on the column and extending outward, and a sorting mechanism disposed at the end of the sorting mechanism and perpendicular to the sorting mechanism; wherein: The sorting mechanism performs multi-degree-of-freedom movements (up, down, forward, backward, and rotation) via control commands from the central processing module, transporting the sorting mechanism to the target position. The sorting mechanism is used to grab and place the target fruits and vegetables.

3. The robotic intelligent fruit and vegetable sorting system according to claim 2, characterized in that, The sorting mechanism adopts a crank-type robotic arm structure, including: a rotating platform, a lifting mechanism, a robotic arm rotation axis, and an end effector push rod mechanism; wherein: The rotating platform is located at the bottom of the column and is driven to rotate by a motor. The lifting mechanism is installed in the column and connected to the rotating shaft of the robotic arm, driving the robotic arm to move up and down through up and down motion. The robotic arm's rotation axis is connected to the end effector push rod mechanism to control the planar rotation of the robotic arm; The end-effector mechanism enables the sorting mechanism to move up and down, thereby allowing for fine-tuning of its vertical position during sorting operations. The sorting mechanism is connected to the end of the end push rod mechanism. The direction of the sorting mechanism is changed by a rotary motor, and the opening and closing operation or adsorption and release operation of the sorting mechanism is controlled by a drive motor to realize the grasping and placement of target fruits and vegetables.

4. The robotic intelligent fruit and vegetable sorting system according to claim 2, characterized in that, The sorting mechanism includes flexible grippers and suction cups. The flexible grippers are used to grip fragile fruits and vegetables, and the suction cups are used to absorb hard fruits and vegetables.

5. The robotic intelligent fruit and vegetable sorting system according to claim 1, characterized in that, The central processing module uses a pre-trained fruit and vegetable feature extraction model to obtain the geometric features of the fruit and vegetable mask, including: Acquire sufficient fruit and vegetable image samples from real-world scenarios, and use image enhancement techniques to expand the fruit and vegetable image samples to obtain a sample dataset; The deep learning-based segmentation architecture model was trained using the sample dataset to obtain a fruit and vegetable feature extraction model. The fruit and vegetable feature extraction model was optimized. Images of the fruits and vegetables to be sorted and their storage boxes are obtained and used as input to the optimized fruit and vegetable segmentation model. Feature detection is performed on the fruits and vegetables in the images to obtain the geometric features of the fruit and vegetable mask.

6. The robotic intelligent fruit and vegetable sorting system according to claim 1, characterized in that, The central processing module obtains the actual placement posture of the target fruits and vegetables based on the features, including: Based on the obtained geometric features of the fruit and vegetable mask, the contour lines of the features are extracted, the minimum bounding rectangle of the contour lines is calculated, and the coordinates of the center point, the direction of the long side, and the direction of the short side of the bounding rectangle are obtained to obtain the required contour. The principal component analysis method is used to analyze the pixels within the contour, calculate the principal axis and secondary axis of the fruits and vegetables, and obtain the tilt angle θ of the fruits and vegetables in the image plane. Based on the area of ​​the geometric features of the fruit and vegetable mask and the area of ​​its circumscribed rectangle, the fullness ratio R=S is calculated. mask / S rect S mask Let S be the area of ​​the mask. rect The area of ​​the circumscribed rectangle; Based on the fullness ratio R and the convex hull feature of the contour, the shape category of the fruit and vegetable is determined, and the optimal gripping point position is determined. Using the tilt angle θ, the main axis direction, and the shape category of the fruit and vegetable, the actual placement posture parameters of the target fruit and vegetable are generated, including the rotation angle of the fruit and vegetable in the plane, the orientation of the major axis, and the grasping priority.

7. The robotic intelligent fruit and vegetable sorting system according to claim 1, characterized in that, The central processing module obtains the actual placement position of the target fruits and vegetables based on the hand-eye calibration module, including: The camera's intrinsic parameters are calibrated to obtain the camera's intrinsic parameters; The extrinsic parameters of the camera and robotic arm are calibrated to obtain the extrinsic parameter matrix; The camera captures images of real-world fruits and vegetables, and the camera's intrinsic parameters are used to transform the pixel coordinates in the fruit and vegetable images to the camera coordinate system. Using the aforementioned extrinsic parameter matrix, the coordinates in the camera coordinate system are transformed to the robotic arm coordinate system, thus completing the positioning of fruits and vegetables in reality.

8. The robotic intelligent fruit and vegetable sorting system according to claim 1, characterized in that, The central processing module generates control commands based on the actual placement posture and location of the target fruits and vegetables, controlling the robotic arm to move to the target fruits and vegetables and perform a grasping operation, including: Based on the actual placement of the target fruits and vegetables, the optimal height is calculated, and up-and-down movement control commands are generated to control the operation of the conveyor belt of the robotic arm so that the end of the robotic arm can move up and down to the appropriate height. Based on the actual placement posture and position of the target fruits and vegetables, the rotation angle of each degree of freedom motor is calculated using planar spatial coordinates, and the optimal gripping direction is calculated. Then, forward and backward motion and rotational motion control commands are generated to control the multi-degree-of-freedom motors of the robotic arm to drive the telescopic mechanism and rotation mechanism to achieve forward and backward motion and planar motion of the robotic arm end to reach the target position and perform the task of gripping or placing vegetables.

9. The robotic intelligent fruit and vegetable sorting system according to claim 8, characterized in that, The step of calculating the optimal height based on the actual placement of the target fruits and vegetables includes: Based on the actual placement of the target fruits and vegetables, obtain the three-dimensional coordinates (x, y, y) of the vegetables in the robotic arm coordinate system. target y target , z target ); Based on the geometric characteristics of the fruit and vegetable mask, calculate the vertical height h of the vegetables. veg The height h veg The distance is determined by the top edge of the fruit and vegetable outline and the bottom of the storage box; Obtain the thickness parameter d of the end effector sorting mechanism of the robotic arm. gripper and preset safety margin d safe ; Calculate the target height h at the end of the robotic arm target =z target +h veg +d gripper +d safe ; For the target height h target Perform constraint checks to ensure h min ≤h target ≤h max , where h min and h max These are the minimum and maximum positions allowed for the robotic arm. If the position exceeds the range, it will be adjusted to the boundary value, and an alarm message will be generated. The target height h target The motion commands are converted into motion commands for the lifting mechanism and push rod mechanism, and closed-loop control is achieved through encoder feedback. The step of calculating the rotation angle of each degree of freedom motor using planar spatial coordinates based on the actual placement posture and position of the target fruits and vegetables, and calculating the optimal grasping direction, includes: Establish a kinematic model of the crank arm and obtain the link length parameters L1 and L2, where L1 is the link length of the lifting mechanism and L2 is the length of the rotating arm; Based on the planar coordinates (x, y) of the target fruit and vegetable in the robot arm's coordinate system target y target The following inverse kinematics solution is then performed: Calculate the target's position angle θ1 in the robot arm's coordinate system as θ1 = arctan(y target / x target ); The rotation angle θ2 of the rotating arm is calculated using the law of cosines: θ2 = arccos((d² - L1² - L2²) / (2 × L1 × L2)), where d is the distance from the target point to the column. ; Calculate the rotation angle θ3 of the rotating mechanism: θ3 = θ1 - arccos((L2² - d² - L1²) / (2 × d × L1)); Based on the actual placement parameters of the target fruits and vegetables, the optimal grasping direction is calculated as follows: Obtain the principal axis direction angle α of the fruits and vegetables; Calculate the angle β that the end gripping mechanism should rotate so that the gripping direction of the jaws is aligned with or perpendicular to the main axis of the fruit and vegetable; for long strip-shaped fruits and vegetables, the gripping direction is perpendicular to the main axis, and β = α - θ1; ​​for spherical or block-shaped fruits and vegetables, the gripping direction is parallel to the main axis, and β = α - θ1 + 90°. Boundary constraints are applied to the rotation angles to ensure that the angles of each joint are within the allowable range.

10. A sorting method using a robotic intelligent fruit and vegetable sorting system, characterized in that, include: Initialize the robotic arm position; The camera captures images of fruits and vegetables on site, and a pre-trained fruit and vegetable feature extraction model is used to obtain the geometric features of the fruit and vegetable mask. Based on these features, the actual placement posture of the target fruits and vegetables is obtained. Using a camera and a robotic arm, target fruits and vegetables are located based on hand-eye calibration to obtain their actual placement positions. Based on the actual placement posture and location of the target fruits and vegetables, control commands are generated to control the robotic arm to move to the target fruits and vegetables and perform a grasping operation, thereby completing the intelligent sorting of fruits and vegetables.

Images