Fruit sorting robot control method and system based on machine vision

Abstract

The invention relates to the technical field of robot control, in particular to a fruit sorting robot control method and system based on machine vision, and the method comprises the steps: obtaining the initial fruit image data of a to-be-sorted fruit on a sorting conveyor belt and the current position data of a robot end effector; performing feature extraction and sorting grade identification on the initial fruit image data to obtain fruit feature data and a target sorting grade; comparing the fruit feature data with the current position data, and calculating to obtain position deviation data of the grabbing points; and correcting a preset grabbing coordinate corresponding to the target sorting grade based on the position deviation data to obtain target grabbing attitude data. According to the method, real-time image acquisition and target grabbing posture data comparison are carried out, so that the target sub-track sections can be dynamically corrected in the moving process, accurate grabbing is achieved, in this way, the robot can automatically adjust the track according to the actual position and posture of the fruit, and high-precision grabbing is ensured.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, specifically to a control method and system for a fruit sorting robot based on machine vision. Background Technology

[0002] Currently, traditional methods often rely on preset trajectories, lacking real-time feedback and dynamic adjustments. When the environment changes, the robot may not be able to adapt in time, leading to inaccurate grasping or errors. Moreover, traditional methods may not have a smooth trajectory design, resulting in stiff robot movements, which are prone to vibration or unnecessary sharp turns, affecting the stability of grasping.

[0003] Furthermore, traditional methods generally lack real-time correction capabilities when encountering posture deviations. This can lead to the robot being unable to adjust quickly during grasping, thus affecting grasping efficiency. Moreover, traditional methods may not be able to determine whether grasping requirements are met by judging sensor data in real time, resulting in invalid or unstable grasping, which affects the accuracy of fruit sorting. In addition, in traditional methods, the speed and movement of robot joints are usually not precisely controlled, which may lead to excessively fast or slow movements, increasing system wear and affecting service life. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: a machine vision-based fruit sorting robot control method, comprising:

[0005] Acquire initial fruit image data of the fruit to be sorted on the sorting conveyor belt and the current position data of the robot's end effector; perform feature extraction and sorting level recognition on the initial fruit image data to obtain fruit feature data and target sorting level;

[0006] The fruit feature data is compared with the current location data to calculate the position deviation data of the grasping point; the preset grasping coordinates corresponding to the target sorting level are corrected based on the position deviation data to obtain the target grasping posture data;

[0007] The target grasping posture data is input into the robot kinematics model, and the data of multiple intermediate posture points in the joint space are obtained by combining the preset velocity constraint model.

[0008] Based on the target grasping posture data, the multiple intermediate posture point data, and the preset trajectory smoothing index, a continuous B-spline curve segment is constructed and interpolation is performed to generate a smooth target sub-trajectory segment.

[0009] Based on the target sub-trajectory segment, robot joint control commands are generated to drive the sorting robotic arm to move to the area above the target grasping posture data; during the movement, the current visual image data of the end effector is collected in real time, and the current visual image data is compared with the target grasping posture data to obtain real-time posture deviation data;

[0010] The target sub-trajectory segment is dynamically corrected based on the real-time attitude deviation data to obtain the corrected target sub-trajectory segment; the joint control command is updated based on the corrected target sub-trajectory segment to control the grasping mechanism to perform fruit sorting action and place the fruit into the corresponding storage area.

[0011] Preferably, the target grasping posture data is input into the robot's kinematic model, and multiple intermediate posture point data in the joint space are obtained by combining the data with a preset velocity constraint model, including:

[0012] The first constraint relationship between the starting gripping point and the first intermediate attitude point, and the second constraint relationship between the ending gripping point and the second intermediate attitude point are determined based on the velocity constraint model.

[0013] Obtain the joint tangent vector of the next grasping point after the starting grasping point as the first joint tangent vector, and obtain the first trajectory smoothing index of the starting grasping point;

[0014] The first joint tangent vector and the first trajectory smoothing index are weighted and the difference is calculated to obtain the joint tangent vector of the starting grasping point.

[0015] Based on the first constraint relationship, the joint tangent vector of the starting gripping point, and the target gripping posture data, the intermediate posture data of the first intermediate posture point is calculated;

[0016] Based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the intermediate posture data of the second intermediate posture point is calculated.

[0017] Preferably, obtaining the joint tangent vector of the next grasping point after the starting grasping point as the first joint tangent vector, and obtaining the first trajectory smoothing index of the starting grasping point, includes:

[0018] When the starting gripping point is the gripping point of the first fruit to be sorted on the conveyor belt, the joint tangent vector of the next gripping point is obtained and used as the first joint tangent vector;

[0019] Obtain the acceleration change relationship between the starting gripping point and the next gripping point, and use it as the first trajectory smoothing index;

[0020] The joint tangent vector of the starting grab point is generated by performing a weighted difference calculation using the first joint tangent vector and the first trajectory smoothing index.

[0021] Preferably, based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the intermediate posture data of the second intermediate posture point is calculated, including:

[0022] When the termination gripping point is the gripping point of the last fruit to be sorted on the conveyor belt, the joint tangent vector of the previous gripping point is obtained and used as the second joint tangent vector.

[0023] Obtain the trajectory smoothing index of the termination point;

[0024] The joint tangent vector of the termination gripping point is obtained by performing a weighted difference calculation on the trajectory smoothing index of the second joint tangent vector and the termination gripping point.

[0025] Based on the joint tangent vector of the termination gripping point and the second constraint relationship, the intermediate attitude data of the second intermediate attitude point is obtained.

[0026] Preferably, when the starting grasping point is a grasping point other than the first grasping point and the ending grasping point is a grasping point other than the last grasping point, the method further includes:

[0027] Obtain the trajectory smoothing index of the current capture point and the trajectory smoothing index of the next capture point, and use them as the third trajectory smoothing index and the fourth trajectory smoothing index, respectively.

[0028] Obtain the node weight coefficient of the current crawl point, whereby the node weight coefficient represents the ratio between the time interval of the current crawl point and the time interval of the next crawl point.

[0029] The joint tangent vector of the current capture point is determined by weighted summation of the node weight coefficient, the third trajectory smoothing index, and the fourth trajectory smoothing index.

[0030] Substitute the joint tangent vector into the velocity constraint model to obtain the corresponding intermediate attitude point data;

[0031] Wherein, the trajectory smoothing index is the ratio of the attitude vector difference between two adjacent grasping points to the time interval; the node weight coefficient is the ratio of the time interval of the current grasping point to the total time interval, and the total time interval is the sum of the time interval of the current grasping point and the time interval of the next grasping point.

[0032] Preferably, based on the target grasping posture data, the multiple intermediate posture point data, and a preset trajectory smoothing index, a continuous B-spline curve segment is constructed and interpolation is performed to generate a smooth target sub-trajectory segment, including:

[0033] Input the target grasping posture data at different grasping points into the preset node scalar constraint model to obtain the node vector corresponding to the trajectory segment where the starting grasping point is located.

[0034] Based on the node vector and the corresponding multiple intermediate attitude point data, a control point sequence of a cubic B-spline curve is constructed.

[0035] The B-spline curve is subjected to time parameterization, and the trajectory smoothing index is used as a weighting factor to allocate node time, thereby generating a smooth target sub-trajectory segment.

[0036] Preferably, before generating robot joint control commands based on the target sub-trajectory segment to drive the sorting robot arm to move to the area above the target grasping posture data, the method further includes:

[0037] Determine whether the conveyor belt's operating speed is stable;

[0038] When the operating speed is stable, read the signal from the origin sensor and start the joint encoder;

[0039] The light source compensation module of the control image acquisition device is turned on to acquire images of the standard color card;

[0040] After the brightness value of the acquired image matches the preset standard brightness value, the sorting robot arm is controlled to move to the calibration zero point according to the second motion parameter;

[0041] If the coordinate difference between the actual stopping position of the sorting robot arm and the calibrated zero point is greater than or equal to the second preset distance, the speed factor of the second motion parameter is adjusted according to the coordinate difference to obtain the initial motion parameters.

[0042] Preferably, the gripping mechanism includes a vacuum suction cup and a pressure sensor. Controlling the gripping mechanism to perform fruit sorting and place the fruit into the corresponding storage area includes:

[0043] Determine if the reading of the barometric pressure sensor is within the normal operating range;

[0044] When the reading is normal, control the vacuum suction cup to start and detect the magnitude of the suction force;

[0045] If the adsorption force meets the grasping requirements, control the diversion baffle to switch to the channel corresponding to the target sorting level;

[0046] Control the grasping mechanism to perform the grasping action and place the fruit into the corresponding storage area.

[0047] Preferably, the step of acquiring the current visual image data of the end effector in real time during movement, and comparing the current visual image data with the target grasping posture data to obtain real-time posture deviation data includes:

[0048] Real-time acquisition of current visual image data from the robot's end effector;

[0049] Extract the current location features of the fruit from the current visual image data;

[0050] Calculate the difference between the current position features and the theoretical position features in the target grasping posture data;

[0051] The difference is used as real-time attitude deviation data, and it is determined whether the deviation exceeds the allowable range.

[0052] If the deviation exceeds the allowable range, a correction vector is generated based on the deviation and superimposed on the original target sub-trajectory segment to achieve dynamic adjustment of the trajectory.

[0053] A machine vision-based fruit sorting robot control system, applicable to the aforementioned machine vision-based fruit sorting robot control method, includes:

[0054] The feature extraction unit is used to acquire initial fruit image data of the fruit to be sorted on the sorting conveyor belt and the current position data of the robot's end effector; to perform feature extraction and sorting level recognition on the initial fruit image data to obtain fruit feature data and target sorting level;

[0055] The coordinate correction unit is used to compare the fruit feature data with the current position data to calculate the position deviation data of the grasping point; and to correct the preset grasping coordinates corresponding to the target sorting level based on the position deviation data to obtain the target grasping posture data.

[0056] The constraint solving unit is used to input the target grasping posture data into the robot kinematic model and solve for multiple intermediate posture point data in the joint space by combining the preset velocity constraint model.

[0057] The trajectory modeling unit is used to construct continuous B-spline curve segments and perform interpolation operations based on the target grasping posture data, the multiple intermediate posture point data and the preset trajectory smoothing index, to generate smooth target sub-trajectory segments.

[0058] The attitude comparison unit is used to generate robot joint control commands based on the target sub-trajectory segment, drive the sorting robot arm to move to the area above the target grasping attitude data; during the movement, it collects the current visual image data of the end effector in real time, compares the current visual image data with the target grasping attitude data, and obtains real-time attitude deviation data.

[0059] The sorting control unit is used to dynamically correct the target sub-trajectory segment based on the real-time attitude deviation data to obtain the corrected target sub-trajectory segment; and to update the joint control command based on the corrected target sub-trajectory segment to control the gripping mechanism to perform fruit sorting action and place the fruit into the corresponding storage area.

[0060] Compared with the prior art, the beneficial effects of the present invention are:

[0061] (1) By comparing real-time image acquisition with target grasping posture data, the present invention can dynamically correct the target sub-trajectory segment during the movement process, thereby achieving accurate grasping. In this way, the robot can automatically adjust the trajectory according to the actual position and posture of the fruit to ensure high-precision grasping and avoid erroneous grasping due to position deviation. Moreover, by constructing B-spline curves and interpolating, smooth target sub-trajectory segments are generated, making the movement of the robotic arm more stable and smooth, avoiding unnecessary vibration and sharp turning, and improving the stability of the grasping process.

[0062] (2) The present invention dynamically corrects the trajectory by using real-time posture deviation data, enabling the robot to make timely adjustments based on real-time feedback, thereby enhancing the system's adaptability to different environmental changes, such as changes in conveyor belt speed or fruit shape. Furthermore, by combining a preset speed constraint model and trajectory smoothing index, the movement of the robot joints can be effectively controlled to avoid excessively fast or slow movements, thereby improving grasping efficiency and reducing system wear. In addition, by judging the air pressure sensor and adsorption force, it ensures that the grasping action can only be performed when the grasping requirements are met, thus avoiding invalid or unstable grasping and improving the accuracy and reliability of the fruit sorting process. Attached Figure Description

[0063] Figure 1 This is a schematic flowchart of the overall method in one embodiment of the present invention;

[0064] Figure 2 This is a schematic diagram of the overall system architecture in one embodiment of the present invention.

[0065] In the figure: 1. Feature extraction unit; 2. Coordinate correction unit; 3. Constraint solving unit; 4. Trajectory modeling unit; 5. Attitude comparison unit; 6. Sorting and control unit. Detailed Implementation

[0066] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1, please refer to Figure 1 This invention provides a technical solution: a machine vision-based fruit sorting robot control method, comprising:

[0068] S1. Obtain initial fruit image data of the fruit to be sorted on the sorting conveyor belt and the current position data of the robot's end effector; perform feature extraction and sorting level recognition on the initial fruit image data to obtain fruit feature data and target sorting level;

[0069] S2. Compare the fruit feature data with the current location data to calculate the position deviation data of the grasping point; based on the position deviation data, correct the preset grasping coordinates corresponding to the target sorting level to obtain the target grasping posture data;

[0070] S3. Input the target grasping posture data into the robot's kinematic model, and solve the data of multiple intermediate posture points in the joint space by combining the preset velocity constraint model.

[0071] S4. Based on the target capture posture data, multiple intermediate posture point data and preset trajectory smoothing index, construct continuous B-spline curve segments and perform interpolation calculations to generate smooth target sub-trajectory segments.

[0072] S5. Generate robot joint control commands based on the target sub-trajectory segment to drive the sorting robot arm to move to the area above the target grasping posture data; during the movement, collect the current visual image data of the end effector in real time, compare the current visual image data with the target grasping posture data to obtain real-time posture deviation data;

[0073] S6. Dynamically correct the target sub-trajectory segment based on real-time attitude deviation data to obtain the corrected target sub-trajectory segment; update the joint control commands based on the corrected target sub-trajectory segment to control the gripping mechanism to perform fruit sorting actions and place the fruit into the corresponding storage area.

[0074] It should be noted that the system acquires images of the fruit on the sorting conveyor belt using a vision sensor, while simultaneously acquiring position data of the robot's end effector. This image data includes the fruit's appearance characteristics (such as color, shape, and size), while the position data records the specific location of the robot's end effector (gripper). The system then processes the image data, extracting features to identify the fruit's characteristics and determine its sorting grade. For example, if the fruit on the conveyor belt is an apple, the system will analyze its color (red, green, etc.), size, appearance, and other characteristics to determine whether the apple is of high quality or substandard grade.

[0075] The system compares the identified fruit feature data with the robot's current position data to calculate the deviation of the grasping point. The deviation refers to the difference in distance and angle between the robot's end effector and the actual position of the fruit. Based on this deviation, the system adjusts the robot's grasping coordinates to ensure that the robot can accurately grasp the fruit. For example, suppose the robot's preset grasping point is 10 centimeters above the conveyor belt, but due to a slight deviation in the position of the fruit on the conveyor belt, the robot calculates that the actual position of the fruit is offset by 3 centimeters from the preset point. In this case, the robot will correct the position of the target grasping point to ensure accurate grasping.

[0076] The robot will input the corrected target grasping posture data into the kinematic model and, in conjunction with the preset velocity constraint model, calculate multiple intermediate posture points on the robot's motion path. These intermediate posture points are the path points that the robot needs to pass through in the process of moving from the current position to the target grasping position. For example, if the robot needs to move from the current position to the target fruit position, it will calculate the angle and motion steps of each joint on the path to ensure smooth movement and avoid hitting obstacles.

[0077] The robot constructs a smooth motion trajectory based on the target grasping posture data, multiple intermediate posture point data, and preset trajectory smoothing indicators. The trajectory is represented by a B-spline curve, which ensures the smoothness and accuracy of the robot's movement and avoids sudden acceleration or deceleration. For example, if the robot wants to move from the starting position (point A) to the target grasping position (point B), it needs to pass through multiple intermediate points (points C, D, and E). Through the B-spline curve, the robot can move smoothly without vibration.

[0078] Based on the constructed target trajectory, the robot generates joint control commands and drives the robotic arm to move towards the target grasping posture area. During the movement, the robot collects image data from the end effector in real time, observes the difference between the current position and the target posture, and compares the real-time image data with the target grasping posture data to obtain real-time posture deviation data. For example, the robot is moving towards the target position, taking real-time images and comparing them with the preset grasping posture. If it finds that the end effector's position deviates from the target position, it will adjust it.

[0079] The robot dynamically corrects its target trajectory based on real-time posture deviation data to ensure it can accurately reach the target location. The corrected trajectory data updates the joint control commands, and the robot continues to perform the fruit sorting action, picking up the fruit from the sorting conveyor belt and placing it into the corresponding storage area. For example, if the robot encounters a slight error or obstacle during its movement, it will adjust its path and correct the final picking position. The corrected trajectory ensures that the robot can accurately pick up and place the fruit in the designated location (such as a basket or box).

[0080] In an optional embodiment, the target grasping posture data is input into the robot's kinematic model, and multiple intermediate posture point data in the joint space are obtained by combining the data with a preset velocity constraint model, including:

[0081] The first constraint relationship between the starting gripping point and the first intermediate attitude point, and the second constraint relationship between the ending gripping point and the second intermediate attitude point are determined based on the velocity constraint model.

[0082] Obtain the joint tangent vector of the next grasping point after the starting grasping point as the first joint tangent vector, and obtain the first trajectory smoothing index of the starting grasping point;

[0083] The weighted difference calculation is performed on the first joint tangent vector and the first trajectory smoothing index to obtain the joint tangent vector of the starting grab point;

[0084] Based on the first constraint relationship, the joint tangent vector of the starting gripping point, and the target gripping posture data, the intermediate posture data of the first intermediate posture point is calculated;

[0085] Based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the intermediate posture data of the second intermediate posture point is calculated.

[0086] It should be noted that, through a preset velocity constraint model, the constraint relationships between the initial grasping point and the first intermediate posture point, as well as the constraint relationships between the final grasping point and the second intermediate posture point, are determined. These constraints typically involve velocity limits (e.g., the maximum angle that can be moved per second) to ensure that the robot does not exceed the safe range during movement. For example, suppose the robot needs to grasp a fruit from one position and move it to the next. Since the robot has a velocity limit (e.g., a maximum angular velocity of 10 degrees per second), the system uses this limit to calculate the required velocity from the initial grasping point to the first intermediate point, as well as the velocity constraint from the final grasping point to the second intermediate point.

[0087] The system obtains the joint tangent vectors of the next gripping point from the initial gripping point. These vectors represent the direction of movement of each joint of the robot, reflecting how the robot joints adjust towards the target pose point. Simultaneously, the system also obtains the trajectory smoothness index of the initial gripping point. This is typically a parameter measuring the smoothness of the path, used to ensure that the robot does not experience violent jumps or vibrations during movement. For example, suppose the robot needs to move from point A to point B. The system calculates the tangent vectors of the robot joints at point A (i.e., the directional changes of each joint between points A and B) and evaluates the smoothness of the path to avoid violent movements of the robot joints.

[0088] The weighted difference calculation is performed on the first joint tangent vector and the first trajectory smoothing index to obtain the joint tangent vector of the initial grasping point. This is to balance the direction of the joint and the smoothness of the path during the robot's movement, ensuring that the robot can accurately reach the target position while moving without errors or instability caused by path problems. For example, suppose the first joint tangent vector indicates that the robot joint needs to rotate 45 degrees, while the trajectory smoothing index tells the system that this rotation cannot be too abrupt. The system will adjust the joint tangent vector through weighted difference to obtain a final joint direction that meets both velocity constraints and ensures smoothness.

[0089] Based on the first constraint relationship, the joint tangent vector of the starting gripping point, and the target gripping posture data, the system calculates the intermediate posture data of the first intermediate posture point. These data represent the path and joint state of the robot from the starting gripping point to the first intermediate point. For example, during the process of the robot moving from point A to point B, the system will determine the position and angle of the robot joints at each moment based on the previous calculations to ensure that the movement path is smooth and meets the constraint conditions.

[0090] Based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the system calculates the intermediate posture data of the second intermediate posture point. The purpose of this step is to ensure that the path from the target posture to the second intermediate posture point also conforms to the speed and angle constraints, ensuring that the robot can move smoothly to the target position. For example, when the robot is about to complete the gripping and move to the next action, the system will adjust the motion according to the joint tangent vector of the termination gripping point to ensure a smooth transition from the target posture to the second intermediate point.

[0091] In an optional embodiment, obtaining the joint tangent vector of the next grasping point after the starting grasping point as the first joint tangent vector, and obtaining the first trajectory smoothing index of the starting grasping point, includes:

[0092] When the starting gripping point is the gripping point of the first fruit to be sorted on the conveyor belt, obtain the joint tangent vector of the next gripping point and use it as the first joint tangent vector;

[0093] Obtain the acceleration change relationship between the starting gripping point and the next gripping point, and use it as the first trajectory smoothing index;

[0094] The joint tangent vector of the starting grab point is generated by performing a weighted difference calculation using the first joint tangent vector and the first trajectory smoothing index.

[0095] It's important to note that determining the next gripping point (the next object the robot will grasp) from the initial gripping point is crucial. The joint tangent vector represents the direction of motion of each joint of the robot, describing the trajectory from the initial gripping point to the next gripping point. This joint tangent vector will be used as the first joint tangent vector. For example, suppose a fruit sorting robot on a conveyor belt is working. The initial gripping point is the position of the first fruit to be sorted. The robot will move from this position to the position of the next fruit to be sorted. At this point, the robot calculates the joint tangent vector for the next gripping point (the gripping point of the second fruit), representing the movement path from the first fruit to the second fruit. For instance, the robot might need to rotate its joint to the right and move forward.

[0096] The system calculates the acceleration change from the initial gripping point to the next gripping point. This acceleration change determines the smoothness of the robot's motion. If the acceleration change is too large, the robot's motion will appear abrupt, or even vibrate. Therefore, the system calculates the acceleration change along this path to measure the smoothness of the path, known as the first trajectory smoothness index. For example, suppose the robot's trajectory from the first fruit gripping point to the second fruit gripping point follows the direction of the conveyor belt. If the robot suddenly accelerates or decelerates (e.g., due to improper speed control), it may cause mechanical vibration or unstable movement. The system evaluates the acceleration change from the initial gripping point to the next gripping point. If the change is too large, the robot's motion needs to be adjusted to ensure that the acceleration change remains smooth, thus avoiding sudden movements.

[0097] The robot needs to perform weighted difference calculation using the first joint tangent vector and the first trajectory smoothing index. The purpose of the weighted difference calculation is to balance the joint motion direction (joint tangent vector) and the path smoothness (trajectory smoothing index), ensuring that the robot can accurately reach the target position during movement without experiencing severe vibrations or unstable movements. For example, suppose the robot moves from the first fruit position (point A) to the second fruit position (point B). First, the system calculates the joint tangent vector from point A to point B, representing the motion direction of each joint. At the same time, the system also calculates the acceleration change between point A and point B. If the acceleration change is too large, the system will adjust the joint tangent vector through weighted difference, so that the robot can accurately reach point B during movement while ensuring smooth acceleration changes, thereby avoiding unnecessary vibrations.

[0098] In an optional embodiment, based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the intermediate posture data of the second intermediate posture point is calculated, including:

[0099] When the termination gripping point is the gripping point of the last fruit to be sorted on the conveyor belt, obtain the joint tangent vector of the previous gripping point and use it as the second joint tangent vector;

[0100] Obtain the trajectory smoothing index of the termination point;

[0101] The joint tangent vector of the termination gripping point is obtained by performing a weighted difference calculation on the trajectory smoothing index of the second joint tangent vector and the termination gripping point.

[0102] Based on the joint tangent vector of the termination gripping point and the second constraint relationship, the intermediate attitude data of the second intermediate attitude point is obtained.

[0103] It should be noted that when the final gripping point is the last fruit to be sorted on the conveyor belt, the robot needs to calculate the joint tangent vector of the previous gripping point. The joint tangent vector represents the direction of the robot's movement from the previous gripping point to the current gripping point (i.e., the final gripping point). Therefore, this joint tangent vector will be used as the second joint tangent vector. For example, suppose the fruits on the conveyor belt are gripped sequentially, and the final gripping point is the last fruit; the previous gripping point may be the position of the second to last fruit; the robot needs to move from the position of the second to last fruit (the previous gripping point) to the position of the last fruit (the final gripping point). At this time, the joint tangent vector from the previous gripping point to the final gripping point is calculated, which is the direction of the robot's joint movement, and this joint tangent vector is used as the second joint tangent vector.

[0104] The trajectory smoothness index reflects the robot's acceleration change and motion smoothness from the starting point to the ending grasping point. The purpose of calculating this index is to ensure that the robot's movement does not suddenly accelerate or decelerate, avoiding vibrations or instability. This index will be used in subsequent weighted difference calculations. For example, the robot's trajectory smoothness needs to be evaluated from the second-to-last fruit to the last fruit. If the robot suddenly accelerates or decelerates during this period, it may cause unstable movement. The system calculates the trajectory smoothness index to ensure that the acceleration change during movement is not too large, thereby avoiding vibrations or uncoordinated movements.

[0105] By combining the second joint tangent vector (representing the direction of motion from the previous gripping point to the final gripping point) and the trajectory smoothness index of the final gripping point (representing the smoothness of motion), the joint tangent vector of the final gripping point is calculated through weighted difference. This joint tangent vector describes how the robot smoothly transitions from the final gripping point to the next target gripping point. For example, suppose the robot has already calculated the joint tangent vector (second joint tangent vector) from the previous gripping point to the final gripping point, and also has the smoothness index for the final gripping point. Through weighted difference calculation, the system will combine these two data to adjust the robot's joint tangent vector, so that the robot's motion at the final gripping point maintains both directionality and smoothness, avoiding vibration or unsuitable movements during the transition.

[0106] The second intermediate posture point is an intermediate posture point that the robot needs to pass through during its movement. By combining the joint tangent vector of the termination gripping point and the second constraint relationship (which may involve physical constraints such as velocity, acceleration, and posture angles), the robot can calculate the intermediate posture data that needs to be achieved during this process. This data includes information such as joint angles, velocity, and acceleration, which is used to guide the robot's precise movement. For example, in the fruit sorting process, the robot calculates an intermediate posture point between the gripping point of the second to last fruit and the gripping point of the last fruit. This intermediate posture point is a state that needs to be achieved during the movement. For example, it may correspond to a certain stage in the robot's movement, where the robot's joint angles and velocities need to reach specific values ​​to ensure smooth and accurate movement. By combining the joint tangent vector of the termination gripping point and the second constraint relationship, the system calculates these intermediate posture data and guides the robot to successfully complete the task.

[0107] In an optional embodiment, when the starting grab point is a grab point other than the first grab point and the ending grab point is a grab point other than the last grab point, the method further includes:

[0108] Obtain the trajectory smoothing index of the current capture point and the trajectory smoothing index of the next capture point, and use them as the third trajectory smoothing index and the fourth trajectory smoothing index, respectively.

[0109] Obtain the node weight coefficient of the current crawl point. The node weight coefficient represents the ratio between the time interval of the current crawl point and the time interval of the next crawl point.

[0110] The joint tangent vector of the current grab point is determined by weighted summation of the node weight coefficient, the third trajectory smoothing index, and the fourth trajectory smoothing index.

[0111] Substitute the joint tangent vector into the velocity constraint model to obtain the corresponding intermediate attitude point data;

[0112] Among them, the trajectory smoothing index is the ratio of the attitude vector difference between two adjacent grasping points to the time interval; the node weight coefficient is the ratio of the time interval of the current grasping point to the total time interval, and the total time interval is the sum of the time interval of the current grasping point and the time interval of the next grasping point.

[0113] It's important to note that trajectory smoothing metrics represent the smoothness of motion between two adjacent gripping points. It's derived by comparing the pose changes (i.e., pose vector differences) between adjacent gripping points and the time interval between them. For the current gripping point and the next gripping point, two trajectory smoothing metrics are calculated separately, called the third and fourth trajectory smoothing metrics. For example, suppose a robot is performing a sorting task, with the current gripping point being the second fruit and the next gripping point being the third fruit. The system calculates the pose vector difference from the second fruit to the third fruit and considers the time interval between them, thus deriving the third and fourth trajectory smoothing metrics. These metrics reflect the smoothness of the robot's motion between these two points, ensuring that the robot does not exhibit unnatural acceleration or deceleration during movement.

[0114] Node weight coefficients represent the ratio between the time interval of the current grasp point and the time interval of the next grasp point. They can be used to assess the relative importance of the current grasp point throughout the entire motion process and influence subsequent joint tangent vector calculations. For example, if the time interval between the current grasp point and the next grasp point is long, the node weight coefficient of the current grasp point will be large, meaning that the current grasp point has a greater impact on the entire motion process; conversely, if the time interval is short, the node weight coefficient will be small. For instance, if the grasping time from the second fruit to the third fruit is 1 second, and the time from the third fruit to the fourth fruit is 2 seconds, then the node weight coefficient of the current grasp point (the second fruit) will be relatively small.

[0115] By weighted summing the node weight coefficients of the current grasping point, the third trajectory smoothing index, and the fourth trajectory smoothing index, the joint tangent vector of the current grasping point is obtained. This joint tangent vector represents the direction and amount of change of the robot's movement from the current grasping point to the next grasping point. The purpose of weighted summation is to comprehensively consider the time interval, smoothness, and node weights to make the robot's movement more accurate and smooth. For example, suppose that when the robot grasps the second fruit to the third fruit, the time interval is short, so the node weight coefficient is small. At the same time, the third trajectory smoothing index shows that the movement between the second fruit and the third fruit is relatively smooth. After comprehensively considering these two indices and the time interval, the system calculates a joint tangent vector to guide the robot on how to move smoothly from the second fruit to the third fruit.

[0116] After calculation, the joint tangent vectors are incorporated into a velocity constraint model to ensure that the robot's joint movements do not violate any physical constraints in terms of velocity and acceleration. This yields data on intermediate posture points, including joint angles, velocities, and accelerations, which guide the robot's state at each critical moment during movement. For example, after calculating the joint tangent vectors, the system incorporates them into the velocity constraint model to ensure that the robot's velocity and acceleration are not too fast or too slow during the transition from the second fruit to the third fruit. Through such constraints, the robot can maintain a reasonable motion state while ensuring a smooth transition, avoiding sudden acceleration or deceleration.

[0117] In an optional embodiment, based on target grasping posture data, multiple intermediate posture point data, and a preset trajectory smoothing index, continuous B-spline curve segments are constructed and interpolation operations are performed to generate smooth target sub-trajectory segments, including:

[0118] Input the target grasping posture data at different grasping points into the preset node scalar constraint model to obtain the node vector corresponding to the trajectory segment where the starting grasping point is located.

[0119] Based on the node vectors and the corresponding multiple intermediate attitude point data, construct the control point sequence of the cubic B-spline curve;

[0120] The B-spline curve is parameterized over time, and the trajectory smoothing index is used as a weighting factor to allocate node time, thereby generating a smooth target sub-trajectory segment.

[0121] It should be noted that the target posture data (i.e., the robot's target grasping position, orientation, etc.) for different grasping points are processed and input into a preset node scalar constraint model. This model generates a node vector for each grasping point based on the target posture data and trajectory requirements. This node vector is a key factor describing the position and time of the grasping point within the trajectory, helping to determine the control points and time allocation for the trajectory. For example, suppose the robot needs to grasp a set of items; the target grasping points include the first fruit (target 1), the second fruit (target 2), and the third fruit (target 3). The data for these target grasping points (e.g., position, orientation, and velocity requirements) are input into the preset node scalar constraint model, which generates a node vector for each grasping point based on this data. For instance, the node vector for target 1 might represent the robot's desired position, orientation, and velocity at that point, while the node vectors for targets 2 and 3 represent their desired states, respectively.

[0122] After obtaining the node vectors of each grasping point, the next step is to construct a B-spline curve based on these node vectors and intermediate attitude point data. A B-spline curve is a mathematical tool widely used in trajectory planning; it describes a smooth curve using a sequence of control points. Here, the node vectors and intermediate attitude point data are combined to form a sequence of control points for a cubic B-spline curve. These control points define the shape and variation of the trajectory. For example, continuing with the fruit grasping example, suppose there are multiple intermediate attitude points, representing key locations the robot should traverse during the grasping process. For instance, the robot might pass through an intermediate point A between grasping target 1 and target 2; and another intermediate point B between target 2 and target 3. By combining the target grasping points (such as target 1, target 2, and target 3) with these intermediate attitude points (such as A and B), a sequence of control points can be formed. These control points act like anchor points, defining the path of the B-spline curve and ensuring a smooth transition for the robot along these points.

[0123] After constructing the B-spline curve, time parameterization is required. The purpose of time parameterization is to ensure that the robot's movement speed is reasonable and smooth throughout the entire trajectory. Here, the trajectory smoothness index (representing the smoothness of movement between adjacent grasping points) will be used as a weighting factor to help allocate the time of each control point reasonably, ensuring that the robot's movement does not suddenly accelerate or decelerate, thus maintaining a stable trajectory. For example, suppose the robot's movement from target 1 to target 2 is relatively smooth, while the movement from target 2 to target 3 is slightly abrupt with a large speed change. In this case, the trajectory smoothness index will reflect this difference, making the transition from target 1 to target 2 smoother, while the transition from target 2 to target 3 will be appropriately adjusted according to the smoothness index to prevent excessive acceleration or deceleration. In this way, each segment of the robot's movement will be adapted according to the trajectory smoothness requirements, ultimately generating a smooth and accurate target sub-trajectory segment.

[0124] In an optional embodiment, before generating robot joint control commands based on the target sub-trajectory segment to drive the sorting robot arm to move to the area above the target grasping posture data, the method further includes:

[0125] Determine whether the conveyor belt's operating speed is stable;

[0126] When the operating speed is stable, read the signal from the origin sensor and start the joint encoder;

[0127] The light source compensation module of the control image acquisition device is turned on to acquire images of the standard color card;

[0128] After the brightness value of the acquired image matches the preset standard brightness value, the sorting robot arm is controlled to move to the calibration zero point according to the second motion parameter;

[0129] If the coordinate difference between the actual stopping position of the sorting robot arm and the calibrated zero point is greater than or equal to the second preset distance, the speed factor of the second motion parameter is adjusted according to the coordinate difference to obtain the initial motion parameters.

[0130] It's important to note that before starting the grasping process, it's crucial to ensure the conveyor belt's operation is stable. Unstable conveyor belt speed can cause positional deviations during grasping, affecting accuracy. For example, imagine an automated fruit sorting system where oranges are moving along a conveyor belt. Before grasping, the system monitors the conveyor belt speed. If the speed fluctuates frequently over a period (e.g., from 0.5 m / s to 0.3 m / s), the system waits until the speed stabilizes within a predetermined range (e.g., 0.5 ± 0.05 m / s).

[0131] Once the conveyor belt speed is confirmed to be stable, the system will then read the signal from the origin sensor. The origin sensor is typically used to determine the position of the robotic arm, ensuring it is at the correct starting point. At the same time, the joint encoders are activated to monitor the position of the robotic arm in real time for subsequent control. For example, under stable conditions, the system detects a signal from the origin sensor, indicating that the robotic arm has returned to its initial position. At this point, the joint encoders start working, providing real-time feedback on the angles and positions of each joint of the robotic arm to ensure that the upcoming action starts from the correct position.

[0132] To ensure the quality and accuracy of image acquisition, the system activates the light source compensation module. This module adjusts the light source according to changes in ambient light to ensure stable lighting conditions during image acquisition. The system then acquires images of the standard color chart for subsequent color recognition and processing. For example, the system activates the light source compensation module to adjust the light intensity and eliminate interference from ambient light. Then, the system photographs the standard color chart placed on the conveyor belt. Assuming the standard color chart is a vibrant red, the system analyzes the image acquisition results to determine the color of the actual item.

[0133] After image acquisition is complete, the system compares the acquired image brightness value with a preset standard brightness value. If they match, it indicates that the lighting conditions are good and the image quality is acceptable. At this point, the system controls the robotic arm to move to a preset calibration zero point position to prepare for subsequent grasping. For example, suppose that after image acquisition, the system finds that the acquired image brightness value is 150, which matches the preset standard brightness value of 150. Therefore, the system instructs the robotic arm to move smoothly to the calibration zero point along the second motion parameter (such as speed and acceleration), ensuring that the robotic arm is in the ideal position when ready to grasp.

[0134] After the robotic arm reaches the calibration zero point, the system checks the distance between the actual stopping position and the calibration zero point. If this distance is greater than or equal to a preset threshold, it means that the robotic arm has not accurately reached the target position. At this time, the motion parameters need to be adjusted to ensure that the next action is more accurate. For example, suppose the coordinates of the actual stopping position are (5,5) and the coordinates of the calibration zero point are (5,3). The difference between them is 2 units, which exceeds the preset threshold of 1 unit. The system will adjust the second motion parameter according to this difference, such as reducing the speed factor, so that the robotic arm can approach the calibration zero point with a lower speed and higher accuracy in the next movement, ensuring the accuracy of the grasping position.

[0135] In an optional embodiment, the gripping mechanism includes a vacuum suction cup and a pressure sensor, controlling the gripping mechanism to perform fruit sorting actions and place the fruit into the corresponding storage area, including:

[0136] Determine if the reading of the barometric pressure sensor is within the normal operating range;

[0137] When the reading is normal, control the vacuum suction cup to start and detect the magnitude of the suction force;

[0138] If the adsorption force meets the grasping requirements, control the diversion baffle to switch to the channel corresponding to the target sorting level;

[0139] Control the grasping mechanism to perform the grasping action and place the fruit into the corresponding storage area.

[0140] It should be noted that the pressure sensor is used to detect the working status of the vacuum suction cup, ensuring its suction function is normal. When the pressure sensor reading is within the predetermined normal operating range, it indicates that the vacuum suction cup is working properly; otherwise, insufficient vacuum may occur, leading to unstable suction. For example, in a fruit sorting system, a pressure sensor reading between 0.8 and 1.0 bar is considered normal. Assuming the pressure sensor reading is 0.9 bar, the system will determine that the vacuum suction cup is working properly and can continue with subsequent operations. If the pressure reading is 0.7 bar, it indicates that the suction force of the vacuum suction cup is insufficient, the system will trigger an alarm and stop operation until the problem is resolved.

[0141] When the air pressure sensor reading is normal, the system will activate the vacuum suction cup and begin measuring its suction force. The suction force of the vacuum suction cup needs to be strong enough to ensure a firm grip on the fruit. The magnitude of the suction force directly affects the success rate of the grasping process. For example, during operation, assuming the air pressure sensor reads within the normal range, the system will activate the vacuum suction cup and begin measuring the suction force. For instance, the suction force may vary between 0 and 100. If the suction force reaches 70 or higher, it indicates that the grip is strong enough, and the grasping operation can continue. If the suction force is lower than 70, the system will automatically adjust the working state of the vacuum suction cup to increase the suction force until sufficient suction force is achieved.

[0142] Once the suction force of the vacuum suction cup is confirmed to be strong enough, the system will control the diversion baffle to switch to the channel of the target sorting level. This diversion baffle diverts the fruit to different storage areas based on the size, weight, or other characteristics of the fruit. For example, suppose the fruit system sorts fruits according to size, and the diversion baffle has three channels, corresponding to small, medium, and large fruits respectively. After the vacuum suction cup successfully picks up a large orange, the system will determine the size of the fruit and switch the diversion baffle to the large channel, sending the orange to the dedicated storage area. If it is a small apple, it will switch to the small channel and send it to another storage area.

[0143] Once all preparations are complete, the system will instruct the gripping mechanism to perform the gripping action, picking up the fruit from the conveyor belt and placing it into the corresponding storage area. During the gripping process, the action must be smooth and accurate to avoid damaging the fruit. For example, assuming the system grips a banana, the vacuum suction cup will lift the banana from the conveyor belt and place it into the corresponding storage area according to the type and size of the fruit. For instance, if the fruit is a medium-sized banana and the diversion baffle is pointing towards the middle channel, the system will place the banana in a dedicated medium-sized fruit storage area. The gripping action must be very precise to prevent the banana from falling or being crushed during placement.

[0144] In one optional embodiment, the current visual image data of the end effector is acquired in real time during the movement, and the current visual image data is compared with the target grasping posture data to obtain real-time posture deviation data, including:

[0145] Real-time acquisition of current visual image data from the robot's end effector;

[0146] Extract the current location features of the fruit from the current visual image data;

[0147] Calculate the difference between the current position features and the theoretical position features in the target grasping posture data;

[0148] The difference is used as real-time attitude deviation data, and it is determined whether the deviation exceeds the allowable range.

[0149] If the deviation exceeds the allowable range, a correction vector is generated based on the deviation and superimposed on the original target sub-trajectory segment to achieve dynamic adjustment of the trajectory.

[0150] It should be noted that during the grasping process, the robot needs to collect real-time visual image data of the environment in which the end effector (such as the grasping device on the robotic arm) is located. This image data is acquired through cameras or other visual sensors to help the robot understand the current environmental state and the position of objects. For example, suppose the robot is operating in a fruit sorting system, and the camera at the end of the robot takes real-time images of the fruit. If the robot is about to grasp an apple, the camera will take an image of the apple, providing information about the apple's current position, shape, and orientation.

[0151] By analyzing the currently acquired visual image data, the system extracts the positional features of the fruit in the image, such as the fruit's center point, angle, and size. These features help the system identify the fruit's precise location and orientation. For example, suppose a robot is picking up an orange, and the image captured by the camera shows that the orange is roughly located at a certain coordinate. The system uses image processing techniques (such as target recognition and edge detection) to extract the orange's center point coordinates and angle, determining its current actual position relative to the robot's end effector.

[0152] When planning a grasping task, the robot sets theoretical target position features (such as target position and orientation) based on the target posture data. Then, the system calculates the difference between the actual fruit position features and the preset target position features. This difference is the posture deviation data. For example, suppose the theoretical grasping posture requires the robot's end effector to grasp the orange at the exact center, but the actual position shown in the image is the orange slightly to the left or right. The system will calculate this deviation. For instance, if the target position feature is the exact center of the orange, but the actual position deviates from the target position by 10 centimeters, the calculated deviation is 10 centimeters.

[0153] Once the posture deviation data is calculated, the system compares this deviation with the set allowable error range. If the deviation exceeds the set allowable range, the system considers the current posture unsuitable and requires correction. For example, suppose the system sets a tolerable posture deviation range of 5 centimeters. If the currently calculated deviation is 10 centimeters, then the deviation exceeds the allowable range, indicating that the robot's end effector's grasping posture needs correction. If the deviation is within 5 centimeters, the system considers the current grasping posture to be accurate enough and does not require correction.

[0154] If the deviation exceeds the allowable range, the system will generate a correction vector to adjust the robot's trajectory. This means that the system will calculate an adjustment direction and apply it to the original target trajectory, dynamically correcting the robot's actions to ensure accurate grasping. For example, if the deviation is 10 centimeters, exceeding the allowable error range of 5 centimeters, the system will calculate a correction vector, such as moving the robot's end effector 10 centimeters to the left, and superimposing this correction vector onto the original trajectory. In this way, the robot will dynamically adjust its path to ensure that it can accurately grasp the orange.

[0155] Example 2, please refer to Figure 2 This invention provides a technical solution: a machine vision-based fruit sorting robot control system, applicable to the aforementioned machine vision-based fruit sorting robot control method, comprising:

[0156] Feature extraction unit 1 is used to acquire initial fruit image data of the fruit to be sorted on the sorting conveyor belt and the current position data of the robot end effector; to perform feature extraction and sorting level recognition on the initial fruit image data to obtain fruit feature data and target sorting level;

[0157] The coordinate correction unit 2 is used to compare the fruit feature data with the current position data and calculate the position deviation data of the grasping point; based on the position deviation data, the preset grasping coordinates corresponding to the target sorting level are corrected to obtain the target grasping posture data.

[0158] Constraint solving unit 3 is used to input the target grasping posture data into the robot kinematic model and solve for multiple intermediate posture point data in the joint space by combining the preset velocity constraint model.

[0159] The trajectory modeling unit 4 is used to construct continuous B-spline curve segments and perform interpolation calculations based on the target grasping posture data, multiple intermediate posture point data and preset trajectory smoothing index, to generate smooth target sub-trajectory segments.

[0160] The attitude comparison unit 5 is used to generate robot joint control commands based on the target sub-trajectory segment, drive the sorting robot arm to move to the area above the target grasping attitude data; during the movement, the current visual image data of the end effector is collected in real time, and the current visual image data is compared with the target grasping attitude data to obtain real-time attitude deviation data.

[0161] The sorting control unit 6 is used to dynamically correct the target sub-trajectory segment based on real-time attitude deviation data to obtain the corrected target sub-trajectory segment; and to update the joint control command based on the corrected target sub-trajectory segment to control the gripping mechanism to perform fruit sorting action and place the fruit into the corresponding storage area.

[0162] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited thereto. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of the present invention.

Claims

1. A control method for a fruit sorting robot based on machine vision, characterized in that, include: Acquire initial fruit image data of the fruits to be sorted on the sorting conveyor belt and the current position data of the robot's end effector; Feature extraction and sorting level identification are performed on the initial fruit image data to obtain fruit feature data and target sorting level; The fruit feature data is compared with the current location data to calculate the position deviation data of the grasping point; the preset grasping coordinates corresponding to the target sorting level are corrected based on the position deviation data to obtain the target grasping posture data; The target grasping posture data is input into the robot kinematics model, and the data of multiple intermediate posture points in the joint space are obtained by combining the preset velocity constraint model. Based on the target grasping posture data, the multiple intermediate posture point data, and the preset trajectory smoothing index, a continuous B-spline curve segment is constructed and interpolation is performed to generate a smooth target sub-trajectory segment. Based on the target sub-trajectory segment, robot joint control commands are generated to drive the sorting robotic arm to move to the area above the target grasping posture data; during the movement, the current visual image data of the end effector is collected in real time, and the current visual image data is compared with the target grasping posture data to obtain real-time posture deviation data; The target sub-trajectory segment is dynamically corrected based on the real-time attitude deviation data to obtain the corrected target sub-trajectory segment; the joint control command is updated based on the corrected target sub-trajectory segment to control the grasping mechanism to perform fruit sorting action and place the fruit into the corresponding storage area.

2. The control method for a fruit sorting robot based on machine vision according to claim 1, characterized in that, The target grasping posture data is input into the robot's kinematic model, and combined with a preset velocity constraint model, multiple intermediate posture point data in the joint space are obtained, including: The first constraint relationship between the starting gripping point and the first intermediate attitude point, and the second constraint relationship between the ending gripping point and the second intermediate attitude point are determined based on the velocity constraint model. Obtain the joint tangent vector of the next grasping point after the starting grasping point as the first joint tangent vector, and obtain the first trajectory smoothing index of the starting grasping point; The first joint tangent vector and the first trajectory smoothing index are weighted and the difference is calculated to obtain the joint tangent vector of the starting grasping point. Based on the first constraint relationship, the joint tangent vector of the starting gripping point, and the target gripping posture data, the intermediate posture data of the first intermediate posture point is calculated; Based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the intermediate posture data of the second intermediate posture point is calculated.

3. The fruit sorting robot control method based on machine vision according to claim 2, characterized in that, Obtaining the joint tangent vector of the next grasping point after the initial grasping point as the first joint tangent vector, and obtaining the first trajectory smoothing index of the initial grasping point, including: When the starting gripping point is the gripping point of the first fruit to be sorted on the conveyor belt, the joint tangent vector of the next gripping point is obtained and used as the first joint tangent vector; Obtain the acceleration change relationship between the starting gripping point and the next gripping point, and use it as the first trajectory smoothing index; The joint tangent vector of the starting grab point is generated by performing a weighted difference calculation using the first joint tangent vector and the first trajectory smoothing index.

4. The fruit sorting robot control method based on machine vision according to claim 3, characterized in that, Based on the second constraint relationship, the joint tangent vector of the termination gripping point, and the target gripping posture data, the intermediate posture data of the second intermediate posture point is calculated, including: When the termination gripping point is the gripping point of the last fruit to be sorted on the conveyor belt, the joint tangent vector of the previous gripping point is obtained and used as the second joint tangent vector. Obtain the trajectory smoothing index of the termination point; The joint tangent vector of the termination gripping point is obtained by performing a weighted difference calculation on the trajectory smoothing index of the second joint tangent vector and the termination gripping point. Based on the joint tangent vector of the termination gripping point and the second constraint relationship, the intermediate attitude data of the second intermediate attitude point is obtained.

5. The machine vision-based fruit sorting robot control method according to claim 4, characterized in that, When the starting grab point is a grab point other than the first grab point and the ending grab point is a grab point other than the last grab point, the method further includes: Obtain the trajectory smoothing index of the current capture point and the trajectory smoothing index of the next capture point, and use them as the third trajectory smoothing index and the fourth trajectory smoothing index, respectively. Obtain the node weight coefficient of the current crawl point, whereby the node weight coefficient represents the ratio between the time interval of the current crawl point and the time interval of the next crawl point. The joint tangent vector of the current capture point is determined by weighted summation of the node weight coefficient, the third trajectory smoothing index, and the fourth trajectory smoothing index. Substitute the joint tangent vector into the velocity constraint model to obtain the corresponding intermediate attitude point data; Wherein, the trajectory smoothing index is the ratio of the attitude vector difference between two adjacent grasping points to the time interval; the node weight coefficient is the ratio of the time interval of the current grasping point to the total time interval, and the total time interval is the sum of the time interval of the current grasping point and the time interval of the next grasping point.

6. The machine vision-based fruit sorting robot control method according to claim 5, characterized in that, Based on the target grasping posture data, the multiple intermediate posture point data, and the preset trajectory smoothing index, continuous B-spline curve segments are constructed and interpolation operations are performed to generate smooth target sub-trajectory segments, including: Input the target grasping posture data at different grasping points into the preset node scalar constraint model to obtain the node vector corresponding to the trajectory segment where the starting grasping point is located. Based on the node vector and the corresponding multiple intermediate attitude point data, a control point sequence of a cubic B-spline curve is constructed. The B-spline curve is subjected to time parameterization, and the trajectory smoothing index is used as a weighting factor to allocate node time, thereby generating a smooth target sub-trajectory segment.

7. The machine vision-based fruit sorting robot control method according to claim 6, characterized in that, Before generating robot joint control commands based on the target sub-trajectory segment to drive the sorting robot arm to move to the area above the target grasping posture data, the method further includes: Determine whether the conveyor belt's operating speed is stable; When the operating speed is stable, read the signal from the origin sensor and start the joint encoder; The light source compensation module of the control image acquisition device is turned on to acquire images of the standard color card; After the brightness value of the acquired image matches the preset standard brightness value, the sorting robot arm is controlled to move to the calibration zero point according to the second motion parameter; If the coordinate difference between the actual stopping position of the sorting robot arm and the calibrated zero point is greater than or equal to the second preset distance, the speed factor of the second motion parameter is adjusted according to the coordinate difference to obtain the initial motion parameters.

8. The machine vision-based fruit sorting robot control method according to claim 7, characterized in that, The gripping mechanism includes a vacuum suction cup and a pressure sensor. Controlling the gripping mechanism to perform fruit sorting and place the fruit into the corresponding storage area includes: Determine if the reading of the barometric pressure sensor is within the normal operating range; When the reading is normal, control the vacuum suction cup to start and detect the magnitude of the suction force; If the adsorption force meets the grasping requirements, control the diversion baffle to switch to the channel corresponding to the target sorting level; Control the grasping mechanism to perform the grasping action and place the fruit into the corresponding storage area.

9. The fruit sorting robot control method based on machine vision according to claim 8, characterized in that, The process of acquiring current visual image data of the end effector in real time during movement, comparing the current visual image data with the target grasping posture data to obtain real-time posture deviation data includes: Real-time acquisition of current visual image data from the robot's end effector; Extract the current location features of the fruit from the current visual image data; Calculate the difference between the current position features and the theoretical position features in the target grasping posture data; The difference is used as real-time attitude deviation data, and it is determined whether the deviation exceeds the allowable range. If the deviation exceeds the allowable range, a correction vector is generated based on the deviation and superimposed on the original target sub-trajectory segment to achieve dynamic adjustment of the trajectory.

10. A machine vision-based fruit sorting robot control system, applicable to the machine vision-based fruit sorting robot control method according to any one of claims 1-9, characterized in that, include: The feature extraction unit is used to acquire initial fruit image data of the fruits to be sorted on the sorting conveyor belt and the current position data of the robot's end effector; Feature extraction and sorting level identification are performed on the initial fruit image data to obtain fruit feature data and target sorting level; The coordinate correction unit is used to compare the fruit feature data with the current position data to calculate the position deviation data of the grasping point; and to correct the preset grasping coordinates corresponding to the target sorting level based on the position deviation data to obtain the target grasping posture data. The constraint solving unit is used to input the target grasping posture data into the robot kinematic model and solve for multiple intermediate posture point data in the joint space by combining the preset velocity constraint model. The trajectory modeling unit is used to construct continuous B-spline curve segments and perform interpolation operations based on the target grasping posture data, the multiple intermediate posture point data and the preset trajectory smoothing index, to generate smooth target sub-trajectory segments. The attitude comparison unit is used to generate robot joint control commands based on the target sub-trajectory segment, drive the sorting robot arm to move to the area above the target grasping attitude data; during the movement, it collects the current visual image data of the end effector in real time, compares the current visual image data with the target grasping attitude data, and obtains real-time attitude deviation data. The sorting control unit is used to dynamically correct the target sub-trajectory segment based on the real-time attitude deviation data to obtain the corrected target sub-trajectory segment; and to update the joint control command based on the corrected target sub-trajectory segment to control the gripping mechanism to perform fruit sorting action and place the fruit into the corresponding storage area.

Images