A control method, apparatus, and storage medium for a robotic arm used for fruit grasping.

Abstract

This invention provides a robotic arm control method, device, and storage medium for fruit grasping, belonging to the field of robotic arm control technology. The method includes: acquiring images of the grasping area using an RGB-D depth camera mounted on the end effector of the robotic arm to obtain an RGB image of the fruit and a depth map of the fruit; performing pose analysis on the RGB image and depth map of the fruit to obtain a set of spatial physical poses; and performing predictive analysis on the set of spatial physical poses to obtain a spatial pose prediction vector function. This invention solves the technical problems of control phase lag, reciprocating oscillation, and frequent grasping failures caused by the superposition of camera processing delay and mechanical response lag in traditional pose feedback tracking systems. It achieves high-precision, soft-contact, and non-destructive harvesting of dynamically low-frequency forced oscillation fruits under natural field conditions.

Description

Technical Field

[0001] This invention relates to the field of robotic arm control technology, specifically to a robotic arm control method, device, and storage medium for fruit grasping. Background Technology

[0002] In the field of automated agricultural harvesting, using computer vision to guide a robotic arm to track and locate target fruits is the core foundation for achieving autonomous operation. Commonly known technologies typically employ single-stage target detection networks (such as the standard YOLO network) to capture images at high frequency, calculate the absolute three-dimensional coordinates of the fruit's center in the camera coordinate system, and use this as the position input source for the robotic arm's planning module. Based on these coordinates, the robotic arm's kinematic control module uses interpolation algorithms such as cubic or quintic polynomials and cubic spline curves to plan a smooth macroscopic approaching motion trajectory in joint space or Cartesian space. This trajectory drives the end effector (such as an electric gripper) to approach and close towards the target point, thus completing the conventional quasi-static grasping task of the fruit.

[0003] Existing technologies attempt to achieve real-time tracking of moving targets by increasing the closed-loop frequency of visual sampling and low-level control. However, when faced with natural wind disturbances, this approach suffers from unavoidable defects such as "control phase lag" and "system excitation overload" due to the lack of mathematical modeling of the temporal oscillation patterns. In real agricultural field environments, fruits exhibit nonlinear low-frequency resonant oscillations (like the periodic swinging of an inverted pendulum) after being blown by wind, and their spatial motion has accelerations that change drastically over time. Logically, this approach is a typical "posterior-reaction control," meaning that the control loop only responds after the fruit's position has changed.

[0004] To address the target's motion properties, some improved existing technologies have introduced velocity estimation mechanisms. However, these improved schemes suffer from serious control loopholes, including "linear misjudgment of the motion model" and "excessive relative contact kinetic energy damaging the fruit." The fruit's swaying, induced by natural wind, is a forced oscillation or damped resonance. Its motion equation conforms to a nonlinear temporal distribution of sine or cosine functions, and its velocity vector exhibits periodic reversals in direction and abrupt changes in magnitude (maximum kinetic energy and peak velocity at its equilibrium position; zero kinetic energy and minimum velocity at the highest turning point). This scheme blindly simplifies the nonlinear swaying into a constant-velocity linear motion, resulting in a significant mathematical error in the calculated geometric interception point, leading to extremely low interception accuracy. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to address the shortcomings of the prior art by providing a robotic arm control method, device and storage medium for fruit grasping.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: A robotic arm control method for fruit grasping, comprising the following steps: The RGB-D depth camera installed on the end eye of the robotic arm acquires images of the area to be grasped, resulting in multiple RGB images of the fruit and a fruit depth map corresponding to each RGB image. Pose analysis is performed on each of the fruit RGB images and the fruit depth map corresponding to each of the fruit RGB images to obtain a set of spatial physical poses; The spatial physical pose set is subjected to predictive analysis to obtain the spatial pose prediction vector function; The spatial pose prediction vector function is screened and analyzed, and the results of the screening and analysis are used as the endpoint target boundary. Import the current real-time position group, the current real-time velocity group, and the current real-time acceleration group of the robotic arm end effector at the current moment. Perform reconstruction analysis on the current real-time position group, the current real-time velocity group, the current real-time acceleration group, and the endpoint target boundary to obtain the spatial pose reconstruction vector function. The spatial pose reconstruction vector function is solved and analyzed, and the robotic arm is controlled based on the solution and analysis results.

[0007] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: A robotic arm control device for fruit grasping, comprising: The image acquisition module is used to acquire images of the area to be grasped by using an RGB-D depth camera set on the end eye of the robotic arm, and to obtain multiple RGB images of the fruit and a fruit depth map corresponding to each RGB image of the fruit. The pose analysis module is used to perform pose analysis on each of the fruit RGB images and the fruit depth map corresponding to each of the fruit RGB images to obtain a set of spatial physical poses. The predictive analysis module is used to perform predictive analysis on the set of spatial physical poses to obtain a spatial pose prediction vector function. The filtering and analysis module is used to filter and analyze the spatial pose prediction vector function and use the filtering and analysis results as the endpoint target boundary. The reconstruction analysis module is used to import the current real-time position group, the current real-time velocity group, and the current real-time acceleration group of the robotic arm end effector at the current moment, and to perform reconstruction analysis on the current real-time position group, the current real-time velocity group, the current real-time acceleration group, and the endpoint target boundary to obtain the spatial pose reconstruction vector function. The robotic arm control module is used to solve and analyze the spatial pose reconstruction vector function, and control the robotic arm based on the solution and analysis results.

[0008] Based on the above-mentioned robotic arm control method for fruit grasping, the present invention also provides a robotic arm control system for fruit grasping.

[0009] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a robotic arm control system for fruit grasping, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robotic arm control method for fruit grasping as described above.

[0010] Based on the above-described robotic arm control method for fruit grasping, the present invention also provides a computer-readable storage medium.

[0011] Another technical solution of the present invention to solve the above-mentioned technical problems is as follows: a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the robotic arm control method for fruit grasping as described above.

[0012] The beneficial effects of this invention are as follows: An RGB-D depth camera mounted on the end effector of a robotic arm acquires images of the fruit's RGB image and depth map of the area to be grasped. Pose analysis of the RGB image and depth map yields a set of spatial physical poses. Predictive analysis of this set yields a spatial pose prediction vector function. The spatial pose prediction vector function is then filtered, and the results are used as the endpoint target boundary. Finally, reconstruction analysis of the current real-time position set, current real-time velocity set, current real-time acceleration set, and endpoint target boundary yields a spatial pose reconstruction vector function. The system calculates and analyzes the spatial pose reconstruction vector function, and controls the robotic arm based on the results. This solves the technical problems of control phase lag, reciprocating oscillation, and frequent missed grasping caused by the superposition of camera processing delay and mechanical response lag in traditional pose feedback tracking systems. It achieves high-precision, soft-contact, and lossless collaborative harvesting of dynamically low-frequency forced oscillation fruits under natural field conditions, eliminates the reciprocating oscillation and overshoot of the robotic arm, ensures perfect soft contact when the gripper fingertips close and wrap the fruit, avoids any local normal or tangential physical impact loads, and significantly improves the success rate of collaborative tracking and harvesting and the fruit loss rate under natural field disturbance environments. Attached Figure Description

[0013] Figure 1A flowchart illustrating the robotic arm control method for fruit grasping provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the multimodal spatiotemporal pose synchronization and coordinate system transformation principle of the robotic arm control method for fruit grasping provided in an embodiment of the present invention. Figure 3 A schematic diagram of sliding window fitting of the nonlinear low-frequency simple resonant oscillation of the target fruit in the robotic arm control method for fruit grasping provided in an embodiment of the present invention. Figure 4 The absolute motion velocity curve and the spatiotemporal window discrimination diagram for relative zero-speed grasping of the robotic arm control method for fruit grasping provided in the embodiments of the present invention; Figure 5 A comparison diagram of the kinematic impact of the conventional follower control and the feedforward reconstruction of the present invention in the control method of the robotic arm for fruit grasping provided in the embodiments of the present invention; Figure 6 A block diagram illustrating the dynamic timing prediction and feedforward compensation control loop principle of the robotic arm control method for fruit grasping provided in this embodiment of the invention; Figure 7 The main flowchart of the robotic arm control method for fruit grasping provided in the embodiment of the present invention; Figure 8 This is a block diagram of a robotic arm control device for fruit grasping provided in an embodiment of the present invention. Detailed Implementation

[0014] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0015] Figure 1 This is a flowchart illustrating a robotic arm control method for fruit grasping, provided as an embodiment of the present invention.

[0016] like Figure 1 As shown, a robotic arm control method for fruit grasping includes the following steps: S1: The RGB-D depth camera set on the end eye of the robotic arm acquires images of the area to be grasped, and obtains multiple RGB images of the fruit and a fruit depth map corresponding to each of the RGB images of the fruit. S2: Perform pose analysis on each of the fruit RGB images and the fruit depth map corresponding to each of the fruit RGB images to obtain a set of spatial physical poses; S3: Perform predictive analysis on the set of spatial physical poses to obtain the spatial pose prediction vector function; S4: Perform a screening analysis on the spatial pose prediction vector function, and use the screening analysis result as the endpoint target boundary; S5: Import the current real-time position group, the current real-time velocity group, and the current real-time acceleration group of the robotic arm end effector at the current moment; perform reconstruction analysis on the current real-time position group, the current real-time velocity group, the current real-time acceleration group, and the endpoint target boundary to obtain the spatial pose reconstruction vector function. S6: Perform solution analysis on the spatial pose reconstruction vector function, and control the robotic arm based on the solution analysis results.

[0017] It should be understood that after the robotic arm moves to the initial preset imaging pose of the field harvesting area (i.e., the area to be grasped), the end-eye RGB-D depth camera activates a high-frequency hard-triggered sampling mode to fix the sampling period. (For example Corresponding frame rate High-frequency synchronous acquisition of standard RGB images (i.e., fruit RGB images) of dynamically shaking target fruits and time-domain aligned depth maps (i.e., fruit depth maps).

[0018] In the above embodiments, an RGB-D depth camera mounted on the end effector of the robotic arm acquires images of the fruit's RGB image and depth map of the area to be grasped. Pose analysis of the fruit's RGB image and depth map yields a set of spatial physical poses. Predictive analysis of the set of spatial physical poses yields a spatial pose prediction vector function. This spatial pose prediction vector function is then filtered, and the results are used as the endpoint target boundary. Finally, reconstruction analysis of the current real-time position set, current real-time velocity set, current real-time acceleration set, and endpoint target boundary yields a spatial pose reconstruction vector function. The spatial pose reconstruction vector function is solved and analyzed, and the robotic arm is controlled based on the solution analysis results. This solves the technical problems of control phase lag, reciprocating oscillation, and frequent missed grasping caused by the superposition of camera processing delay and mechanical response lag in traditional pose feedback tracking systems. It realizes high-precision, soft contact, and non-destructive collaborative harvesting of dynamically low-frequency forced oscillation fruits under natural field conditions, eliminates the phenomenon of reciprocating oscillation and overshoot of the robotic arm, ensures perfect soft contact when the gripper fingertips close and wrap the fruit, avoids any local normal or tangential physical impact loads, and significantly improves the success rate of collaborative tracking and harvesting and the fruit non-destruction rate under natural field perturbation environment.

[0019] Optionally, as an embodiment of the present invention, such as Figure 1 and 2As shown, the process of performing pose analysis on each of the fruit RGB images and the corresponding fruit depth maps to obtain a set of spatial physical poses includes: The target detection model is used to solve the RGB images of each fruit to obtain the two-dimensional pixel center coordinates corresponding to each RGB image of the fruit. The two-dimensional pixel center coordinates include the X-axis two-dimensional pixel center coordinates and the Y-axis two-dimensional pixel center coordinates. Extract the depth values ​​corresponding to the RGB images of each fruit from the depth maps of each fruit respectively; The X-axis image coordinate system origin offset, Y-axis image coordinate system origin offset, X-axis camera imaging scaling ratio, and Y-axis camera imaging scaling ratio are extracted from the RGB-D depth camera. The first formula is used to calculate the X-axis three-dimensional spatial coordinates corresponding to each fruit RGB image by calculating the origin offset of the X-axis image coordinate system, the X-axis camera imaging scaling ratio, the center coordinates of each X-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image. , in, For the first Time of the first The X-axis three-dimensional spatial coordinates corresponding to the RGB image of each fruit For the first The X-axis two-dimensional pixel center coordinates corresponding to the RGB image of each fruit This is the offset of the origin of the X-axis image coordinate system. This refers to the X-axis camera imaging scaling ratio. For the first Time of the first The depth value corresponding to the RGB image of each fruit; The second equation is used to calculate the offset of the origin of the Y-axis image coordinate system, the Y-axis camera imaging scaling ratio, the center coordinates of each Y-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image, to obtain the Y-axis three-dimensional spatial coordinates corresponding to each fruit RGB image. A three-dimensional absolute spatial pose set is constructed using all the X-axis three-dimensional spatial coordinates, all the Y-axis three-dimensional spatial coordinates, and all the depth values. The second equation is: , in, For the first Time of the first The Y-axis three-dimensional spatial coordinates corresponding to the RGB image of the fruit For the first The Y-axis two-dimensional pixel center coordinates corresponding to the RGB image of the fruit This is the offset of the origin of the Y-axis image coordinate system. The scaling factor for the camera image on the Y-axis. For the first Time of the first The depth value corresponding to the RGB image of each fruit; Import the homogeneous rotation and translation matrix and multiple joint absolute rotation angles, solve for all the joint absolute rotation angles, and obtain the homogeneous transformation matrix; The spatial physical pose set is obtained by calculating the homogeneous rotation and translation matrix, the homogeneous transformation matrix, and the three-dimensional absolute spatial pose set using the third equation. The third equation is: , in, For the first The set of spatial physical poses at any given time. For the first The homogeneous transformation matrix at time step, It is a homogeneous rotation and translation matrix. For the first The set of three-dimensional absolute spatial poses at any given time.

[0020] Preferably, the target detection model can be a YOLOv11s branch network.

[0021] It should be understood that the current moment collected... The RGB image (i.e., the RGB image of the fruit) is used to perform lightweight object detection inference (such as a YOLOv11s branch network) to calculate the two-dimensional pixel center coordinates of the target fruit region. Subsequently, the vision controller extracts the depth values ​​of the corresponding pixel coordinates from the depth map. .

[0022] Specifically, using a pre-calibrated camera intrinsic parameter matrix (including focal length) Coordinates of the principal point of the optical center (i.e., the offset of the origin of the X-axis image coordinate system, the offset of the origin of the Y-axis image coordinate system, the X-axis camera imaging scaling ratio, and the Y-axis camera imaging scaling ratio), which transforms the feature points in the two-dimensional space of the image into the three-dimensional absolute spatial pose in the camera coordinate system through real-time inverse projection. (i.e., a set of poses in three-dimensional absolute space). Its mathematical transformation equation is: , , To eliminate the dynamic deviation disturbance of the robotic arm's own movement on the camera's field of view, a joint absolute encoder is used. High-frequency refresh rate synchronous reading of the current moment The absolute rotation angles of each joint of the robotic arm are calculated using standard forward kinematics to obtain the homogeneous transformation matrix of the robotic arm's end effector flange relative to the robot's base coordinate system at the current moment. .

[0023] Specifically, the fixed homogeneous transformation matrix of the camera relative to the end flange obtained from pre-calibrated hand-eye alignment is fused. (i.e., homogeneous transformation matrix) to construct a unified dynamic spatiotemporal pose synchronization operator. This involves synchronizing the fruit's swaying pose in the camera coordinate system. (i.e., the set of three-dimensional absolute spatial poses) is converted into the absolute spatial physical pose in the global static robot base coordinate system. (i.e., the set of spatial physical poses), as shown in the following formula: , like Figure 2 As shown, it will include a high-resolution absolute hardware clock stamp. Spatial coordinates of the robot base coordinate system Multidimensional feature data The data is sent to the timing sliding window register to complete the high-frequency sensing stage.

[0024] In the above embodiments, pose analysis is performed on each fruit RGB image and fruit depth map to obtain a set of spatial physical poses. This eliminates the influence of the robot arm's own motion on the dynamic deviation of the camera's viewpoint and solves the technical problems of control phase lag, reciprocating oscillation, and frequent misses caused by the superposition of camera processing delay and mechanical response lag in traditional pose feedback tracking systems.

[0025] Optionally, as an embodiment of the present invention, such as Figure 3 As shown, the process of performing predictive analysis on the set of spatial physical poses to obtain the spatial pose prediction vector function includes: Multiple filtered spatial physical poses are selected from the set of spatial physical poses by using a preset dynamic spatiotemporal moving sliding window, and all filtered spatial physical poses are combined to obtain a set of filtered spatial physical poses. Construct X-axis time series parameter fitting models, Y-axis time series parameter fitting models, and Z-axis time series parameter fitting models. The X-axis time series parameter fitting model is as follows: , The Y-axis time series parameter fitting model is as follows: , The Z-axis time series parameter fitting model is as follows: , in, For the first The theoretical spatial position of the X-axis at time [time]. The regression coefficients for the static equilibrium position on the X-axis. The initial X-axis excitation amplitude, The X-axis air damping coefficient is... The natural principal angular frequency of the X-axis. The initial X-axis phase, For the first The theoretical spatial position of the Y-axis at time t. The regression coefficients for the static equilibrium position on the Y-axis. The initial Y-axis excitation amplitude, The Y-axis air damping coefficient is... The natural principal angular frequency of the Y-axis. The initial Y-axis phase, For the first The theoretical spatial position of the Z-axis at time t. The regression coefficients for the static equilibrium position along the Z-axis. The initial Z-axis excitation amplitude, The Z-axis air damping coefficient is... The natural principal angular frequency of the Z-axis. The initial Z-axis phase; The X-axis coordinates, Y-axis coordinates, and Z-axis coordinates of the fruit were extracted from the filtered set of spatial physical poses. The objective function for the X-axis is obtained by calculating the actual observed X-axis coordinates of the fruit, the regression coefficient of the static equilibrium position of the X-axis, the initial X-axis excitation amplitude, the X-axis air damping coefficient, the natural principal angular frequency of the X-axis, and the initial X-axis phase using the fourth equation. The fourth equation is as follows: , in, The objective function is the X-axis. The regression coefficients for the static equilibrium position on the X-axis. The initial X-axis excitation amplitude, The number of spatial physical poses after filtering. The X-axis air damping coefficient is... The natural principal angular frequency of the X-axis. The initial X-axis phase, For the first The actual X-axis coordinate of the fruit at that moment. For L2 norm squaring operations; The objective function for the Y-axis is obtained by calculating the actual observed Y-axis coordinates of the fruit, the regression coefficient of the static equilibrium position of the Y-axis, the initial Y-axis excitation amplitude, the Y-axis air damping coefficient, the natural principal angular frequency of the Y-axis, and the initial Y-axis phase using the fifth equation. The fifth equation is as follows: , in, The objective function is the one along the Y-axis. For the first The actual Y-axis coordinate of the fruit at that moment. The regression coefficients for the static equilibrium position on the Y-axis. The initial Y-axis excitation amplitude, The Y-axis air damping coefficient is... The natural principal angular frequency of the Y-axis. The initial Y-axis phase, This represents the number of spatial physical poses. For L2 norm squaring operations; The Z-axis objective function is obtained by calculating the actual observed Z-axis coordinates of the fruit, the regression coefficient of the static equilibrium position of the Z-axis, the initial Z-axis excitation amplitude, the Z-axis air damping coefficient, the natural principal angular frequency of the Z-axis, and the initial Z-axis phase using the sixth equation. The sixth equation is as follows: , in, The objective function is the Z-axis. For the first The actual Z-axis coordinate of the fruit at that moment. The regression coefficients for the static equilibrium position along the Z-axis. The initial Z-axis excitation amplitude, The Z-axis air damping coefficient is... The natural principal angular frequency of the Z-axis. The initial Z-axis phase, This represents the number of spatial physical poses. For L2 norm squaring operations; The X-axis time series parameter fitting model is updated according to the X-axis objective function to obtain the updated X-axis time series parameter fitting model. The Y-axis time series parameter fitting model is updated according to the Y-axis objective function to obtain the updated Y-axis time series parameter fitting model. The parameters of the Z-axis time series parameter fitting model are updated according to the Z-axis objective function to obtain the updated Z-axis time series parameter fitting model. The updated X-axis temporal parameter fitting model, the updated Y-axis temporal parameter fitting model, and the updated Z-axis temporal parameter fitting model are solved respectively to obtain the X-axis spatial pose vector, the Y-axis spatial pose vector, and the Z-axis spatial pose vector. A spatial pose prediction vector function is constructed using the X-axis spatial pose vector, the Y-axis spatial pose vector, and the Z-axis spatial pose vector.

[0026] It should be understood that, since natural winds in the field are usually low-frequency and have specific pulsating disturbances, unstructured fruits (such as tomatoes) hanging at the end of the fruit stalk conform to the nonlinear pendulum oscillation model with initial damping and forced excitation characteristics in terms of physical behavior.

[0027] Specifically, a window of length N (e.g., ...) is allocated in the timing sliding window register. The module uses a dynamic spatiotemporal sliding window. Once the accumulated temporal pose data points within the sliding window meet the required number of fitting samples, the module then moves to the robot's base coordinate system. The three spatial axes are used to construct explicit time-dependent nonlinear dynamic sway equations, either independently or in a multidimensional coupled manner.

[0028] With a single axis Taking the shaft as an example, the high-order time-series parameter fitting model, which includes the basic suspension center position compensation term, low-frequency forced principal amplitude term, principal vibration angular frequency, initial phase, and damping term, is constructed as follows: , In the formula, This is the static equilibrium position regression term; This represents the initial excitation amplitude; The coefficient of friction between branches and leaves in the field and the air damping coefficient; The inherent principal angular frequency is determined by the physical length of the fruit stalk and the acceleration due to gravity. This is the initial phase.

[0029] Specifically, to achieve fast and accurate online calculations at the millisecond level, the sliding nonlinear least squares method (Sliding Levenberg-Marquardt, LM algorithm) or a higher-order recursive extended Kalman filter (EKF) is used to perform online residual minimization iterative solutions for N pairs of historical spatiotemporal coordinate data within the sliding window. The optimization objective function is set as follows: , Through iterative correction of the Jacobian matrix using the LM algorithm, at the current time... Real-time calculation of the optimal set of core estimates for the dynamic model parameters at the current moment. At this point, the forward timestamp t of the target fruit in any future time domain (where...) Predicted continuous spatial pose vector (i.e., the spatial pose prediction vector function) has a definite analytical mathematical expression as follows: .

[0030] In the above embodiments, predictive analysis is performed on the spatial physical pose set to obtain the spatial pose prediction vector function, which realizes millisecond-level online fast and accurate calculation. This enables high-precision, soft-contact, and non-destructive harvesting of dynamically low-frequency forced oscillation fruits under natural field conditions, eliminating the phenomena of reciprocating oscillation and overshoot of the robotic arm.

[0031] Optionally, as an embodiment of the present invention, such as Figure 4 As shown, the process of performing a screening analysis on the spatial pose prediction vector function and using the screening analysis result as the endpoint target boundary includes: Taking the first derivative of the spatial pose prediction vector function yields the swaying velocity vector function; The minimum point equation is constructed using the swaying velocity vector function. The minimum point equation is as follows: , in, For the first The sway velocity vector function at time t, For L2 norm squaring operations; The minimum point equation is solved according to the preset screening conditions to obtain the golden capture window time; Based on the golden grasping window time, the spatial predicted three-dimensional coordinates, spatial predicted velocity vector, and spatial predicted acceleration vector are extracted from the shaking velocity vector function; The endpoint target boundary is constructed using the golden capture window time, the spatially predicted three-dimensional coordinates, the spatially predicted velocity vector, and the spatially predicted acceleration vector.

[0032] Understandably, after establishing a continuous and predictable equation of motion, the decision module obtains the continuous absolute swaying velocity vector function of the target fruit in the future time domain by taking the first derivative with respect to time t. : , To completely eliminate the problem of rigid impact kinetic energy damage to the fruit caused by traditional forced intersection interception control, this invention proposes a "Relative Zero-Velocity Grasping Window" discrimination mechanism. This mechanism identifies the "highest inflection point time stamp of the swing direction" where the absolute velocity of the target fruit drops to a minimum due to a reversal of its swing direction, meaning the system's kinetic energy is completely converted into potential energy. At these inflection points, although the fruit's spatial acceleration is not zero, its instantaneous velocity vector magnitude approaches zero.

[0033] Specifically, through the next one and a half cycles ( Perform a high-frequency numerical scan over the time domain to solve the minimum point equation (i.e., the minimum point equation) of the velocity scalar norm, as follows: , The solution filters out the points on the future timeline that are a distance from the current time. The most recent minimum point that satisfies the physical reachability time constraint of the robotic arm is defined as the golden grasping window timestamp. (i.e., the golden window of opportunity).

[0034] once (i.e., the golden capture window time) is locked, and the three-dimensional absolute coordinates of the fruit space corresponding to that timestamp are immediately predicted. (i.e., spatial prediction of three-dimensional coordinates), prediction of velocity vector (i.e., spatial predicted velocity vector) and predicted acceleration vector (i.e., spatially predicted acceleration vector) is latched in the spatiotemporal state. These parameters will be used as unique endpoint boundary conditions with hard constraints on higher-order kinematic boundaries, and will be sent to the trajectory for reconstruction.

[0035] In the above embodiments, the spatial pose prediction vector function is screened and analyzed, and the screening and analysis results are used as the endpoint target boundary. This completely eliminates the problem of rigid impact kinetic energy damaging the fruit caused by traditional forced intersection interception control, and realizes high-precision, soft contact and non-destructive collaborative harvesting of dynamic low-frequency forced oscillation fruits under natural field conditions.

[0036] Optionally, as an embodiment of the present invention, the real-time position group of the robotic arm end effector at the current moment includes the real-time position of the robotic arm end effector on the X-axis, the real-time position of the robotic arm end effector on the Y-axis, and the real-time position of the robotic arm end effector on the Z-axis at the current moment; the real-time velocity group of the robotic arm end effector at the current moment includes the real-time velocity of the robotic arm end effector on the X-axis, the real-time velocity of the robotic arm end effector on the Y-axis, and the real-time velocity of the robotic arm end effector on the Z-axis at the current moment; the real-time acceleration group at the current moment includes the real-time acceleration of the X-axis, the real-time acceleration of the Y-axis, and the real-time acceleration of the Z-axis at the current moment; and the endpoint target boundary includes the predicted three-dimensional coordinates of the X-axis, the predicted three-dimensional coordinates of the Y-axis, the predicted three-dimensional coordinates of the Z-axis, the predicted velocity vector of the X-axis, the predicted velocity vector of the Y-axis, the predicted velocity vector of the Z-axis, the predicted acceleration vector of the X-axis, the predicted acceleration vector of the Y-axis, and the predicted acceleration vector of the Z-axis. The process of reconstructing and analyzing the real-time position set, real-time velocity set, and real-time acceleration set of the robotic arm end effector at the current moment, as well as the boundary of the target endpoint, to obtain the spatial pose reconstruction vector function includes: An X-axis boundary condition matrix is ​​constructed using the current real-time position of the X-axis robotic arm end effector, the current real-time velocity of the X-axis robotic arm end effector, the current real-time X-axis acceleration, the predicted 3D coordinates in X-axis space, the predicted velocity vector in X-axis space, and the predicted acceleration vector in X-axis space. The X-axis boundary condition matrix is ​​as follows: , The Y-axis boundary condition matrix is ​​constructed using the current real-time position of the Y-axis robotic arm end effector, the current real-time velocity of the Y-axis robotic arm end effector, the current real-time acceleration of the Y-axis, the predicted 3D coordinates of the Y-axis space, the predicted velocity vector of the Y-axis space, and the predicted acceleration vector of the Y-axis space. The Y-axis boundary condition matrix is ​​as follows: , The Z-axis boundary condition matrix is ​​constructed using the current real-time position of the Z-axis robotic arm end effector, the current real-time velocity of the Z-axis robotic arm end effector, the current real-time acceleration of the Z-axis, the predicted three-dimensional coordinates of the Z-axis, the predicted velocity vector of the Z-axis, and the predicted acceleration vector of the Z-axis. The Z-axis boundary condition matrix is ​​as follows: , in, , in, For the first The real-time position of the X-axis robotic arm end effector at any given moment. For the first Real-time speed of the X-axis robotic arm end effector at any given moment. For the first Real-time X-axis acceleration at time t. For the first Predict the three-dimensional coordinates of the X-axis at time [time]. For the first The predicted velocity vector along the X-axis at time t. For the first The predicted spatial acceleration vector along the X-axis at time t. , , , , as well as All are X-axis control coefficient vectors. For the first The real-time position of the Y-axis robotic arm end effector at any given moment. For the first Real-time speed of the Y-axis robotic arm end effector at any given moment. For the first Real-time Y-axis acceleration at time t. For the first Predict the three-dimensional coordinates of the Y-axis at time t. For the first The predicted spatial velocity vector along the Y-axis at time t. For the first The spatially predicted acceleration vector along the Y-axis at time t. , , , , as well as All are Y-axis control coefficient vectors. For the first The real-time position of the Z-axis robotic arm end effector at any given moment. For the first Real-time speed of the Z-axis robotic arm end effector at any given moment. For the first Real-time Z-axis acceleration at time t. For the first Predict the three-dimensional coordinates of the Z-axis at time [time]. For the first The spatially predicted velocity vector along the Z-axis at time t. For the first The spatially predicted acceleration vector along the Z-axis at time t. , , , , as well as All are Z-axis control coefficient vectors. To capture the golden window of opportunity, For the current moment, This is the net time-domain window length; Solve the X-axis boundary condition matrix, the Y-axis boundary condition matrix, and the Z-axis boundary condition matrix respectively to obtain multiple X-axis control coefficient vectors, multiple Y-axis control coefficient vectors, and multiple Z-axis control coefficient vectors; The seventh equation is used to calculate the multiple X-axis control coefficient vectors to obtain the... The X-axis spatial pose reconstruction vector at time t, the seventh equation is: , in, For the first The X-axis spatial pose reconstruction vector at time t. , , , , as well as All are X-axis control coefficient vectors. For the first time, For the first time; The eighth equation is used to calculate the multiple Y-axis control coefficient vectors to obtain the... The Y-axis spatial pose reconstruction vector at time t is given by the eighth equation: , in, For the first The Y-axis spatial pose reconstruction vector at time t. , , , , as well as All are Y-axis control coefficient vectors. For the first time, For the first time; The ninth equation is used to calculate the multiple Z-axis control coefficient vectors, resulting in the... The Z-axis spatial pose reconstruction vector at time t is given by the ninth equation: , in, For the first The Z-axis spatial pose reconstruction vector at time t. , , , , as well as All are Z-axis control coefficient vectors. For the first time, For the first time; Through the first The X-axis spatial pose reconstruction vector at time t, the first The Y-axis spatial pose reconstruction vector at time t and the first The spatial pose reconstruction vector function is constructed by reconstructing the Z-axis spatial pose reconstruction vector at time t.

[0037] It should be understood that the endpoint target boundary of the latch is obtained. Subsequently, instead of using traditional follow-up posterior feedback control, active trajectory planning based on feedforward compensation is initiated.

[0038] To ensure that when the robotic arm reaches the preset grasping spatiotemporal point, its end gripper not only perfectly coincides with the fruit in geometric position, but also maintains absolute synchronization with the fruit's swaying behavior in terms of higher-order derivatives such as velocity and acceleration (i.e., the spatial relative velocity and relative acceleration both return to zero), a fifth-order polynomial trajectory reconstruction is performed in the robotic arm's Cartesian space or joint space.

[0039] Specifically, taking X-axis planning in Cartesian space as an example, let's set the end effector of the robotic arm from the current moment... At the target interception window The spatial motion equations (i.e., the X-axis boundary condition matrix) are: , Extract the current real-time front boundary conditions (i.e.) Real-time position of the robotic arm end effector Real-time speed Real-time acceleration (i.e., the real-time position of the X-axis robotic arm end effector, the real-time velocity of the X-axis robotic arm end effector, and the real-time acceleration of the X-axis at the current moment), and the latched future time domain endpoint boundary conditions (i.e. The position of the fruit shaking at that time ,speed acceleration (i.e., the endpoint target boundary), construct the boundary condition matrix of the linear equation system containing 6 unknown coefficients, as follows: , In the formula, This is the net time-domain window length (i.e., the net time-domain window length) left for the robotic arm to dynamically execute the feedforward sprint.

[0040] By performing millisecond-level algebraic solutions on the Vandermonde inverse matrix, a unique sixth-order control coefficient vector is directly determined. (i.e., X-axis control coefficient vector, Y-axis control coefficient vector, or Z-axis control coefficient vector). Through this feedforward compensation mechanism, the motion trajectory of the robotic arm's end effector is completely reconstructed, ensuring that the control commands overcome the interference caused by physical lag at the source.

[0041] In the above embodiments, the real-time position group, real-time velocity group, real-time acceleration group, and target boundary of the robotic arm at the current moment are reconstructed and analyzed to obtain the spatial pose reconstruction vector function. This ensures that the control command overcomes the interference caused by physical lag at the source, enabling the robotic arm end gripper to not only perfectly coincide with the fruit in geometric position, but also maintain absolute synchronization with the fruit's swaying behavior in terms of higher-order derivatives such as velocity and acceleration.

[0042] Optionally, as an embodiment of the present invention, such as Figure 5 As shown, the process of solving and analyzing the spatial pose reconstruction vector function and controlling the robotic arm based on the solution analysis results includes: The spatial pose reconstruction vector function is solved to obtain multiple joint motor feedforward commands, and the robotic arm is controlled according to all the joint motor feedforward commands.

[0043] Understandably, the reconstructed Cartesian kinematics curve (i.e., the spatial pose reconstruction vector function) is solved in real time by the inverse kinematics of the robotic arm and converted into a continuous drive position command sequence for the six joint motors and a high-order feedforward joint velocity curve (i.e., joint motor feedforward command), which is then sent down to the underlying joint servo driver.

[0044] Specifically, in the time domain span Inside, the drive motors of each joint of the robotic arm are controlled by a three-loop servo circuit of current, speed and position, which controls the end gripper to approach the preset interception time and space window at full speed with a smooth, continuous and oscillating dynamic response curve.

[0045] When the absolute hardware master clock strictly evolves to the point of locking the golden timestamp Instantly, the control system achieves the following dynamically coordinated convergence state across all dimensions: Spatial position vector alignment: That is, the center of the claw's fingertip envelope is physically aligned with the geometric center of the target fruit; Velocity vectors converge at the same velocity: That is, the absolute velocity of the grippers at the moment of intersection is exactly equal to the instantaneous zero velocity of the fruit when it swings to its highest point.

[0046] Due to the relative motion velocity vector The robotic arm's end gripper and the fragile fruit exhibit an almost absolute "quasi-static relative zero-speed intersection" in three-dimensional physical space.

[0047] At this point, a high-response grasping closure pulse is sent to the end effector via the bus, and the fingertips of the adaptive electric / pneumatic gripper at the end effector close and wrap around the target fruit in two or more directions. Since the relative velocity is zero, no rigid collision shear stress or high-frequency pulse impact load is generated on the contact surface, achieving perfect "soft contact".

[0048] Understandably, after the gripper securely holds the target fruit, the robotic arm joint controller automatically and seamlessly takes over and triggers subsequent smooth retraction torsion or lifting torque planning. The drive mechanism pulls the fruit off the stem according to a preset high-order smooth transition trajectory, completely avoiding stem tearing or secondary extrusion and abrasion of the skin caused by traditional delayed strong pulling, and efficiently completing the fully closed-loop non-destructive intelligent harvesting task in a dynamic perturbation environment in the field.

[0049] In the above embodiments, the spatial pose reconstruction vector function is solved and analyzed, and the robotic arm is controlled according to the solution analysis results. This completely avoids the fruit stem tearing or secondary squeezing and abrasion of the skin caused by traditional hysteresis strong pulling, and efficiently completes the fully closed-loop non-destructive intelligent harvesting task in the dynamic perturbation environment of the field.

[0050] Optionally, as another embodiment of the present invention, the present invention first acquires RGB images and depth maps of the swaying fruit at high frequency and synchronously, constructs a dynamic pose synchronization operator to calculate the absolute spatial physical pose of the fruit in the robot base coordinate system; then, within a sliding window of movement time, a nonlinear simple harmonic oscillation dynamic equation of a pendulum is constructed, and the parameters are solved online iteratively using the sliding least squares method to establish a forward continuous predicted spatial pose vector; the first derivative of the predicted equation is obtained to obtain the absolute swaying velocity function, and the velocity scalar norm minimum equation is solved by online numerical scanning to lock and store the golden grasping window timestamp of the highest inflection point of the fruit's swing direction; finally, this is used as a high-order kinematic boundary hard constraint, and by performing algebraic inverse solution on the boundary condition matrix, a fifth-order polynomial feedforward trajectory in Cartesian space is reconstructed in real algebra to distribute control commands. The present invention eliminates the control phase lag caused by the accumulation of perception calculation and mechanism physical response, and realizes high-order smooth and continuous quasi-static relative zero-velocity intersection lossless grasping throughout the entire stroke.

[0051] Optionally, as another embodiment of the present invention, the proposal of the present invention aims to solve the major technical bottlenecks of the prior art when facing dynamic swaying targets in complex unstructured farmland, such as "tracking phase lag", "linear misjudgment of motion model" and "large damage caused by transient collision stress". It provides a flexible and non-destructive grasping method that can achieve high-precision prediction of dynamic swaying trajectory, accurate discrimination of relative zero velocity time window and active feedforward compensation and collaborative tracking of end effector under natural wind deflection disturbance conditions.

[0052] Optionally, as another embodiment of the present invention, the technical problem to be solved by the present invention includes: How to construct a dynamic prediction model for the nonlinear swaying trajectory of fruit based on multi-frame temporal spatial coordinate fusion: to solve the technical problems of control phase lag, reciprocating oscillation and frequent misses caused by the superposition of camera processing delay and mechanical response lag in traditional pose feedback tracking systems.

[0053] How to accurately predict and lock the "relative zero-speed grasping time-space window timestamp" between the swinging fruit and the end gripper of the robotic arm: solve the intersection and truncation error caused by the blind simplification of linear motion, and the technical defects of traditional interception control that cause instantaneous physical impact damage to the fruit, fruit peel cracking and fruit stem tearing when the relative kinetic energy is too large when the intersection is not in the preset time domain.

[0054] How to design a multi-order feedforward compensation kinematic trajectory reconstruction loop that can be dynamically corrected online: guide the absolute velocity vector and position space vector of the robotic arm and end effector to achieve high-order smooth three-dimensional spatial synchronization with the nonlinear velocity of the fruit's swaying within the predicted zero-velocity grasping window, and solve the defects of passive tracking control such as large rigid collision stress, low flexible non-damage rate, and poor robustness in field engineering.

[0055] Optionally, as another embodiment of the present invention, the present invention fundamentally eliminates the phase deviation caused by the machine vision processing delay and the response lag of the robotic arm power actuator by constructing a swaying trajectory prediction operator based on time series physical modeling, an online discrimination logic for relative zero velocity time windows, and a multi-order kinematic trajectory active reconstruction loop with time-domain feedforward compensation. This enables high-precision, soft-contact, and non-destructive collaborative harvesting of dynamically low-frequency forced oscillation fruits under natural field conditions.

[0056] Alternatively, as another embodiment of the present invention, such as Figure 6 and 7 As shown, the present invention includes: The advanced multimodal temporal perception brain module physically comprises an eye-in-hand RGB-D depth camera mounted at the end of a six-DOF robotic arm, and an edge computing terminal connected to it via a high-frequency gigabit Ethernet bus (GigE Vision). This module is responsible for high-frequency acquisition of two-dimensional image features and three-dimensional point cloud flows in a field environment, and outputs the geometric features of the fruit frame by frame through a deep feature extraction network.

[0057] Motion equation solving and spatiotemporal prediction module: The physical carrier is a high-performance edge-side industrial host computer or microprocessor. This module integrates a nonlinear dynamics filtering operator, a time-series physical state sliding window, and a spatiotemporal latch register. It is used to receive high-frequency spatial coordinates sent by the sensory brain and to fit and derive the nonlinear low-frequency swaying motion equation of the fruit online in the time domain.

[0058] Temporal feedforward motion planning and trajectory reconstruction module: The physical carrier is the trajectory generator of the main control central processing unit. This module receives the predicted timestamp and spatial state vector output by the spatiotemporal prediction module, and calculates and actively reconstructs the kinematic control curves of each joint of the robotic arm in real time by solving the high-order polynomial boundary value problem.

[0059] The synchronous convergence servo execution module includes the robotic arm's underlying joint servo drivers, a high-resolution absolute encoder, and an end-effector flexible adaptive electric / pneumatic gripper. This module receives feedforward reconstructed motion commands and drives the motors of each joint of the robotic arm to perform high-precision position-velocity composite servo responses, achieving full-dimensional velocity vector alignment and enveloping grasping of the wobbling fruit within a preset spatial pose.

[0060] Optionally, as another embodiment of the present invention, this invention completely solves the control phase lag caused by the superposition of machine vision processing time and physical inertial delay of mechanism power response from the perception and prediction level. This invention enables the control system to possess "forward-looking predictive cognition" of dynamically swaying targets, allowing it to perceive every spatiotemporal coordinate point of the fruit in the future time domain in advance, eliminating the reciprocating oscillation and overshoot phenomenon of the robotic arm, and providing a core mathematical solution framework for high-precision trajectory feedforward interception.

[0061] Optionally, as another embodiment of the present invention, the present invention elevates the traditional "blind spatial geometric intersection interception" to "relative zero-speed convergence with relative velocity closed-loop control" at the decision-making logic level. By strictly locking the picking and grasping action at the moment of the physical inflection point where the absolute swing speed of the fruit is at its lowest, the transient relative kinetic energy impact during the intersection collision is fundamentally eliminated, reducing the traditional high-risk dynamic hard collision to "quasi-static relative zero-speed intersection", providing the safest time-domain execution window for the protection of vulnerable organisms.

[0062] Alternatively, as another embodiment of the present invention, the present invention constructs a high-order continuous closed-loop feedforward correction loop at the underlying execution control level, guiding the end effector gripper of the robotic arm to evolve strictly to the golden time stamp. At that instant, the "perfect alignment of position space vectors" and the "convergence of absolute velocity vectors at the same speed (relative velocity vectors)" are achieved synchronously with the fruit. The dual synergistic state ensures perfect flexible "soft contact" when the claw tips close and wrap around the fruit, avoiding any local normal or tangential physical impact loads, and greatly improving the success rate of synergistic tracking and harvesting and the fruit damage-free rate under natural field disturbance environments.

[0063] Optionally, as another embodiment of the present invention, the present invention addresses the fatal defects of existing technologies in unstructured field environments when harvesting fruits in a low-frequency nonlinear resonant swaying state due to natural wind disturbance, such as "severe phase lag caused by posterior reaction control", "large truncation error caused by blind linearization of the motion model", and "excessive relative kinetic energy at the moment of intersection causing severe collision damage to the fruit". It proposes an embodied servo control scheme based on time-series prediction, steering latching and high-order feedforward trajectory reconstruction.

[0064] Alternatively, as another embodiment of the present invention, compared with the prior art, the present invention enables the system to have the ability to anticipate the future full-time-domain position evolution of a swaying target before the robotic arm initiates its approach stroke. This overcomes the cascading delay caused by camera inference time and the physical inertia of the robotic arm joints from a mathematical perspective. This completely eliminates the overshoot and high-frequency reciprocating oscillation of the control system, improves the alignment success rate of the spatial envelope window to an industrial-grade robust standard, and avoids defects such as the robotic arm accidentally scratching the fruit, hitting surrounding branches and leaves, or frequently missing its target.

[0065] Alternatively, as another embodiment of the present invention, compared with the prior art, the present invention strictly limits the timing of the capture to the golden window when the fruit swings at its lowest speed. This avoids the catastrophic consequences of collision when the fruit swings through the equilibrium position (the region of maximum absolute speed and peak kinetic energy) from the source of the control logic, completely eliminating the huge normal impact and tangential shear stress generated by high relative kinetic energy at the moment of contact, and protecting the cell tissues of the fragile and tender organism from being crushed, broken, or torn by the fruit stalk.

[0066] Alternatively, as another embodiment of the present invention, compared with the prior art, the feedforward reconfiguration loop of the present invention ensures that the end effector gripper of the robotic arm evolves strictly to the golden timestamp. At that instant, the "perfect alignment of position space vectors" and the "convergence of absolute velocity vectors at the same speed (relative velocity vectors)" are achieved synchronously with the fruit. The dual collaborative state of "" and "" is achieved. Throughout the entire process of the gripper's fingertips closing and wrapping the fragile fruit, the control signal and physical waveform maintain a high-order continuity, realizing a near-static flexible "soft contact". This avoids any local normal or tangential physical impact loads, and significantly improves the success rate of collaborative tracking and harvesting and the fruit damage-free rate under natural field disturbance environments.

[0067] Optionally, as another embodiment of the present invention, the present invention includes: Turning Point & Relative Zero-VelocityWindow: Under natural wind disturbance conditions, the swing of an inverted fruit conforms to a nonlinear pendulum model. During its trajectory, the fruit reaches its maximum absolute velocity and kinetic energy when it passes through the central equilibrium position, making it highly susceptible to shattering during interception. The "relative zero-velocity grabbing window" proposed in this invention specifically refers to the instantaneous turning inflection point where the fruit swings to both ends of its amplitude and the velocity vector direction reverses. At this instant, the absolute swing velocity scalar norm of the fruit reaches a minimum (approaching zero), its physical essence being the transformation of the high-risk dynamic intersection process into a "quasi-static relative zero-velocity in-situ envelopment."

[0068] Posterior passive feedback and active feedforward compensation: Existing visual servoing technologies (IBVS / PBVS) employ a posteriori passive feedback, meaning the robotic arm only passively pursues the target after detecting a change in its position. This inevitably leads to cumulative phase lag caused by "network inference time + mechanical inertia delay of multi-joint motors." This invention predicts the future spatiotemporal intersection endpoint through timing equations, and directly uses a fifth-order polynomial matrix inverse calculation to implement active feedforward control of the robotic arm, enabling the control commands to directly overcome and offset physical time delay interference.

[0069] Velocity Vector Convergence (VoVVC Convergence) across all dimensions: Not only refers to the gripper at the end of the robotic arm Reaching the precise geometric three-dimensional point where the fruit is located at every moment requires that, at the instant of intersection, the absolute velocity vector of the end gripper itself and the swaying velocity vector of the fruit are completely equal in three-dimensional components (at this point, due to the turning point, the absolute velocities of both are close to zero). This triple coordination of "same speed, same direction, and same position" achieves relative velocity at the instant of contact and collision at the underlying control level. This eliminates local physical shear stress.

[0070] Figure 8This is a block diagram of a robotic arm control device for fruit grasping, provided as an embodiment of the present invention.

[0071] Alternatively, as another embodiment of the present invention, such as Figure 8 As shown, a robotic arm control device for fruit grasping includes: The image acquisition module is used to acquire images of the area to be grasped by using an RGB-D depth camera set on the end eye of the robotic arm, and to obtain multiple RGB images of the fruit and a fruit depth map corresponding to each RGB image of the fruit. The pose analysis module is used to perform pose analysis on each of the fruit RGB images and the fruit depth map corresponding to each of the fruit RGB images to obtain a set of spatial physical poses. The predictive analysis module is used to perform predictive analysis on the set of spatial physical poses to obtain a spatial pose prediction vector function. The filtering and analysis module is used to filter and analyze the spatial pose prediction vector function and use the filtering and analysis results as the endpoint target boundary. The reconstruction analysis module is used to import the current real-time position group, the current real-time velocity group, and the current real-time acceleration group of the robotic arm end effector at the current moment, and to perform reconstruction analysis on the current real-time position group, the current real-time velocity group, the current real-time acceleration group, and the endpoint target boundary to obtain the spatial pose reconstruction vector function. The robotic arm control module is used to solve and analyze the spatial pose reconstruction vector function, and control the robotic arm based on the solution and analysis results.

[0072] Optionally, as an embodiment of the present invention, the pose analysis module is specifically used for: The target detection model is used to solve the RGB images of each fruit to obtain the two-dimensional pixel center coordinates corresponding to each RGB image of the fruit. The two-dimensional pixel center coordinates include the X-axis two-dimensional pixel center coordinates and the Y-axis two-dimensional pixel center coordinates. Extract the depth values ​​corresponding to the RGB images of each fruit from the depth maps of each fruit respectively; The X-axis image coordinate system origin offset, Y-axis image coordinate system origin offset, X-axis camera imaging scaling ratio, and Y-axis camera imaging scaling ratio are extracted from the RGB-D depth camera. The first formula is used to calculate the X-axis three-dimensional spatial coordinates corresponding to each fruit RGB image by calculating the origin offset of the X-axis image coordinate system, the X-axis camera imaging scaling ratio, the center coordinates of each X-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image. , in, For the first Time of the first The X-axis three-dimensional spatial coordinates corresponding to the RGB image of each fruit For the first The X-axis two-dimensional pixel center coordinates corresponding to the RGB image of each fruit This is the offset of the origin of the X-axis image coordinate system. This refers to the X-axis camera imaging scaling ratio. For the first Time of the first The depth value corresponding to the RGB image of each fruit; The second equation is used to calculate the offset of the origin of the Y-axis image coordinate system, the Y-axis camera imaging scaling ratio, the center coordinates of each Y-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image, to obtain the Y-axis three-dimensional spatial coordinates corresponding to each fruit RGB image. A three-dimensional absolute spatial pose set is constructed using all the X-axis three-dimensional spatial coordinates, all the Y-axis three-dimensional spatial coordinates, and all the depth values. The second equation is: , in, For the first Time of the first The Y-axis three-dimensional spatial coordinates corresponding to the RGB image of the fruit For the first The Y-axis two-dimensional pixel center coordinates corresponding to the RGB image of the fruit This is the offset of the origin of the Y-axis image coordinate system. The scaling factor for the camera image on the Y-axis. For the first Time of the first The depth value corresponding to the RGB image of each fruit; Import the homogeneous rotation and translation matrix and multiple joint absolute rotation angles, solve for all the joint absolute rotation angles, and obtain the homogeneous transformation matrix; The spatial physical pose set is obtained by calculating the homogeneous rotation and translation matrix, the homogeneous transformation matrix, and the three-dimensional absolute spatial pose set using the third equation. The third equation is: , in, For the first The set of spatial physical poses at any given time. For the first The homogeneous transformation matrix at time step, It is a homogeneous rotation and translation matrix. For the first The set of three-dimensional absolute spatial poses at any given time.

[0073] Optionally, another embodiment of the present invention provides a robotic arm control system for fruit grasping, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the robotic arm control method for fruit grasping as described above. This system can be a computer or similar system.

[0074] Optionally, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robotic arm control method for fruit grasping as described above.

[0075] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0079] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0080] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A control method for a robotic arm used for fruit grasping, characterized in that, Includes the following steps: The RGB-D depth camera installed on the end eye of the robotic arm acquires images of the area to be grasped, resulting in multiple RGB images of the fruit and a fruit depth map corresponding to each RGB image. Pose analysis is performed on each of the fruit RGB images and the fruit depth map corresponding to each of the fruit RGB images to obtain a set of spatial physical poses; The spatial physical pose set is subjected to predictive analysis to obtain the spatial pose prediction vector function; The spatial pose prediction vector function is screened and analyzed, and the results of the screening and analysis are used as the endpoint target boundary. Import the current real-time position group, the current real-time velocity group, and the current real-time acceleration group of the robotic arm end effector at the current moment. Perform reconstruction analysis on the current real-time position group, the current real-time velocity group, the current real-time acceleration group, and the endpoint target boundary to obtain the spatial pose reconstruction vector function. The spatial pose reconstruction vector function is solved and analyzed, and the robotic arm is controlled based on the solution and analysis results.

2. The robotic arm control method for fruit grasping according to claim 1, characterized in that, The process of performing pose analysis on each of the fruit RGB images and the corresponding fruit depth maps to obtain a set of spatial physical poses includes: The target detection model is used to solve the RGB images of each fruit to obtain the two-dimensional pixel center coordinates corresponding to each RGB image of the fruit. The two-dimensional pixel center coordinates include the X-axis two-dimensional pixel center coordinates and the Y-axis two-dimensional pixel center coordinates. Extract the depth values ​​corresponding to the RGB images of each fruit from the depth maps of each fruit respectively; The X-axis image coordinate system origin offset, Y-axis image coordinate system origin offset, X-axis camera imaging scaling ratio, and Y-axis camera imaging scaling ratio are extracted from the RGB-D depth camera. The first formula is used to calculate the X-axis three-dimensional spatial coordinates corresponding to each fruit RGB image by calculating the origin offset of the X-axis image coordinate system, the X-axis camera imaging scaling ratio, the center coordinates of each X-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image. ， in, For the first Time of the first The X-axis three-dimensional spatial coordinates corresponding to the RGB image of each fruit For the first The X-axis two-dimensional pixel center coordinates corresponding to the RGB image of each fruit This is the offset of the origin of the X-axis image coordinate system. This refers to the X-axis camera imaging scaling ratio. For the first Time of the first The depth value corresponding to the RGB image of each fruit; The second equation is used to calculate the offset of the origin of the Y-axis image coordinate system, the Y-axis camera imaging scaling ratio, the center coordinates of each Y-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image, to obtain the Y-axis three-dimensional spatial coordinates corresponding to each fruit RGB image. A three-dimensional absolute spatial pose set is constructed using all the X-axis three-dimensional spatial coordinates, all the Y-axis three-dimensional spatial coordinates, and all the depth values. The second equation is: ， in, For the first Time of the first The Y-axis three-dimensional spatial coordinates corresponding to the RGB image of the fruit For the first The Y-axis two-dimensional pixel center coordinates corresponding to the RGB image of the fruit This is the offset of the origin of the Y-axis image coordinate system. The scaling factor for the camera image on the Y-axis. For the first Time of the first The depth value corresponding to the RGB image of each fruit; Import the homogeneous rotation and translation matrix and multiple joint absolute rotation angles, solve for all the joint absolute rotation angles, and obtain the homogeneous transformation matrix; The spatial physical pose set is obtained by calculating the homogeneous rotation and translation matrix, the homogeneous transformation matrix, and the three-dimensional absolute spatial pose set using the third equation. The third equation is: ， in, For the first The set of spatial physical poses at any given time. For the first The homogeneous transformation matrix at time step, It is a homogeneous rotation and translation matrix. For the first The set of three-dimensional absolute spatial poses at any given time.

3. The robotic arm control method for fruit grasping according to claim 1, characterized in that, The process of performing predictive analysis on the set of spatial physical poses to obtain the spatial pose prediction vector function includes: Multiple filtered spatial physical poses are selected from the set of spatial physical poses by using a preset dynamic spatiotemporal moving sliding window, and all filtered spatial physical poses are combined to obtain a set of filtered spatial physical poses. Construct X-axis time series parameter fitting models, Y-axis time series parameter fitting models, and Z-axis time series parameter fitting models. The X-axis time series parameter fitting model is as follows: ， The Y-axis time series parameter fitting model is as follows: ， The Z-axis time series parameter fitting model is as follows: ， in, For the first The theoretical spatial position of the X-axis at time [time]. The regression coefficients for the static equilibrium position on the X-axis. The initial X-axis excitation amplitude, The X-axis air damping coefficient is... The natural principal angular frequency of the X-axis. The initial X-axis phase, For the first The theoretical spatial position of the Y-axis at time t. The regression coefficients for the static equilibrium position on the Y-axis. The initial Y-axis excitation amplitude, The Y-axis air damping coefficient is... The natural principal angular frequency of the Y-axis. The initial Y-axis phase, For the first The theoretical spatial position of the Z-axis at time t. The regression coefficients for the static equilibrium position along the Z-axis. The initial Z-axis excitation amplitude, The Z-axis air damping coefficient is... The natural principal angular frequency of the Z-axis. The initial Z-axis phase; The X-axis coordinates, Y-axis coordinates, and Z-axis coordinates of the fruit were extracted from the filtered set of spatial physical poses. The objective function for the X-axis is obtained by calculating the actual observed X-axis coordinates of the fruit, the regression coefficient of the static equilibrium position of the X-axis, the initial X-axis excitation amplitude, the X-axis air damping coefficient, the natural principal angular frequency of the X-axis, and the initial X-axis phase using the fourth equation. The fourth equation is as follows: ， in, The objective function is the X-axis. The regression coefficients for the static equilibrium position on the X-axis. The initial X-axis excitation amplitude, The number of spatial physical poses after filtering. The X-axis air damping coefficient is... The natural principal angular frequency of the X-axis. The initial X-axis phase, For the first The actual X-axis coordinate of the fruit at that moment. For L2 norm squaring operations; The objective function for the Y-axis is obtained by calculating the actual observed Y-axis coordinates of the fruit, the regression coefficient of the static equilibrium position of the Y-axis, the initial Y-axis excitation amplitude, the Y-axis air damping coefficient, the natural principal angular frequency of the Y-axis, and the initial Y-axis phase using the fifth equation. The fifth equation is as follows: ， in, The objective function is the one along the Y-axis. For the first The actual Y-axis coordinate of the fruit at that moment. The regression coefficients for the static equilibrium position on the Y-axis. The initial Y-axis excitation amplitude, The Y-axis air damping coefficient is... The natural principal angular frequency of the Y-axis. The initial Y-axis phase, This represents the number of spatial physical poses. For L2 norm squaring operations; The Z-axis objective function is obtained by calculating the actual observed Z-axis coordinates of the fruit, the regression coefficient of the static equilibrium position of the Z-axis, the initial Z-axis excitation amplitude, the Z-axis air damping coefficient, the natural principal angular frequency of the Z-axis, and the initial Z-axis phase using the sixth equation. The sixth equation is as follows: ， in, The objective function is the Z-axis. For the first The actual Z-axis coordinate of the fruit at that moment. The regression coefficients for the static equilibrium position along the Z-axis. The initial Z-axis excitation amplitude, The Z-axis air damping coefficient is... The natural principal angular frequency of the Z-axis. The initial Z-axis phase, This represents the number of spatial physical poses. For L2 norm squaring operations; The X-axis time series parameter fitting model is updated according to the X-axis objective function to obtain the updated X-axis time series parameter fitting model. The Y-axis time series parameter fitting model is updated according to the Y-axis objective function to obtain the updated Y-axis time series parameter fitting model. The parameters of the Z-axis time series parameter fitting model are updated according to the Z-axis objective function to obtain the updated Z-axis time series parameter fitting model. The updated X-axis temporal parameter fitting model, the updated Y-axis temporal parameter fitting model, and the updated Z-axis temporal parameter fitting model are solved respectively to obtain the X-axis spatial pose vector, the Y-axis spatial pose vector, and the Z-axis spatial pose vector. A spatial pose prediction vector function is constructed using the X-axis spatial pose vector, the Y-axis spatial pose vector, and the Z-axis spatial pose vector.

4. The robotic arm control method for fruit grasping according to claim 1, characterized in that, The process of performing a screening analysis on the spatial pose prediction vector function and using the screening analysis result as the endpoint target boundary includes: Taking the first derivative of the spatial pose prediction vector function yields the swaying velocity vector function; The minimum point equation is constructed using the swaying velocity vector function. The minimum point equation is as follows: ， in, For the first The sway velocity vector function at time t, For L2 norm squaring operations; The minimum point equation is solved according to the preset screening conditions to obtain the golden capture window time; Based on the golden grasping window time, the spatial predicted three-dimensional coordinates, spatial predicted velocity vector, and spatial predicted acceleration vector are extracted from the shaking velocity vector function; The endpoint target boundary is constructed using the golden capture window time, the spatially predicted three-dimensional coordinates, the spatially predicted velocity vector, and the spatially predicted acceleration vector.

5. The robotic arm control method for fruit grasping according to claim 4, characterized in that, The current real-time position group of the robotic arm end effector includes the current real-time position of the robotic arm end effector along the X-axis, the current real-time position of the robotic arm end effector along the Y-axis, and the current real-time position of the robotic arm end effector along the Z-axis. The current real-time velocity group of the robotic arm end effector includes the current real-time velocity of the robotic arm end effector along the X-axis, the current real-time velocity of the robotic arm end effector along the Y-axis, and the current real-time velocity of the robotic arm end effector along the Z-axis. The current real-time acceleration group includes the current real-time acceleration of the X-axis, the current real-time acceleration of the Y-axis, and the current real-time acceleration of the Z-axis. The endpoint target boundary includes the predicted three-dimensional coordinates of the X-axis, the predicted three-dimensional coordinates of the Y-axis, the predicted three-dimensional coordinates of the Z-axis, the predicted velocity vector of the X-axis, the predicted velocity vector of the Y-axis, the predicted velocity vector of the Z-axis, the predicted acceleration vector of the X-axis, the predicted acceleration vector of the Y-axis, and the predicted acceleration vector of the Z-axis. The process of reconstructing and analyzing the real-time position set, real-time velocity set, and real-time acceleration set of the robotic arm end effector at the current moment, as well as the boundary of the target endpoint, to obtain the spatial pose reconstruction vector function includes: An X-axis boundary condition matrix is ​​constructed using the current real-time position of the X-axis robotic arm end effector, the current real-time velocity of the X-axis robotic arm end effector, the current real-time X-axis acceleration, the predicted 3D coordinates in X-axis space, the predicted velocity vector in X-axis space, and the predicted acceleration vector in X-axis space. The X-axis boundary condition matrix is ​​as follows: ， The Y-axis boundary condition matrix is ​​constructed using the current real-time position of the Y-axis robotic arm end effector, the current real-time velocity of the Y-axis robotic arm end effector, the current real-time acceleration of the Y-axis, the predicted 3D coordinates of the Y-axis space, the predicted velocity vector of the Y-axis space, and the predicted acceleration vector of the Y-axis space. The Y-axis boundary condition matrix is ​​as follows: ， The Z-axis boundary condition matrix is ​​constructed using the current real-time position of the Z-axis robotic arm end effector, the current real-time velocity of the Z-axis robotic arm end effector, the current real-time acceleration of the Z-axis, the predicted three-dimensional coordinates of the Z-axis, the predicted velocity vector of the Z-axis, and the predicted acceleration vector of the Z-axis. The Z-axis boundary condition matrix is ​​as follows: ， in, , in, For the first The real-time position of the X-axis robotic arm end effector at any given moment. For the first Real-time speed of the X-axis robotic arm end effector at any given moment. For the first Real-time X-axis acceleration at time t. For the first Predict the three-dimensional coordinates of the X-axis at time [time]. For the first The predicted velocity vector along the X-axis at time t. For the first The predicted spatial acceleration vector along the X-axis at time t. , , , , as well as All are X-axis control coefficient vectors. For the first The real-time position of the Y-axis robotic arm end effector at any given moment. For the first Real-time speed of the Y-axis robotic arm end effector at any given moment. For the first Real-time Y-axis acceleration at time t. For the first Predict the three-dimensional coordinates of the Y-axis at time t. For the first The predicted spatial velocity vector along the Y-axis at time t. For the first The spatially predicted acceleration vector along the Y-axis at time t. , , , , as well as All are Y-axis control coefficient vectors. For the first The real-time position of the Z-axis robotic arm end effector at any given moment. For the first Real-time speed of the Z-axis robotic arm end effector at any given moment. For the first Real-time Z-axis acceleration at time t. For the first Predict the three-dimensional coordinates of the Z-axis at time [time]. For the first The spatially predicted velocity vector along the Z-axis at time t. For the first The spatially predicted acceleration vector along the Z-axis at time t. , , , , as well as All are Z-axis control coefficient vectors. To capture the golden window of opportunity, For the current moment, This is the net time-domain window length; Solve the X-axis boundary condition matrix, the Y-axis boundary condition matrix, and the Z-axis boundary condition matrix respectively to obtain multiple X-axis control coefficient vectors, multiple Y-axis control coefficient vectors, and multiple Z-axis control coefficient vectors; The seventh equation is used to calculate the multiple X-axis control coefficient vectors to obtain the... The X-axis spatial pose reconstruction vector at time t, the seventh equation is: ， in, For the first The X-axis spatial pose reconstruction vector at time t. , , , , as well as All are X-axis control coefficient vectors. For the first time, For the first time; The eighth equation is used to calculate the multiple Y-axis control coefficient vectors to obtain the... The Y-axis spatial pose reconstruction vector at time t is given by the eighth equation: ， in, For the first The Y-axis spatial pose reconstruction vector at time t. , , , , as well as All are Y-axis control coefficient vectors. For the first time, For the first time; The ninth equation is used to calculate the multiple Z-axis control coefficient vectors, resulting in the... The Z-axis spatial pose reconstruction vector at time t is given by the ninth equation: ， in, For the first The Z-axis spatial pose reconstruction vector at time t. , , , , as well as All are Z-axis control coefficient vectors. For the first time, For the first time; Through the first The X-axis spatial pose reconstruction vector at time t, the first The Y-axis spatial pose reconstruction vector at time t and the first The spatial pose reconstruction vector function is constructed by reconstructing the Z-axis spatial pose reconstruction vector at time t.

6. The robotic arm control method for fruit grasping according to claim 1, characterized in that, The process of solving and analyzing the spatial pose reconstruction vector function, and controlling the robotic arm based on the solution and analysis results, includes: The spatial pose reconstruction vector function is solved to obtain multiple joint motor feedforward commands, and the robotic arm is controlled according to all the joint motor feedforward commands.

7. A robotic arm control device for fruit grasping, characterized in that, include: The image acquisition module is used to acquire images of the area to be grasped by using an RGB-D depth camera set on the end eye of the robotic arm, and to obtain multiple RGB images of the fruit and a fruit depth map corresponding to each RGB image of the fruit. The pose analysis module is used to perform pose analysis on each of the fruit RGB images and the fruit depth map corresponding to each of the fruit RGB images to obtain a set of spatial physical poses. The predictive analysis module is used to perform predictive analysis on the set of spatial physical poses to obtain a spatial pose prediction vector function. The filtering and analysis module is used to filter and analyze the spatial pose prediction vector function and use the filtering and analysis results as the endpoint target boundary. The reconstruction analysis module is used to import the current real-time position group, the current real-time velocity group, and the current real-time acceleration group of the robotic arm end effector at the current moment, and to perform reconstruction analysis on the current real-time position group, the current real-time velocity group, the current real-time acceleration group, and the endpoint target boundary to obtain the spatial pose reconstruction vector function. The robotic arm control module is used to solve and analyze the spatial pose reconstruction vector function, and control the robotic arm based on the solution and analysis results.

8. The robotic arm control device for fruit grasping according to claim 7, characterized in that, The pose analysis module is specifically used for: The target detection model is used to solve the RGB images of each fruit to obtain the two-dimensional pixel center coordinates corresponding to each RGB image of the fruit. The two-dimensional pixel center coordinates include the X-axis two-dimensional pixel center coordinates and the Y-axis two-dimensional pixel center coordinates. Extract the depth values ​​corresponding to the RGB images of each fruit from the depth maps of each fruit respectively; The X-axis image coordinate system origin offset, Y-axis image coordinate system origin offset, X-axis camera imaging scaling ratio, and Y-axis camera imaging scaling ratio are extracted from the RGB-D depth camera. The first formula is used to calculate the X-axis three-dimensional spatial coordinates corresponding to each fruit RGB image by calculating the origin offset of the X-axis image coordinate system, the X-axis camera imaging scaling ratio, the center coordinates of each X-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image. ， in, For the first Time of the first The X-axis three-dimensional spatial coordinates corresponding to the RGB image of each fruit For the first The X-axis two-dimensional pixel center coordinates corresponding to the RGB image of each fruit This is the offset of the origin of the X-axis image coordinate system. This refers to the X-axis camera imaging scaling ratio. For the first Time of the first The depth value corresponding to the RGB image of each fruit; The second equation is used to calculate the offset of the origin of the Y-axis image coordinate system, the Y-axis camera imaging scaling ratio, the center coordinates of each Y-axis two-dimensional pixel, and the depth value corresponding to each fruit RGB image, to obtain the Y-axis three-dimensional spatial coordinates corresponding to each fruit RGB image. A three-dimensional absolute spatial pose set is constructed using all the X-axis three-dimensional spatial coordinates, all the Y-axis three-dimensional spatial coordinates, and all the depth values. The second equation is: ， in, For the first Time of the first The Y-axis three-dimensional spatial coordinates corresponding to the RGB image of the fruit For the first The Y-axis two-dimensional pixel center coordinates corresponding to the RGB image of the fruit This is the offset of the origin of the Y-axis image coordinate system. The scaling factor for the camera image on the Y-axis. For the first Time of the first The depth value corresponding to the RGB image of each fruit; Import the homogeneous rotation and translation matrix and multiple joint absolute rotation angles, solve for all the joint absolute rotation angles, and obtain the homogeneous transformation matrix; The spatial physical pose set is obtained by calculating the homogeneous rotation and translation matrix, the homogeneous transformation matrix, and the three-dimensional absolute spatial pose set using the third equation. The third equation is: ， in, For the first The set of spatial physical poses at any given time. For the first The homogeneous transformation matrix at time step, It is a homogeneous rotation and translation matrix. For the first The set of three-dimensional absolute spatial poses at any given time.

9. A robotic arm control device for fruit grasping, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the robotic arm control method for fruit grasping as described in any one of claims 1 to 6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the robotic arm control method for fruit grasping as described in any one of claims 1 to 6.

Images