Multi-stage laser picking method and system based on industrial vision intelligence

Abstract

The application relates to the technical field of agricultural intelligent equipment, and discloses a robot multi-stage laser picking method and system based on industrial visual intelligence, wherein the method comprises the following steps: solving a noise-resistant three-dimensional coordinate of a petiole attachment point through a deep neighborhood sampling window in cooperation with median filtering and performing time-domain anti-shake correlation processing; driving a mechanical arm to complete open-loop coarse positioning; performing image visual servo closed-loop alignment based on a damping least square at a rate not lower than a predetermined frequency; loading laser physical optical axis model parameters fitted through singular value decomposition and performing Cartesian space translation compensation fine adjustment; and actuating a centrifugal air extractor and a fiber laser in sequence according to asymmetric time delay characteristics to complete negative pressure cooperative harvesting. The application solves the technical problem that three-dimensional positioning of a petiole attachment point is unstable due to edge noise of a depth image and high-frequency coordinate shaking of adjacent frames by adopting a deep neighborhood sampling window in cooperation with median filtering and combining time-domain anti-shake correlation processing, and the stability of three-dimensional coordinate solving of the petiole attachment point is improved.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of agricultural intelligent equipment technology, and in particular to a multi-stage laser harvesting method and system for robots based on industrial vision intelligence, electronic devices, and storage media. Background Technology

[0002] With the continuous integration of precision agriculture and intelligent manufacturing technologies, automated selective harvesting of premium tea leaves has become an important research direction in the field of agricultural robots. The grade requirements for premium tea leaves impose extremely stringent requirements on the harvesting precision of tender buds (one bud and one leaf). The diameter of the petiole of tea buds is usually only 1 to 3 millimeters, and the real tea garden environment has interference factors such as variable light, shading by branches and leaves, and cluttered backgrounds. This places extremely high demands on the perception accuracy and motion control capabilities of the harvesting equipment.

[0003] Current agricultural harvesting equipment generally employs non-selective mechanical shearing methods, which cannot achieve precise selective harvesting of individual tender buds. By introducing laser cutting technology into tea harvesting, researchers have attempted to combine it with visual sensors to locate the target and guide the laser to perform the cut, forming the basic technical route for vision-guided laser harvesting. In this route, a depth camera is used to acquire the target's three-dimensional coordinates, a robotic arm performs positioning actions based on these coordinates, the laser then performs the cut, and a pneumatic device is used to retrieve the harvested material.

[0004] Current vision-guided laser harvesting solutions fail to consider key factors such as scattering noise and zero-value voids generated by depth sensors at the delicate edges of plants, continuous slight target offsets caused by natural wind disturbances and residual oscillations of robotic arms, nonlinear projection deviations caused by the inherent installation bias of the depth camera's optical axis and the laser's optical axis at different working distances, and the lack of precise coordination between the establishment and shutdown timing of negative pressure airflow and the laser cutting action. These issues result in insufficient accuracy in calculating the three-dimensional coordinates of the petiole, the inability to maintain sub-millimeter level continuous alignment during single-loop positioning, the inability of the laser focus to accurately intersect the delicate petiole in three-dimensional space, and the scattering of harvested material due to the lack of a synchronous physical collection force field. Consequently, this technical approach is difficult to meet the accuracy and integrity requirements for grade harvesting in actual high-quality tea harvesting scenarios. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-stage laser harvesting method and system for robots based on industrial vision intelligence, which solves the technical problems existing in related technologies such as insufficient target positioning accuracy, poor stability of visual servo closed-loop control, difficulty in timing coordination of laser cutting and negative pressure recovery, and lack of fault tolerance in multi-target task scheduling.

[0006] This invention discloses a vision-guided, multi-stage closed-loop laser harvesting method for robots, comprising the following steps: acquiring registered color and depth images; extracting the target bud mask and locating the pixel coordinates of the petiole attachment point using an instance segmentation network; establishing a sampling window in the depth image centered on the pixel coordinates; removing abnormal depth values ​​and performing median filtering on the remaining valid depth values ​​to obtain a noise-resistant reference depth; back-projecting the pixel coordinates and the noise-resistant reference depth into three-dimensional physical coordinates in the camera coordinate system based on camera intrinsic parameters; mapping the three-dimensional physical coordinates to the target position in the base coordinate system using hand-eye calibration parameters; driving the robotic arm to move to the observation range; activating the visual servo closed-loop loop; and extracting pixel residuals and depth residuals in each control cycle to form a... The characteristic error vector is used to solve the six-dimensional generalized control velocity through the image interaction Jacobian matrix using the damped least squares method and then sent to the robotic arm until the characteristic error vector continuously satisfies the locking condition. The parameters of the offline calibrated laser physical optical axis model are loaded, and the Cartesian space translation compensation is solved based on the spatial intersection geometric constraints. The robotic arm is controlled to perform translation fine-tuning to align the laser focus with the target blade position. A hardware actuation command chain with asymmetric time delay characteristics is sent. First, the negative pressure airflow device is started and the flow field preloading time is delayed. Then, the laser is turned on to perform cutting. After the cutting is completed, the laser is turned off first and the pipeline is emptied for a delay before the negative pressure airflow device is turned off. The enabling time window of the negative pressure airflow device wraps around the enabling time window of the laser on the time axis.

[0007] Further, a sampling window is established in the depth image centered on the pixel coordinates. After removing abnormal depth values, median filtering is performed on the remaining valid depth values ​​to obtain the noise-resistant reference depth. This includes: establishing a pixel grid with a side length not less than a predetermined minimum size as the center of the pixel coordinates of the petiole attachment point as the depth neighborhood sampling window; removing depth readings with values ​​of zero, non-numerical values, and values ​​exceeding a preset working distance range within the depth neighborhood sampling window, wherein the lower and upper limits of the preset working distance range correspond to the lower and upper limits of the effective ranging range of the depth camera, respectively; and taking the median value after arranging the remaining valid depth values ​​numerically as the noise-resistant reference depth.

[0008] Furthermore, after obtaining the three-dimensional physical coordinates, the process also includes target temporal stabilization association processing: calculating the three-dimensional Euclidean distance between the three-dimensional physical coordinates extracted in the current frame and the existing target coordinates in the previous frame; when the three-dimensional Euclidean distance is less than a set distance threshold, it is determined to be a continuous dynamic displacement of the same target, the lifecycle timestamp of the target is reset and its tracking identifier is maintained; when the three-dimensional Euclidean distance exceeds the distance threshold, the original tracking identifier is unbound and a new tracking identifier is assigned; after the target temporal stabilization association processing, a stable target list with tracking identifiers is generated.

[0009] Furthermore, the step of solving the six-dimensional generalized control velocity using the damped least squares method via the image-interactive Jacobian matrix includes: deriving the image-interactive Jacobian matrix based on the perspective projection model. The Each row corresponds to the first-order partial derivative of a visual feature component with respect to the six-dimensional generalized velocity of the camera coordinate system. Each partial derivative term is analytically calculated based on perspective projection geometry and the pixel coordinates and depth values ​​of the current frame. The pixel residual components and depth residual components in the feature error vector undergo mean normalization based on their range, using the maximum permissible residual value of each component within the actual working range as the normalization reference range. According to the formula... Solving for the six-dimensional generalized control velocity ,in For velocity gain coefficient, The damping factor, It is the identity matrix. for The transpose of the matrix, This is the normalized feature error vector.

[0010] Furthermore, before sending the six-dimensional generalized control speed to the robotic arm, the six-dimensional generalized control speed is subjected to amplitude limiting and dead-zone filtering: instructions whose absolute value in the linear velocity component exceeds the preset maximum linear velocity are truncated to the preset maximum linear velocity; instructions whose absolute value in the angular velocity component exceeds the preset maximum angular velocity are truncated to the preset maximum angular velocity; signals whose absolute value in the linear velocity component is lower than the preset lower limit of the linear velocity dead zone are forcibly cleared to zero; signals whose absolute value in the angular velocity component is lower than the preset lower limit of the angular velocity dead zone are forcibly cleared to zero.

[0011] Furthermore, the offline calibrated laser physical optical axis model parameters are obtained as follows: A laser beam is controlled to emit ablation markers at multiple known spatial locations; the three-dimensional coordinates of these ablation markers are acquired using a depth camera to form a three-dimensional ablation point cloud dataset; singular value decomposition is performed on the three-dimensional ablation point cloud dataset, and the right singular vector corresponding to the largest singular value is extracted as the unit vector for the optical axis direction. The translation vector of the laser emission origin relative to the tool coordinate system is obtained by fitting the optimal straight line through the scorched mark point. The parameterized representation of the optimal straight line is obtained by... And the direction is A straight line in space.

[0012] Furthermore, the method of solving the Cartesian space translation compensation based on spatial intersection geometric constraints includes: according to the parameterized ray equation. Solve for the ray parameters by setting its vertical component equal to the vertical coordinate of the target point. The coordinates of the intersection point between the laser physical ray and the horizontal plane containing the target point are obtained, where The distance from the origin of the ray is... The spatial coordinates of the intersection point and the target point are calculated; the coordinate difference between the intersection point and the target point in the horizontal plane is used as the lateral translation compensation of the robotic arm end in Cartesian space.

[0013] Furthermore, it also includes asynchronous task scheduling based on a state machine: the task management module receives the stable harvesting target list, converts each target into a harvesting task, and maintains a task queue arranged by priority; the task management module runs a finite state machine containing four states: pending, running, successful, and failed; when the robotic arm is in an idle state, the state of the task at the head of the queue transitions from pending to running, and is asynchronously sent to the execution process; when a single harvesting process is successfully completed, the task state is updated to successful and archived, and the target positioning and the update of the stable harvesting target list are re-executed to enter the next harvesting cycle.

[0014] Furthermore, it also includes anomaly detection and safety interruption: timeout monitoring and status word daemon processes run synchronously in each sub-stage. If at least one of the following abnormal features is detected: path planner execution timeout, visual servo closed-loop residual not converging for multiple consecutive control cycles and showing a diverging trend, image interaction Jacobian matrix continuously in ill-conditioned condition number, or serial communication disconnection, then the hardware safety interruption signal is forcibly pulled low to cut off the power enable of the laser and negative pressure airflow device, and the current task is marked as failed. The task management module performs fault-tolerant scheduling of failed tasks according to the preset retry strategy.

[0015] This invention also provides a vision-guided multi-stage closed-loop laser harvesting system for robots. The system includes: a depth vision perception and 3D spatial positioning module, a multi-stage motion control and servo solution module, and an electromechanical timing control and negative pressure pneumatic actuation module. The depth vision perception and 3D spatial positioning module is configured to acquire registered color images and depth images, extract target bud masks and locate the pixel coordinates of petiole attachment points using an instance segmentation network, establish a sampling window in the depth image and perform median filtering to obtain a noise-resistant reference depth, and back-project the image to obtain 3D physical coordinates in the camera coordinate system based on camera intrinsic parameters. The multi-stage motion control and servo solution module... The calculation module is configured to map the three-dimensional physical coordinates to the base coordinate system through hand-eye calibration parameters and drive the robotic arm to move to the observation range. It starts the visual servo closed loop and uses the damped least squares method to solve the control speed to achieve closed-loop alignment. It loads the laser physical optical axis model parameters and solves the translation compensation amount based on spatial intersection geometric constraints to perform fine-tuning. The electromechanical timing control and negative pressure pneumatic actuation module is configured to send a hardware actuation command chain with asymmetric time delay characteristics. According to the timing constraint that the enable time window of the negative pressure airflow device wraps the enable time window of the laser on the time axis, it coordinates the control of the negative pressure airflow device and the laser to complete the cutting and material recovery.

[0016] This invention solves the technical problem of unstable 3D positioning of petiole attachment points caused by edge noise in depth images and high-frequency coordinate jitter in adjacent frames by using a depth neighborhood sampling window in conjunction with median filtering and temporal anti-shake correlation processing during the 3D coordinate calculation stage, and achieves the technical effect of improving the stability of 3D coordinate calculation of petiole attachment points.

[0017] By continuously executing image vision servo closed-loop control based on damped least squares at a rate not lower than a predetermined frequency, the technical problem that natural wind disturbance and residual oscillation of the robotic arm cause the single open-loop positioning to be unable to maintain continuous alignment accuracy is solved. The technical effect of dynamically correcting the end pose in each control cycle and keeping the control speed bounded and stable when the Jacobian matrix is ​​close to singular is achieved.

[0018] By loading the laser physical optical axis model parameters fitted from the 3D scorched point cloud dataset using singular value decomposition and solving the Cartesian space translation compensation based on spatial triangle intersection geometric constraints, the technical problem of nonlinear projection deviation caused by the inherent installation offset of the depth camera optical axis and the laser optical axis at different working distances, which prevents fixed 2D offset compensation from achieving accurate 3D intersection, was solved. This achieved the technical effect of accurately aligning the laser focus with the blade stalk position under different working depth conditions. By employing a hardware actuation command chain with asymmetric time delay characteristics, the enable time window of the centrifugal fan is made to strictly wrap around the enable time window of the fiber laser on the time axis. This solves the technical problem of the harvested material scattering due to the mismatch between the establishment and shutdown timing of the negative pressure airflow and the laser cutting action. It achieves the technical effect of maintaining an effective negative pressure suction field at the moment of cutting and after cutting to guide the harvested material into the material collection tank. Attached Figure Description

[0019] Figure 1 This is a flowchart of a vision-guided robot multi-stage closed-loop laser harvesting method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall flow chart of the multi-stage closed-loop laser harvesting system provided in the embodiments of the present invention; Figure 3 This is a schematic diagram of the data flow diagram of the depth vision perception and three-dimensional localization module provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the visual servo closed-loop residual convergence process provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the distribution of the validity of depth sampling window readings provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the timing of the asymmetric delay hardware actuation instruction chain provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of the task state machine state transition diagram provided in an embodiment of the present invention; Figure 8 This is a schematic diagram showing the comparison of the status of the multi-target harvesting task queue provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the numerical comparison of the three-dimensional coordinate system transformation link provided in an embodiment of the present invention; Figure 10 This is a three-dimensional view of a vision-guided robot provided in an embodiment of the present invention. Detailed Implementation

[0020] Reference Figures 1-10 This invention discloses a vision-guided, multi-stage closed-loop laser harvesting method for robots, comprising the following steps: Step S100: Noise-resistant depth sampling and target 3D localization.

[0021] Reference Figure 3 , 9 In step S100, the RGB-D depth camera 004 installed at the end of the robotic arm 006 is used to acquire the registered color image and depth image. The mask region of the target bud is extracted from the color image by the instance segmentation network, and the two-dimensional pixel coordinates of the petiole attachment point are located at the base of the mask. A depth neighborhood sampling window of a predetermined size is established in the depth image centered on the pixel coordinates. After removing abnormal depths that are not within the finite value range or exceed the working range, median filtering is performed on the remaining valid depth values ​​within the depth neighborhood sampling window to obtain the noise-resistant reference depth. Based on the intrinsic parameters of the camera calibration, the pixel coordinates are back-projected and transformed into three-dimensional physical coordinates in the camera coordinate system through a perspective projection geometry model. ,in To indicate the transpose, the coordinate components are determined by the following formula: , ,

[0022] in, and The focal length of the camera along the horizontal and vertical directions (in pixels). and The pixel coordinate components (unit: pixels) of the camera principal point on the image plane. Noise level reference depth (unit: meters). and These are the pixel coordinates (in pixels) of the petiole attachment point. Because... , , , , , All are in pixels. and The dimensionless pixel difference is divided by the focal length (also in pixels) to obtain a dimensionless angle ratio, which is then multiplied by the noise reduction reference depth. (Unit: meters), the result , , The units of measurement are all meters, consistent with physical three-dimensional coordinates.

[0023] Furthermore, the instance segmentation network described above uses color image frames as input layer data, and the output layer generates pixel-level mask regions of the target buds and their category labels; the perception program searches for connected components that match the characteristics of tea stems in the bottom row region of the mask, calculates and extracts the centroid of the connected component as the two-dimensional pixel coordinates of the petiole attachment point. The instance segmentation network employs supervised training, using manually labeled bud pixel-level masks as training labels. The loss function is a weighted sum of binary cross-entropy loss and Dice loss between mask predictions and ground truth labels, and the network parameters are updated using the Adam optimization algorithm.

[0024] It should be noted that the above-mentioned deep neighborhood sampling window is based on the leaf stalk pixel coordinates. Centered Pixel grid, in which The side length of the sampling window (in pixels). Not smaller than the predetermined minimum size, for example Within the deep neighborhood sampling window, the program automatically removes values ​​of zero, non-numeric (NaN) values, and values ​​exceeding the preset working distance range. The depth readings, of which and These represent the lower and upper limits (in meters) of the effective ranging range of the RGB-D depth camera 004. The median value of the remaining effective depth values, arranged numerically, is used as the noise-reduction reference depth. The median is used instead of the mean because the median is more robust to outlier scattering noise generated by the depth sensor at the thin edges of plants, and a single or a few extreme outliers will not deviate the final noise-resistant baseline depth calculation result.

[0025] In this embodiment, to suppress high-frequency jitter of target coordinates between adjacent frames, a target temporal anti-shake association process is further included in step S100: The three-dimensional Euclidean distance between the three-dimensional physical coordinates extracted in the current frame and the existing target coordinates in the previous frame is calculated. When this distance is less than a set distance threshold, it is determined to be a continuous dynamic displacement of the same target, the target's lifetime timestamp is reset, and its unified tracking ID is maintained. When the distance exceeds the aforementioned distance threshold, the original tracking ID is unbound and the target is treated as a new entity and assigned a new tracking ID. After the above temporal association process, a stable target list with tracking IDs is generated.

[0026] In at least one example of the embodiments of this disclosure, the distance threshold is configured to be 0.03 meters.

[0027] A tea garden (hereinafter referred to as Tea Garden A) in a famous green tea producing area entered the concentrated budding period of spring tea in mid-April 20XX. The canopy height of the tea trees in the garden was approximately 0.85 to 1.10 meters. The target for picking was a standard tender shoot with one bud and one leaf, and the petiole diameter was approximately 1.5 to 2.2 mm. A mobile chassis 001 slowly advanced along a path with a row spacing of approximately 1.2 meters. A six-DOF collaborative robotic arm 006 adjusted its end-effector orientation in real time according to the chassis displacement. An RGB-D depth camera 004 acquired a color image with a resolution of 640×480 pixels and a corresponding depth map in the current frame. Three candidate tender bud targets were visible in the image. The instance segmentation network performed pixel-level mask extraction on the target with the highest confidence (hereinafter referred to as target bud 001) and located the petiole attachment point at the centroid of the base connected region of the mask. , .

[0028] by Establish for the center The pixel depth sampling window collected a total of 25 depth readings. Three readings were zero (due to edge occlusion), one was NaN (scattering noise), and one was 0.682 meters (exceeding the specified depth). The upper limit of the working range (in meters) was eliminated, and the median of the remaining 20 valid readings was taken after sorting to obtain the noise-resistant reference depth. Meters. Camera internal parameters are: Pixels Pixels Pixels Pixels, substituting into the perspective projection formula: rice rice rice The three-dimensional physical coordinates of target bud 001 in the camera coordinate system Meters. Temporal image stabilization correlation processing calculates the coordinates of this coordinate with the coordinates recorded in the previous frame. The three-dimensional Euclidean distance between them is approximately 0.00024 meters, which is much smaller than the 0.03-meter threshold. It is determined to be a continuous displacement of the same target. The tracking ID is maintained as TRK-017, the life cycle timestamp is reset, and target bud 001 is added to the list of stable harvesting targets.

[0029] Table 1 Effective readings of depth sampling window and noise-resistant reference depth calculation

[0030] After the target bud 001 was located in three dimensions, the list of stable harvested targets is recorded as follows.

[0031] Table 2. List of Stable Harvesting Targets (Current Frame)

[0032] Step S200: Large-range open-loop coarse positioning.

[0033] In step S200, the three-dimensional physical coordinates of the targets in the stable harvesting target list are read. Based on the fixed static hand-eye calibration parameters from the camera to the robotic arm's end effector, the coordinates are mapped to the target position in the robot's base coordinate system. :

[0034] in, The target's three-dimensional physical coordinates in the camera coordinate system obtained in step S100. Let be the rigid body transformation matrix from the camera coordinate system to the tool coordinate system. Let be the rigid body transformation matrix from the tool coordinate system to the base coordinate system. The target position is defined in the robot's base coordinate system. Based on the set safe observation distance, the desired open-loop pose of the end effector of the robotic arm 006 is calculated so that the optical axis of the RGB-D depth camera 004 points to the target and moves back the safe observation distance along the optical axis. The path planner is then called to generate a smooth joint space obstacle avoidance trajectory, driving the robotic arm 006 to move rapidly to the observation range in a joint space motion mode.

[0035] It should be noted that the aforementioned safe observation distance is the reserved distance between the camera's imaging plane and the target along the camera's optical axis. Its value should meet the minimum effective ranging requirements of the RGB-D depth camera 004, while ensuring that the robotic arm's end effector does not collide with the tea tree canopy during movement. For example, in one example, the safe observation distance is configured as 0.30 meters.

[0036] It should be noted that the above path planner receives the initial joint configuration and the target end pose as input, and outputs the joint space motion trajectory that satisfies the joint limit constraints and collision avoidance constraints, for example, using the RRT-Connect algorithm or the TrajOpt algorithm.

[0037] The task scheduling module selects the highest priority target bud 001 (tracking ID: TRK-017) from the stable picking target list as the current picking target, and reads its three-dimensional physical coordinates in the camera coordinate system. Meters. Based on offline hand-eye calibration results, Corresponding to the current robotic arm joint configuration Combined effect, Mapping to the base coordinate system, we obtain the position of the target bud 001 in the base coordinate system. Meters. The robot arm retreats along the camera's optical axis at a safe observation distance of 0.30 meters, calculating the desired open-loop pose of its end effector. The path planner uses the RRT-Connect algorithm, starting with the current six-joint configuration and ending with the desired pose, to plan a joint space trajectory that satisfies joint limits and avoids collisions with the tea tree canopy within approximately 0.31 seconds. This drives the robot arm to move rapidly to the observation area. The actual depth of the end effector (distance from the camera to the target) is approximately 0.308 meters, deviating from the expected value by about 8 millimeters, which falls within the coarse positioning accuracy range. This residual deviation will be eliminated by subsequent visual servo closed-loop steps.

[0038] Table 3. Coordinate system transformation and open-loop coarse positioning results

[0039] Step S300: Image visual servo closed-loop alignment based on damped least squares.

[0040] In step S300, after the robotic arm 006 reaches the coarse positioning range, it initiates a speed control closed-loop loop at a frequency not lower than a predetermined frequency. Within each control cycle, the pixel coordinates and depth values ​​of the leaf stalk in the current image frame are extracted in real time. Using the pixel coordinates of the camera's principal point and the set desired observation depth as the expected values, the feature error vector composed of the pixel residual and the depth residual is calculated. ,in Indicates transpose. Horizontal pixel residual (unit: pixels). Vertical pixel residual (unit: pixels). The depth residual is given in meters; the dimension is derived from the perspective projection model. Image interaction Jacobian matrix , Each row corresponds to the first-order partial derivative of a visual feature component with respect to the six-dimensional generalized velocity in the camera coordinate system. The first two rows consist of the partial derivatives of the pixel coordinates with respect to the camera's translation and rotation velocities, while the third row consists of the partial derivatives of the depth value with respect to the camera's translation velocity along the optical axis. Each partial derivative term is analytically calculated based on perspective projection geometry and the pixel coordinates and depth value of the current frame. The robust Jacobian pseudo-inverse, which includes the damping factor, is solved using the damped least squares method to output the six-dimensional generalized control velocity in the camera coordinate system. :

[0041] in, This is the velocity gain coefficient. The damping factor, for identity matrix For dimension The image interaction Jacobian matrix, for The transpose of the matrix, This is the feature error vector. It should be noted that the feature error vector... The pixel residual component (unit: pixels) and the depth residual component (unit: meters) have different dimensions. Before substituting them into the above formula, the feature error vector needs to be adjusted. Each component undergoes normalization to scale it to a uniform dimensionless numerical range, eliminating the influence of dimensional differences on the velocity calculation results. Specifically, mean normalization based on the range is used, with the maximum permissible residual value of each component within the actual working range serving as the normalization reference range. The aforementioned damping factor... The introduction of the image interaction Jacobian matrix makes the image interaction Jacobian matrix more convenient. When the matrix approaches a rank deficiency or when the robotic arm moves near a singular point of a cooperative joint, It remains positive definite and reversible, thus avoiding numerical divergence in the control velocity.

[0042] The six-dimensional generalized control velocity obtained by the solution After being rotated and mapped to the robot's Cartesian space, the high-frequency signals are continuously transmitted to the robot arm's underlying controller; if the feature error vector If the locking condition is not met, the image will be reacquired in the next control cycle, and the closed-loop calculation process will be iterated back; if the feature error vector If the locking condition is met for several consecutive control cycles, the speed control loop is exited, triggering the robotic arm to brake and come to a stop, thus completing the visual precision alignment.

[0043] For example, the predetermined frequency is 20Hz, which means that the control loop performs a complete iteration of error acquisition, speed calculation and command issuance at a frequency of not less than 20 times per second.

[0044] It should be noted that the above locking condition is the absolute value of the pixel residual. and All are below the preset pixel threshold and the absolute value of the depth residual If the error is below a preset depth threshold, the preset pixel threshold and the preset depth threshold are the convergence criterion thresholds for the corresponding residual components in their original dimensions before normalization. In one example, the pixel threshold is 3 pixels, the depth threshold is 0.01 meters, and this low error state must be maintained continuously in the control loop for no less than 10 control cycles.

[0045] In this embodiment of the application, in order to suppress the continuous high-frequency jitter generated by the robotic arm under minimal visual residuals and to avoid excessive speed commands causing overload impact on the joints of the robotic arm, in addition to step S300, the method further includes six-dimensional generalized control speed. Amplitude limiting and dead-zone filtering are performed: commands with absolute values ​​exceeding the preset maximum linear velocity are truncated to this upper limit, and commands with absolute values ​​exceeding the preset maximum angular velocity are also truncated to this upper limit. Simultaneously, minute signals with absolute values ​​below the preset lower limit of the linear velocity dead zone are forcibly cleared, and minute signals with absolute values ​​below the preset lower limit of the angular velocity dead zone are also forcibly cleared. The speed commands, after the above truncation and clearing processing, are then sent to the lower-level controller.

[0046] In at least one example of the embodiments of this disclosure, the preset maximum linear velocity is 0.05 m / s, the preset maximum angular velocity is 0.15 rad / s, and the preset lower limit of the linear velocity dead zone is 0.0005 m / s.

[0047] Reference Figure 4After robotic arm 006 reaches the coarse positioning range, it initiates a visual servo closed loop at a frequency of 20Hz. The first control cycle acquires the pixel coordinates of the petiole of target bud 001. The current frame depth is 0.308 meters; the expected value is the camera principal point. Given an expected observation depth of 0.300 meters, calculate the original dimensional residuals: Pixels Pixels Meters. The normalized reference range is set to a maximum permissible residual of 160 pixels in the pixel direction and a maximum permissible residual of 0.10 meters in the depth direction. After normalization, the components of the feature error vector are 0.0844, 0.0675, and 0.080, respectively. After unifying the dimensions, they are substituted into the damped least squares formula (…). , Solving the six-dimensional generalized control velocity After amplitude limiting (upper limit of linear velocity 0.05 m / s, upper limit of angular velocity 0.15 rad / s) and dead-zone filtering (dead zone of linear velocity 0.0005 m / s), the effective component of linear velocity is approximately 0.032 m / s, which is then sent to the lower-level controller. After approximately 26 control cycles (approximately 1.3 seconds) of closed-loop iteration, Converging to 2.0 pixels, Converging to 1.6 pixels, The convergence to 0.005 meters means that all three residuals meet the locking conditions, and this state is maintained for 12 consecutive control cycles, triggering the robotic arm to brake and stop, thus completing the visual precision alignment.

[0048] Table 4 Residual Convergence Process in the Critical Cycle of Visual Servo Closed-Loop

[0049] Step S400: Spatial off-axis compensation fine-tuning based on SVD line fitting.

[0050] In step S400, the laser physical optical axis model parameters, which were fitted and extracted from the 3D scorched point cloud dataset through singular value decomposition during the offline calibration phase, are loaded. These parameters include the translation vector of the laser emission origin relative to the tool coordinate system. and the unit vector of the optical axis direction Combined with the actual working height at the current depth after locking in step S300, the Cartesian spatial translation compensation amount required to make the laser physical ray accurately pass through the target's three-dimensional physical coordinates is solved using the spatial triangle intersection geometric constraints; the robotic arm 006 is controlled to perform millimeter-level translation fine-tuning corresponding to this translation compensation amount in a Cartesian linear motion mode, so that the physical focusing point of the fiber laser 005 accurately covers the spatial position of the target leaf stalk.

[0051] It should be noted that the above offline calibration process is as follows: At multiple known spatial locations, the fiber laser 005 emits a beam to generate ablation markers. The 3D coordinates of these ablation points are acquired using an RGB-D depth camera 004 to form a 3D ablation point cloud dataset. Singular value decomposition is performed on the 3D ablation point cloud dataset to fit the optimal straight line passing through these points. This straight line is the physical optical axis of the fiber laser 005 in the tool coordinate system, and its parameterized representation is that it passes through the origin. And the direction is The spatial straight line. The above singular value decomposition takes the three-dimensional coordinate matrix of the scorched point as input and outputs the singular vector representing the principal direction of the point cloud and the corresponding singular value, where the right singular vector corresponding to the largest singular value is extracted as the unit vector of the optical axis direction. .

[0052] It should be noted that the solution process for the above spatial triangle intersection geometric constraints is as follows: the position of the laser emission origin in the current robotic arm end-effector coordinate system is known. Optical axis direction Given the spatial coordinates of the target point at the current working depth, calculate the intersection of the ray extending along the laser axis and the horizontal plane where the target point is located, and then calculate the lateral translation of the robotic arm end in Cartesian space required to make the intersection point coincide with the target point.

[0053] Furthermore, in the calculation of the aforementioned lateral translation, the coordinates of the intersection point between the laser ray and the horizontal plane containing the target point are determined by the parameterized ray equation. Solve for the parameters by setting its vertical component equal to the vertical coordinate of the target point. Received, among which The distance from the origin of the ray is... Spatial coordinates of the location Let be the translation vector of the laser emission origin relative to the tool coordinate system. The unit vector along the optical axis. For the ray parameter, the difference in coordinates between the intersection point and the target point in the horizontal plane is the required Cartesian space lateral translation compensation.

[0054] In this embodiment, the introduction of the Cartesian space translation compensation means that even if there is a nonlinear projection deviation between the optical axis center of the RGB-D depth camera 004 and the physical focus of the fiber laser 005 at different working distances, the robotic arm 006 can still precisely align the laser focus with the stalk position through the Cartesian space fine adjustment, thereby achieving sub-millimeter-level focus intersection under different working depth conditions.

[0055] After visual servo closed-loop locking, the system loads the laser physical optical axis model parameters obtained during the offline calibration phase: the translation vector of the laser emission origin relative to the tool coordinate system. meter, unit vector along the optical axis (Already normalized). This parameter is obtained by fitting the 3D coordinate matrix of the scorch point collected at 7 different spatial locations during the offline calibration phase using singular value decomposition. The right singular vector corresponding to the maximum singular value is... .

[0056] At the current working depth, the vertical coordinate component of the petiole of target bud 001 in the tool coordinate system is 0.3012 meters. Let the vertical component of the parameterized ray equation equal to this value, and solve for the ray parameters. :

[0057] Substituting into the ray equation, the lateral coordinates of the intersection point of the laser ray and the target horizontal plane are: rice rice The horizontal coordinate of the petiole of target bud 001 in the tool coordinate system is: The difference between the lateral coordinates of the meter and the laser intersection point is the lateral translation compensation amount in Cartesian space. Meters (approximately $0.99 millimeters) meters (approximately 39.41 millimeters) in The significant directional deviation reflects the actual projection of the inherent lateral offset between the camera's optical axis and the laser's optical axis at the current working distance of 0.30 meters. The robotic arm 006 performs the corresponding translational fine-tuning in a Cartesian linear motion mode, precisely shifting the laser focus from the center of the camera's optical axis to the spatial position of the petiole of the target bud 001. After the translational fine-tuning is completed, the end point comes to rest.

[0058] Table 5 Laser optical axis calibration parameters and off-axis compensation calculation results

[0059] Step S500: Negative pressure coordinated harvesting under electromechanical interlock.

[0060] Reference Figure 6 In step S500, after the robotic arm 006 completes translational fine-tuning and comes to a rigid stop, it sequentially sends a hardware actuation instruction chain with asymmetric time delay characteristics to the lower-level hardware controller via the serial port to execute the following sub-steps: Step S501: Send an enable command to start the centrifugal exhaust fan 008. The airflow is conducted through the material collection tank 007 and the flexible corrugated hose 002 to the inside of the horn-shaped air collection hood 003 to form a negative pressure suction field, and the preset flow field preloading time is forcibly delayed. This is to wait for the cyclonic flow field inside the cavity to reach a stable state.

[0061] Step S502: Flow field preloading time After the timeout, an enable command is sent to activate the fiber laser 005, which emits a continuous infrared beam to perform timed high-temperature vaporization and ablation cutting on the petiole. At the moment of cutting, the tender leaves that have detached from the plant are immediately sucked into the material collection tank 007 through the flexible corrugated hose 002 under the negative pressure pre-formed in the trumpet-shaped gas collection hood 003. They are then intercepted by the stainless steel filter screen inside the tank and settle and accumulate.

[0062] Step S503: After the preset cutting duration is reached, first send a command to turn off the power to the fiber laser 005, and forcibly delay the preset pipeline emptying time. During this period, the centrifugal exhaust fan 008 maintains suction operation to ensure that any remaining tea material inside the flexible corrugated hose 002 is completely sucked into the bottom of the material collection tank 007; pipeline emptying time After the timeout, a command is sent to shut down the centrifugal exhaust fan 008, completing a single coordinated non-destructive recovery.

[0063] It should be noted that the aforementioned asymmetric time delay characteristic refers to the fact that the enabling time window of the centrifugal fan 008 strictly wraps around the enabling time window of the fiber laser 005 on the time axis. That is, the negative pressure flow field is established before the laser cutting action and shuts off after the laser cutting action. (Flow field preloading time) This is used to ensure that a stable high-speed air cyclone has formed inside the horn-shaped gas collector 003 before laser emission, and to allow for pipeline emptying time. This is used to guide all suspended material within the flexible corrugated hose 002 into the material collection tank 007 using residual suction airflow after the laser is turned off. In one example... The first predetermined duration (e.g., 0.2 seconds). The second predetermined duration (e.g., 0.1 seconds).

[0064] Furthermore, the aforementioned asymmetric time delay characteristic is specifically manifested on the time axis as follows: the enabling start time of the centrifugal fan 008 is earlier than the enabling start time of the fiber laser 005. The enable / deactivation time of centrifugal fan 008 is later than the enable / deactivation time of fiber laser 005. The enable duration of the fiber laser 005 is the preset cutting maintenance time. This time window is completely included within the enable time window of the centrifugal fan 008. The three time intervals are connected sequentially on the time axis without overlapping gaps, thereby ensuring that the negative pressure suction field remains effective throughout the entire laser cutting process and in the reserved time periods before and after it.

[0065] After robotic arm 006 completes the millimeter-level translational fine-tuning in step S400 and comes to a rigid stop, the system begins executing the hardware actuation command chain at 09:37:22 on April 14, 20XX. The set parameters are: The cutting duration is 0.8 seconds. The total enable time of centrifugal exhaust fan 008 is seconds. Second.

[0066] In step S501, at 09:37:22.000, a fan enable command is sent. The airflow is conducted through the material collection tank 007 and the flexible corrugated hose 002 to the horn-shaped gas collection hood 003. Within the 0.2-second preloading time of the flow field, the negative pressure cyclone in the cavity reaches a stable state. In step S502, at 09:37:22.200, a laser enable command is sent. The fiber laser 005 emits a continuous infrared beam to perform high-temperature vaporization ablation and cutting on the petiole of the target bud 001. At the moment of cutting (approximately 09:37:22.350), the tender leaf detaches from the plant. Under the action of the negative pressure suction field that has been stably established in the horn-shaped gas collection hood 003, it is immediately sucked into the material collection tank 007 through the flexible corrugated hose 002 and intercepted and settled by the stainless steel filter screen inside the tank. In step S503, the fiber laser 005 is turned off first at 09:37:23.000, and the exhaust fan continues to run for 0.1 seconds (until 09:37:23.100) to ensure that the residual material in the corrugated hose 002 is completely guided to the bottom of the collection tank. Then the centrifugal exhaust fan 008 is turned off, and the single-time collaborative non-destructive recovery of the target bud 001 is completed.

[0067] Table 6 Timing of Asymmetric Delay Hardware Actuation Instruction Chain

[0068] Step S600: Asynchronous task scheduling based on state machine.

[0069] Reference Figure 7 , 8In step S600, the task management module receives the stable harvesting target list generated in step S100, converts each target into a harvesting task, and maintains a task queue arranged by priority. The task management module runs a finite state machine containing four states: PENDING, RUNNING, SUCCEEDED, and FAILED. When the robotic arm 006 is idle, the scheduling module transitions the state of the task at the head of the queue from PENDING to RUNNING and asynchronously sends it to the execution flow. The execution sequence of steps S200 to S500 is bound to the task context of the RUNNING state. When step S500 is successfully completed, the task state is updated to SUCCEEDED and archived. The scheduling module then backtracks to step S100 to re-execute the target location and stable harvesting target list update, entering the next harvesting cycle.

[0070] After step S500 of target bud 001 (TRK-017) is successfully completed, the task management module updates its status from RUNNING to SUCCEEDED and archives it. It then backtracks to step S100 to reacquire images and update the stable harvesting target list. At this time, TRK-019 (target bud 002) and TRK-022 (target bud 003) in the original list are still in the PENDING state, and TRK-017 has been removed from the queue. The scheduling module detects that the robotic arm is idle, transitions the status of the head task TRK-019 to RUNNING, asynchronously issues the execution sequence of steps S200 to S500, and enters the harvesting cycle for target bud 002. A new bud target is also detected in the reacquired image from step S100, assigned a tracking ID of TRK-031, set its status to PENDING, and appended to the end of the queue. The current task queue status is shown below.

[0071] Table 7 Task State Machine Queue (After Target Bud 001 is Completed)

[0072] Step S700: Exception capture and safe interruption.

[0073] In step S700, independent timeout monitoring and status word daemons run synchronously in each sub-stage from steps S200 to S500. If at least one of the following abnormal characteristics is detected in any sub-stage: path planner execution timeout, visual servo closed-loop residual failing to converge for multiple consecutive control cycles and exhibiting a divergent trend, or image interaction Jacobian matrix... If the system remains in a pathological condition and serial communication is interrupted, the control logic jumps to the failure branch, forcibly pulls the hardware safety interrupt signal low to cut off the power enable of the fiber laser 005 and the centrifugal exhaust fan 008, and marks the current task as FAILED. The task management module performs fault-tolerant scheduling of failed tasks according to the preset retry policy.

[0074] During the visual servo closed-loop operation of step S300 on target bud 002 (TRK-019), a lateral wind disturbance caused significant shaking of the tea branch, resulting in the visual servo closed-loop residual failing to converge within 23 consecutive control cycles. The value showed a continuous increasing trend (from 5.2 pixels in the 10th cycle to 11.7 pixels in the 23rd cycle), triggering the divergence criterion. The timeout monitoring daemon captured this abnormal feature, the control logic jumped to the failure branch, forcibly pulled the hardware safety interrupt signal low, and the power enable of both the fiber laser 005 and the centrifugal exhaust fan 008 was cut off. The TRK-019 task status was marked as FAILED. The task management module reset TRK-019 to the PENDING state according to the preset retry strategy and kept it at the top of the queue. After the robotic arm returned to a safe pose, the scheduling module re-evaluated the harvestable status of the target bud 002 in the next round of image acquisition step S100. If the tea branch returned to stillness, it jumped to RUNNING again to execute the picking; otherwise, its priority was reduced and it skipped to TRK-022 for execution, ensuring the continuous and safe operation of the system in the unstructured tea garden environment.

[0075] The present disclosure provides at least one embodiment of a vision-guided multi-stage closed-loop laser harvesting system for robots, including a depth vision perception and three-dimensional spatial positioning module, a multi-stage motion control and servo calculation module, and an electromechanical timing control and negative pressure pneumatic actuation module.

[0076] For a detailed explanation of the depth vision perception and 3D spatial positioning module, please refer to the relevant description of step S100 in the above embodiment of the vision-guided robot multi-stage closed-loop laser picking method, which will not be repeated here.

[0077] For a detailed explanation of the multi-stage motion control and servo solution module, please refer to the relevant descriptions of steps S200 to S400 in the above embodiment of the vision-guided robot multi-stage closed-loop laser picking method, which will not be repeated here.

[0078] For a detailed explanation of the electromechanical timing control and negative pressure pneumatic actuation module, please refer to the relevant description of step S500 in the above embodiment of the vision-guided robot multi-stage closed-loop laser harvesting method, which will not be repeated here. The pneumatic negative pressure collection hardware includes a centrifugal fan 008 and a material collection tank 007 fixedly deployed on the mobile chassis 001. A flexible corrugated hose 002 extends outward from the air inlet of the material collection tank 007 and is routed along the outer side of the robotic arm 006. The end of the flexible corrugated hose 002 is physically sealed and connected to a horn-shaped gas collection hood 003 rigidly installed around the focusing lens output port of the fiber laser 005.

[0079] The technical effect of the vision-guided robot multi-stage closed-loop laser harvesting system is the same as that of the vision-guided robot multi-stage closed-loop laser harvesting method mentioned above, and will not be repeated here.

[0080] This embodiment uses a noise-resistant depth calculation method that combines a depth neighborhood sampling window with median filtering in step S100. The median operation is not sensitive to outlier scattering noise and zero-value holes generated by the depth sensor at the thin edges of plants. Single or a few extreme abnormal depth readings will not deviate from the final value of the noise-resistant reference depth. Therefore, it overcomes the factor of insufficient accuracy in the three-dimensional coordinate calculation of petiole attachment points caused by edge noise of the depth image. Combined with target temporal anti-shake correlation processing, it filters out high-frequency coordinate jitter between adjacent frames, thus solving the stability problem of bud recognition and petiole three-dimensional positioning.

[0081] This implementation employs image visual servo closed-loop control based on damped least squares in step S300. It continuously acquires the pixel and depth residuals of the current frame at a rate not lower than a predetermined frequency and iteratively solves for the control speed. This ensures that the robot arm's end-effector pose is dynamically corrected in the direction of eliminating residuals in each control cycle. Furthermore, the introduction of the damping factor guarantees the image interaction Jacobian matrix. When approaching the singularity, the control speed remains bounded and stable, thus overcoming the factors that cause continuous slight deviations in the target due to natural wind disturbances and residual oscillations of the robotic arm, which prevent single open-loop positioning from maintaining alignment accuracy. Therefore, the dynamic alignment drift problem is solved.

[0082] This implementation overcomes the problem of nonlinear projection deviation caused by the inherent installation offset of the RGB-D depth camera 004 optical axis and the fiber laser 005 optical axis at different working distances, which leads to the inability of fixed two-dimensional offset compensation to achieve accurate three-dimensional intersection. This is because step S400 loads the laser physical optical axis model parameters obtained by fitting from the three-dimensional scorched point cloud dataset through singular value decomposition, and accurately calculates the Cartesian space translation compensation amount at the current working depth based on the spatial triangle intersection geometric constraints, enabling the robotic arm to perform millimeter-level fine adjustments to translate the laser focus from the center position of the camera optical axis to the target blade position.

[0083] This embodiment employs a hardware actuation command chain with asymmetric time delay characteristics in step S500, ensuring that the enable time window of the centrifugal fan 008 strictly overlaps with the enable time window of the fiber laser 005 on the time axis. The negative pressure flow field is established before the laser cutting, ensuring that a stable suction field already exists within the horn-shaped gas collection hood 003 at the moment of cutting. The negative pressure flow field is shut off after the laser cutting, ensuring that the residual material in the flexible corrugated hose 002 is completely introduced into the material collection tank 007. Therefore, it overcomes the factor that the tender leaves scatter under the action of wind disturbance or vaporization impact force due to the lack of a synchronous physical collection force field after cutting, thus solving the problem of mismatch in the recovery sequence of flexible harvested materials.

[0084] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A vision-guided, multi-stage closed-loop laser harvesting method for robots, characterized in that, Includes the following steps: The registered color image and depth image are obtained. The target bud mask is extracted and the pixel coordinates of the petiole attachment point are located through the instance segmentation network. A sampling window is established in the depth image with the pixel coordinates as the center. After removing abnormal depth values, median filtering is performed on the remaining effective depth values ​​to obtain the noise-resistant reference depth. Based on the camera intrinsic parameters, the pixel coordinates and the noise-resistant reference depth are back-projected into three-dimensional physical coordinates in the camera coordinate system. The three-dimensional physical coordinates are mapped to the target position in the base coordinate system using hand-eye calibration parameters, and the robotic arm is driven to move to the observation range. The visual servo closed-loop loop is started, and the pixel residual and depth residual are extracted to form a feature error vector for each control cycle. The six-dimensional generalized control speed is solved by the damped least squares method through the image interaction Jacobian matrix and sent to the robotic arm until the feature error vector continuously satisfies the locking condition. Load the offline calibrated laser physical optical axis model parameters, solve for the Cartesian spatial translation compensation based on spatial intersection geometric constraints, and control the robotic arm to perform translational fine-tuning to align the laser focus with the target blade position; and A hardware actuation command chain with asymmetric time delay is sent to first activate the negative pressure airflow device and delay the flow field preloading time, then activate the laser to perform cutting, and after the cutting is completed, first shut down the laser and delay the pipeline emptying time before shutting down the negative pressure airflow device. The enabling time window of the negative pressure airflow device wraps around the enabling time window of the laser on the time axis.

2. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 1, characterized in that, in, The step of establishing a sampling window in the depth image centered on the pixel coordinates, removing abnormal depth values, and then performing median filtering on the remaining valid depth values ​​to obtain a noise-resistant reference depth includes: A pixel grid with a side length not less than a predetermined minimum size is established centered on the pixel coordinates of the petiole attachment point as a depth neighborhood sampling window; Within the depth neighborhood sampling window, depth readings with values ​​of zero, non-numerical values, and values ​​exceeding a preset working distance range are removed. The lower and upper limits of the preset working distance range correspond to the lower and upper limits of the effective ranging range of the depth camera, respectively. The median value of the remaining effective depth values ​​is taken as the noise reduction reference depth after being arranged numerically.

3. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 1, characterized in that, in, After obtaining the three-dimensional physical coordinates, the process also includes target temporal stabilization correlation processing: Calculate the three-dimensional Euclidean distance between the three-dimensional physical coordinates extracted in the current frame and the existing target coordinates in the previous frame; When the three-dimensional Euclidean distance is less than the set distance threshold, it is determined to be a continuous dynamic displacement of the same target, the life cycle timestamp of the target is reset and its tracking identifier is maintained. When the three-dimensional Euclidean distance exceeds the distance threshold, the original tracking identifier is unbound and a new tracking identifier is assigned. After the target time-domain image stabilization association processing, a stable list of harvesting targets with tracking tags is generated.

4. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 1, characterized in that, in, The method of solving the six-dimensional generalized control velocity using the damped least squares method via the image-interactive Jacobian matrix includes: Derivation of the image interaction Jacobian matrix based on the perspective projection model The Each row corresponds to the first-order partial derivative of a visual feature component with respect to the six-dimensional generalized velocity of the camera coordinate system. Each partial derivative term is calculated analytically based on the perspective projection geometry and the pixel coordinates and depth values ​​of the current frame. The pixel residual component and depth residual component in the feature error vector are subjected to mean normalization processing based on the range, and the maximum allowable residual value of each component in the actual working range is used as the normalization reference range. According to the formula Solving for the six-dimensional generalized control velocity ,in For velocity gain coefficient, The damping factor, It is the identity matrix. for The transpose of the matrix, This is the normalized feature error vector.

5. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 4, characterized in that, in, Before sending the six-dimensional generalized control velocity to the robotic arm, the six-dimensional generalized control velocity is subjected to amplitude limiting and dead-zone filtering: Commands whose absolute value in the linear velocity component exceeds the preset maximum linear velocity are truncated to the preset maximum linear velocity; Commands whose absolute value in the angular velocity component exceeds the preset maximum angular velocity are truncated to the preset maximum angular velocity; For signals whose absolute value in the linear velocity component is lower than the preset lower limit of the linear velocity dead zone, force them to be cleared to zero. Signals whose absolute value in the angular velocity component is lower than the preset lower limit of the angular velocity dead zone are forcibly cleared to zero.

6. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 1, characterized in that, in, The parameters of the offline calibrated laser physical optical axis model are obtained in the following way: A laser beam is controlled to emit a burning mark at multiple known spatial locations to generate burning mark points. The three-dimensional coordinates of the burning mark points are collected by a depth camera to form a three-dimensional burning point cloud dataset. Singular value decomposition is performed on the three-dimensional scorched point cloud dataset, and the right singular vector corresponding to the maximum singular value is extracted as the unit vector of the optical axis direction. The translation vector of the laser emission origin relative to the tool coordinate system is obtained by fitting the optimal straight line through the scorched mark point. The parameterized representation of the optimal straight line is obtained by... And the direction is A straight line in space.

7. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 6, characterized in that, in, The method for solving the Cartesian space translation compensation based on spatial intersection geometric constraints includes: Based on the parameterized ray equation Solve for the ray parameters by setting its vertical component equal to the vertical coordinate of the target point. The coordinates of the intersection point between the laser physical ray and the horizontal plane containing the target point are obtained, where The parameter on the ray is the distance from the origin of emission. Spatial coordinates of the location; Calculate the coordinate difference between the intersection point and the target point in the horizontal plane, and use the coordinate difference as the lateral translation compensation amount of the robotic arm end effector in Cartesian space.

8. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 3, characterized in that, in, It also includes asynchronous task scheduling based on state machines: The task management module receives the list of stable picking targets, converts each target into a picking task, and maintains a task queue arranged by priority. The task management module runs a finite state machine containing four states: pending, running, successful, and failed. When the robotic arm is idle, the status of the first task in the queue is changed from pending to running, and then asynchronously sent to the execution process. When a single harvesting process is successfully completed, the task status is updated to "successful" and archived. Then, the target location and the updated list of stable harvesting targets are re-executed, and the next harvesting cycle begins.

9. The vision-guided multi-stage closed-loop laser harvesting method for robots according to claim 8, characterized in that, in, It also includes exception handling and safe interruption: The timeout monitoring and status word daemon process runs synchronously in each sub-stage. If at least one of the following abnormal features is detected: path planner execution timeout, visual servo closed-loop residual not converging for multiple consecutive control cycles and showing a diverging trend, image interaction Jacobian matrix continuously in ill-conditioned condition number, or serial communication disconnection, the hardware safety interrupt signal is forcibly pulled low to cut off the power enable of the laser and negative pressure airflow device, and the current task is marked as failed. The task management module performs fault-tolerant scheduling of failed tasks based on a preset retry strategy.

10. A vision-guided multi-stage closed-loop laser picking system for robots, used to execute the vision-guided multi-stage closed-loop laser picking method according to any one of claims 1 to 9, characterized in that, include: The depth vision perception and 3D spatial localization module is configured to acquire registered color images and depth images, extract target bud masks and locate the pixel coordinates of petiole attachment points through an instance segmentation network, establish a sampling window in the depth image and perform median filtering to obtain noise-resistant reference depth, and back-project the depth to the camera coordinate system based on camera intrinsic parameters. The multi-stage motion control and servo solution module is configured to map the three-dimensional physical coordinates to the base coordinate system through hand-eye calibration parameters and drive the robotic arm to move to the observation range, start the visual servo closed loop, use the damped least squares method to solve the control speed to achieve closed-loop alignment, load the laser physical optical axis model parameters and solve the translation compensation amount based on spatial intersection geometric constraints to perform fine-tuning. The electromechanical timing control and negative pressure pneumatic actuation module is configured to send a hardware actuation command chain with asymmetric time delay characteristics. According to the timing constraint that the enable time window of the negative pressure airflow device wraps around the enable time window of the laser on the time axis, the module coordinates the control of the negative pressure airflow device and the laser to complete the cutting and material recovery.

Images