A control method, device, and robot equipment for a robot end effector

Abstract

This application provides a control method, device, and robot equipment for a robot end effector, relating to the field of robot control technology. By acquiring the original contact force signal from the robot end effector and image acquisition information of the object to be grasped, and performing visual processing, impedance correction, adaptive stiffness adjustment, cooperative dissipation, and velocity synthesis, dynamic adaptive visual-touch fusion control is achieved. This solves the problems of rigid control logic and energy conflict, and has dynamic adaptive visual-touch fusion control capability. It avoids abrupt transfer of control, reduces position overshoot and force impact, and improves the compliance and accuracy of grasping. At the same time, the problem of dynamic energy conflict is solved through cooperative dissipation processing.

Description

Technical Field

[0001] This application relates to the field of robot control technology, and more specifically, to a control method, device, and robot equipment for a robot end effector. Background Technology

[0002] With the rapid development of smart agriculture, agricultural robots are increasingly taking on harvesting tasks in greenhouses and open orchards. However, in the automated harvesting of bunch-like fruits such as grapes and tomatoes, the working environment has significant unstructured characteristics (such as wind disturbance and foliage obstruction), which poses a huge challenge to the end-effector grasping control. In order to balance the speed of approaching the target with the flexibility of contacting the fruit, the industry has generally begun to explore strategies for combined vision and tactile control. However, in practical engineering applications, existing vision-tactile fusion technologies still have significant pain points such as rigid control logic and lack of dynamic adaptability.

[0003] First, current vision-touch fusion solutions mostly employ "logic threshold switching" or "fixed-ratio linear weighting" control modes. In "logic threshold switching" mode, the robot relies entirely on visual guidance for approach; once the contact force exceeds a set threshold, the visual signal is forcibly interrupted, and the system abruptly switches to force control mode. This abrupt transfer of control leads to severe position overshoot and force impact at the end effector, easily damaging fragile fruit. While existing "fixed-ratio linear weighting" smooths the signal to some extent, its fusion strategy is static and cannot perceive the robot's current motion state (such as current speed and impact kinetic energy). This leads to an irreconcilable contradiction: if the weight transition is set too slowly, the robot cannot unload its stiffness in high-speed approach conditions, resulting in a hard collision; if the weight transition is set too quickly, visual guidance is lost prematurely during low-speed, precise operations, leading to decreased positioning accuracy. Current technology lacks an adaptive mechanism that can automatically adjust the fusion rate (i.e., variable slope) based on impact energy.

[0004] Secondly, existing multimodal control architectures often neglect the "conservation of dynamic energy" when handling heterogeneous data fusion. Visual servoing is a position-driven rigid control, while haptic impedance control is a force-driven flexible control; the two differ in their mathematical essence and topological structure. When the system simply synthesizes the two types of speed commands, the conflict in the control law generates "virtual energy" numerically, causing non-physical micro-oscillations at the actuator end in the fusion transition zone. This high-frequency oscillation may be negligible for grasping rigid industrial parts, but for delicate, biologically active fruit stems, it can easily cause mechanical damage to the skin or cause fruit to fall off.

[0005] Furthermore, in force control involving contact, existing technologies typically employ a general impedance control model, assuming the contact object to be an ideal rigid body or a simple spring. However, the surface of the stems of fruits such as grapes is covered with viscoelastic biological soft tissue. The nonlinear deformation at the moment of contact can cause a mismatch between the force sensor signal and the actual contact state (i.e., the appearance of a "pseudo-force" signal), thereby falsely triggering the robot's anti-collision logic or causing force control instability. In summary, the industry urgently needs a compliant grasping method that can adapt to different operating speeds, eliminate mode-switching oscillations at the physical level, and adapt to the characteristics of biological soft tissue.

[0006] There is currently no effective technical solution to the above problems. Summary of the Invention

[0007] The purpose of this application is to provide a control method, device, and robot equipment for the end effector of a robot, which realizes dynamic adaptive visual-touch fusion control, reduces control oscillation, improves adaptability to biological soft tissues, and enhances grasping stability and safety.

[0008] In a first aspect, this application provides a control method for a robot end effector, comprising the steps of: Acquire the raw contact force signal at the robot's end effector and the image acquisition information corresponding to the object to be grasped; Visual processing is performed based on the image acquisition information to obtain visual servo speed information; Based on the original contact force signal, impedance correction is performed in combination with preset contact surface deformation compensation information to obtain impedance correction speed information. Based on the current speed information and contact force change information of the robot end effector, and combined with the preset dynamic slope modulation information, adaptive stiffness adjustment is performed to obtain the visual-touch fusion weight information. Based on the visual-touch fusion weight information, combined with the visual servo speed information and impedance correction speed information, collaborative dissipation processing is performed to obtain collaborative dissipation information. Based on the cooperative dissipation information, the visual servo speed information, the impedance correction speed information, and the visual-touch fusion weight information, speed synthesis processing is performed to obtain the control speed information of the robot end effector; The robot end effector is driven according to the control speed information.

[0009] Through the above technical solutions, this application effectively solves the problems of insufficient dynamic adaptability, mode switching oscillation and pseudo force signal caused by contact surface deformation in visual-touch fusion control by adaptively adjusting the visual-touch fusion weights, introducing collaborative dissipation processing and impedance correction, and realizes compliant and efficient grasping of the robot end in unstructured environments.

[0010] Optionally, the step of performing visual processing based on the image acquisition information to obtain visual servo speed information includes: Based on the image acquisition information, the object to be captured and the corresponding feature state information of the object to be captured are determined; When the signal value of the original contact force signal is greater than a preset signal threshold, the response sensitivity of the robot end effector is dynamically adjusted according to the characteristic state information to obtain the visual servo control gain of the robot. Based on the visual servo control gain and combined with the basic gain information, the visual servo speed information is generated.

[0011] Through the above technical solution, the solution of this application can optimize the response according to real-time environmental changes, effectively avoiding the problems of severe position overshoot, force impact and decreased positioning accuracy that may occur in the traditional "logic threshold switching" or "fixed ratio linear weighting" mode, thus achieving significant progress in balancing the speed of approaching the target and the flexibility of contacting the fruit.

[0012] Optionally, the step of performing impedance correction based on the original contact force signal and combined with preset contact surface deformation compensation information to obtain impedance correction speed information includes: Based on the preset contact surface deformation compensation information, the original contact force signal is subjected to soft tissue nonlinear compensation to obtain the contact force correction signal; Obtain the current position of the robot end effector and the desired position output by the servo controller of the robot end effector; The current acceleration of the robot end effector is determined based on the contact force correction signal, the current position and desired position of the robot end effector, and the preset impedance model corresponding to the robot end effector. Based on the current acceleration, the velocity is predicted to obtain the expected velocity at the next moment; The expected velocity at the next moment is used as the impedance-corrected velocity information.

[0013] The above technical solutions provide more accurate and robust impedance correction capabilities for the control method of the robot end effector, enabling the entire vision-touch fusion control system to exhibit excellent compliance and adaptability when facing unstructured and fragile targets (such as clustered fruits), thereby improving the success rate and safety of grasping operations.

[0014] Optionally, the step of performing soft tissue nonlinear compensation on the original contact force signal based on preset contact surface deformation compensation information to obtain a contact force correction signal includes: Obtain preset contact surface deformation compensation information, wherein the contact surface deformation compensation information includes the preset soft tissue relaxation coefficient and reference characteristic force corresponding to the robot end effector; The original contact force signal is nonlinearly compensated using the soft tissue relaxation coefficient and the reference characteristic force to obtain the corrected contact force signal.

[0015] The solution proposed in this application solves the problem of spurious force signals caused by nonlinear deformation of biological soft tissue during impedance correction by introducing a soft tissue nonlinear compensation mechanism, thereby improving the accuracy and stability of force control.

[0016] Optionally, the adaptive stiffness adjustment based on the current velocity information and contact force change information of the robot end effector, combined with preset dynamic slope modulation information, to obtain the view-touch fusion weight information includes: The dynamic slope is calculated based on the current speed information and contact force change information of the robot end effector, combined with preset dynamic slope modulation information. Based on the dynamic slope, weighting is performed using the magnitude of the contact force correction signal and a preset transition threshold to obtain visual-touch fusion weight information.

[0017] The above technical solution ensures that the impedance controller can perform compliant control based on real contact force, avoiding misjudgment and control instability caused by spurious force signals. This enables the robot end effector to achieve more accurate and compliant force control response when in contact with biological soft tissue, significantly improving the reliability and safety of grasping operations.

[0018] Optionally, the collaborative dissipation processing is performed based on the visual-touch fusion weight information, combined with the visual servo speed information and the impedance correction speed information, to obtain collaborative dissipation information; Based on the visual servo speed information and the impedance correction speed information, determine the speed deviation information; Based on the visual-touch fusion weight information, the speed deviation information, and the preset fusion coordination coefficient, a coordination dissipation process is performed to obtain coordination dissipation information.

[0019] Optionally, the step of performing velocity synthesis processing based on the cooperative dissipation information, the visual servoing velocity information, the impedance correction velocity information, and the visual-touch fusion weight information to obtain the control velocity information of the robot end effector includes: Based on the visual servo speed information and the visual-touch fusion weight information, the weighted visual speed information is determined; Based on the impedance-corrected velocity information, velocity difference coupling processing is performed in conjunction with the visual-touch fusion weight information to obtain weighted impedance velocity information. The control speed information of the robot end effector is obtained by combining the weighted visual speed information and the weighted impedance speed information with the cooperative dissipation information to perform speed synthesis.

[0020] Optionally, it also includes: During the gripping and holding phase, the ratio of tangential force to normal force is monitored in real time. When the ratio exceeds the preset safe friction threshold, the ratio is corrected based on the safe friction threshold and the slip risk index information and displacement compensation step information corresponding to the robot end effector to obtain position correction information. A new reference position is obtained by superimposing the position correction information with the current reference position of the robot's end effector. Based on the new reference position, the normal gripping force of the robot end effector is updated.

[0021] Secondly, this application provides a control device for a robot end effector, comprising: The acquisition module is used to acquire the original contact force signal at the robot's end effector and the image acquisition information corresponding to the object to be grasped; The vision processing module is used to perform vision processing based on the image acquisition information to obtain vision servo speed information; The impedance correction module is used to perform impedance correction based on the original contact force signal and combined with preset contact surface deformation compensation information to obtain impedance correction speed information. The adjustment module is used to adaptively adjust the stiffness based on the current speed information and contact force change information of the robot end effector, combined with preset dynamic slope modulation information, to obtain visual-touch fusion weight information. The collaboration module is used to perform collaborative dissipation processing based on the visual-touch fusion weight information, combined with the visual servo speed information and impedance correction speed information, to obtain collaborative dissipation information. The synthesis module is used to perform velocity synthesis processing based on the cooperative dissipation information, the visual servo velocity information, the impedance correction velocity information, and the visual-touch fusion weight information to obtain the control velocity information of the robot end effector. A drive module is used to drive the end effector of the machine according to the control speed information.

[0022] Thirdly, this application provides a robot device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the control method for the robot end effector as described in any of the first aspects.

[0023] As can be seen from the above, the robot end-effector control method, device, and robot equipment provided in this application achieve dynamic adaptive visual-touch fusion control by acquiring the original contact force signal of the robot end-effector and the image acquisition information of the object to be grasped, performing visual processing, impedance correction, adaptive stiffness adjustment, cooperative dissipation, and velocity synthesis. This solves the problems of rigid control logic and energy conflict, and has dynamic adaptive visual-touch fusion control capability. It avoids the step transfer of control, reduces position overshoot and force impact, and improves the compliance and accuracy of grasping. At the same time, it solves the problem of dynamic energy conflict through cooperative dissipation processing.

[0024] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description and the accompanying drawings. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the control method for the robot end effector provided in an embodiment of this application.

[0026] Figure 2 This is a schematic diagram illustrating the response effect of the view-touch fusion weight information as the contact force changes under three different working conditions provided in the embodiments of this application.

[0027] Figure 3 This is a logical schematic diagram of the speed synthesis process provided in an embodiment of this application.

[0028] Figure 4 This is a schematic diagram of the structure of the control device for the robot end effector provided in an embodiment of this application.

[0029] Figure 5 This is a schematic diagram of the structure of the robot device provided in the embodiments of this application.

[0030] Labeling Explanation: 21. Acquisition Module; 22. Vision Processing Module; 23. Impedance Correction Module; 24. Adjustment Module; 25. Coordination Module; 26. Synthesis Module; 27. Drive Module; 111. Processor; 112. Communication Interface; 113. Memory; 114. Communication Bus. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0032] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0033] For example, in a grape-harvesting robot scenario, when the robotic arm's end effector approaches a bunch of grapes at a moderate speed, wind disturbance causes fluctuations in the target position, and the visual guidance signal is continuously updated. At the moment the end effector contacts the stem, the soft tissue on the stem surface undergoes nonlinear deformation, distorting the original contact force signal and creating a spurious force signal. If a logic threshold switching is used, the system may misinterpret it as a hard collision and force-switch to force control mode, causing a sudden change in the end effector position and resulting in the grapes falling off. If a fixed-ratio linear weighting is used, the transition rate of the fused weights cannot adapt to the current impact kinetic energy. In high-speed conditions, insufficient stiffness unloading can cause stem damage, or in low-speed conditions, excessively rapid weight transitions can cause visual guidance to fail, resulting in positioning drift. Simultaneously, the synthesis of visual servo commands and impedance correction commands generates high-frequency oscillations due to virtual energy, exacerbating mechanical damage to the stem's surface.

[0034] If the aforementioned issues are not addressed, the reliability of the vision-touch fusion control system will decrease. Especially in unstructured agricultural environments, micro-oscillations and position overshoot at the robotic arm's end effector will frequently damage delicate fruits, increasing the failure rate and potentially interrupting normal operations due to false force signals triggering anti-collision logic. Therefore, existing systems cannot meet the dual requirements of flexibility and positioning accuracy for automated harvesting of clustered fruits.

[0035] like Figure 1-3 As shown, this application provides a control method for a robot end effector, comprising the following steps: Step S1: Obtain the original contact force signal at the robot end and the image acquisition information corresponding to the object to be grasped; Step S2: Perform visual processing based on the image acquisition information to obtain visual servo speed information; Step S3: Based on the original contact force signal, impedance correction is performed in combination with the preset contact surface deformation compensation information to obtain impedance correction speed information. Step S4: Based on the current speed information and contact force change information of the robot end effector, and combined with the preset dynamic slope modulation information, adaptive stiffness adjustment is performed to obtain the visual-touch fusion weight information. Step S5: Based on the visual-touch fusion weight information, combined with the visual servo speed information and impedance correction speed information, perform collaborative dissipation processing to obtain collaborative dissipation information. Step S6: Based on the cooperative dissipation information, visual servo speed information, impedance correction speed information, and visual-touch fusion weight information, speed synthesis processing is performed to obtain the control speed information of the robot end effector. Step S7: Drive the robot end effector according to the control speed information.

[0036] Image acquisition information refers to visual data about the working environment and the target object (such as fruit to be grasped) obtained by visual sensors (e.g., monocular camera, binocular camera, depth camera, or RGB-D camera) mounted on the robot's end effector. This data is usually in the form of images or video streams. This information is the basis for the robot's visual perception and localization.

[0037] Raw contact force signal: This refers to the unprocessed force feedback signal directly measured by force sensors when the robot's end effector comes into contact with the external environment or target object. This signal reflects the intensity and direction of the contact and is the basis for the robot's tactile perception and force control.

[0038] Visual servo speed information: refers to the motion speed command of the robot's end effector under visual guidance, calculated through visual processing algorithms based on image acquisition information. This speed information aims to enable the robot's end effector to accurately and quickly approach or track targets.

[0039] Impedance-corrected velocity information: This refers to the motion velocity command of the robot end effector in force control mode, calculated using an impedance control algorithm based on the original contact force signal and contact surface deformation compensation information. This velocity information aims to enable the robot end effector to interact smoothly with the environment and effectively adjust the contact force.

[0040] Current speed information: This refers to the real-time speed of the robot's end effector as it approaches the target object. This information reflects the relative motion state between the robot's end effector and the target.

[0041] Cooperative dissipative information: This refers to the compensation term introduced to eliminate potential energy conflicts between visual servo velocity information and impedance-corrected velocity information when they are fused. This information aims to smooth the control signal by introducing "virtual damping" and prevent non-physical oscillations during mode switching.

[0042] Control speed information: This refers to the comprehensive speed command used to drive the robot's end effector movement after visual-touch fusion and cooperative dissipation processing. This information is the basis for the actual movement of the robot's end effector.

[0043] This method first acquires the raw contact force signal from the robot's end effector and the image acquisition information corresponding to the object to be grasped. The image acquisition information can be obtained through a camera installed at or near the robot's end effector, such as a monocular camera, a binocular camera, a depth camera, or an RGB-D camera; the raw contact force signal can be obtained through a six-dimensional force sensor installed on the robot's end effector wrist or fingertip, such as a piezoelectric force sensor or a strain gauge force sensor.

[0044] Visual servo speed information is obtained by performing visual processing on image acquisition information. In one implementation, visual processing may include steps such as object detection, feature point extraction, and target pose estimation. For example, image processing algorithms can be used to identify the position and orientation of a fruit to be grasped, and calculate the speed at which the robot's end effector needs to move to approach the fruit. In another implementation, visual processing may be limited to identifying the target area and calculating a simple visual guidance speed based on the center point of the target area.

[0045] The following example will provide a more detailed explanation of the above technical solution: First, the robot's end effector acquires image information of the object to be grasped (such as a bunch of grapes) using its onboard camera, and simultaneously obtains the raw contact force signal when it comes into contact with the grape bunch using its force sensor. This information is transmitted to the robot controller in real time.

[0046] Next, based on the image acquisition information, the robot controller performs visual processing. For example, it extracts the region of interest (ROI) from the image acquisition information, uses the YOLO algorithm to detect targets while outputting a pixel-level mask for each target, and uses a Kalman filter to perform prediction processing on consecutive image frames to predict the coordinates and velocity magnitude of the grape bunch at the next moment. When the force sensor detects a contact force signal greater than a preset signal threshold, the system determines that the preset contact condition is met. At this time, the controller dynamically adjusts the response sensitivity of the robot end effector according to the predicted velocity magnitude. For example, the controller can introduce a preset bandwidth adaptive adapter and calculate the visual servo control gain according to a preset formula. According to the predicted velocity modulus Combined with dynamic response sensitivity factor Perform dynamic gain calculation to obtain the current dynamic gain. And based on this dynamic gain Combined with the pseudo-inverse matrix of the image Jacobian matrix The predicted coordinates of the object to be crawled (i.e., the bunch of grapes) at the next moment. and the expected coordinates of the image center Calculations are performed to determine the visual servo control gain of the robot's end effector based on the calculation results, i.e., visual servo control gain = The preset bandwidth adaptive adapter can be calculated according to the visual servo control gain formula. The visual servo control gain of the robot can be calculated by using the dynamic response sensitivity factor, predicted velocity modulus, pseudo-inverse of the image Jacobian matrix, predicted coordinates of the object to be grasped at the next moment, and expected coordinates of the image center. For dynamic response sensitivity factor, To predict the velocity modulus, This is the pseudo-inverse of the image Jacobian matrix. The pseudo-inverse of the image Jacobian matrix is ​​the inverse mapping operator in visual servo control, used to map image feature errors to control speed commands for the robot's end effector. The desired image center coordinates are the pre-calibrated principal point coordinates of the camera. The principal point coordinates refer to the coordinates of the intersection of the optical axis and the image plane (usually close to the geometric center of the image resolution). For example, the principal point coordinates of a 640*480 image are approximately (320, 240). These principal point coordinates can be directly set as the desired image center coordinates to ensure the object to be captured is ultimately aligned with the image center. Subsequently, the visual servo controller can calculate the formula based on the preset visual servo speed information: According to the visual servo control gain Combined with basic gain information Calculations are performed to obtain visual servo speed information. ,in, For visual servo speed information, This is based on the gain information. This dynamic adjustment allows the visual servoing system to adaptively adjust the bandwidth according to the swaying speed of the grape bunch, achieving "zero phase difference" tracking and avoiding the phase lag or overshoot oscillation that occurs in traditional visual servoing when the target is swaying.

[0047] Simultaneously, impedance correction is performed based on the original contact force signal and pre-set contact surface deformation compensation information. Specifically, the controller acquires the pre-set soft tissue relaxation coefficient and reference characteristic force. Using these parameters, a nonlinear compensation function is employed to perform soft tissue nonlinear compensation (i.e., soft tissue filtering) on ​​the original contact force signal to obtain the corrected contact force signal. For example, the expression of the nonlinear compensation function is: In the formula, This is the soft tissue relaxation coefficient. This is the original contact force signal. For reference characteristic force, To obtain the contact force correction signal, soft tissue nonlinear compensation is performed on the original contact force signal according to the expression of the nonlinear compensation function, thereby calculating the contact force correction signal. This nonlinear compensation function adaptively scales the input force in the initial contact phase, effectively filtering out the "pseudo-rigid force" signal generated by the deformation of the grape stem skin, preventing the second-order impedance controller from misjudging soft contact as a hard collision. Subsequently, the current position of the robot's end effector is obtained, along with the desired position output by the robot's end effector servo controller. Based on the contact force correction signal, the current position of the robot's end effector, the desired position, and the corresponding preset impedance model, the current acceleration of the robot's end effector is determined. Velocity prediction is performed based on the current acceleration to obtain the desired velocity at the next moment. This desired velocity at the next moment is used as the impedance correction velocity information.

[0048] After generating visual servo speed information and impedance-corrected speed information, the system adaptively adjusts the stiffness based on the robot's end effector's current speed and contact force changes, combined with preset dynamic slope modulation information, to obtain visual-touch fusion weight information. Specifically, the controller can calculate the dynamic slope according to a preset formula. Perform dynamic slope calculation to determine the current dynamic slope. ;in, Based on the basic steepness coefficient, Impact velocity sensitivity factor, unit: , This is the current speed information of the end effector, in m / s. It is an exponent of speed, and Greater than 1, This is the force rate sensitivity factor, with units of s / N. This represents the contact force variation information, measured in N / s. The dynamic slope is adjusted in real-time based on the current speed and contact force variation information, allowing the transition curve of the view-touch fusion weighting information to adaptively "deform."

[0049] For example, when a high-speed impact condition is detected, the corresponding operating condition curve is as follows: Figure 2 The curve for condition A, as shown, exhibits a very high dynamic slope K due to the high-speed impact. The curve for condition A shows a near-vertical step drop, causing the visual-touch fusion weight information to... Instantaneous zeroing within an extremely short stroke enables rapid unloading and "soft landing" of the end effector stiffness, effectively absorbing impact kinetic energy; when a normal grasping condition is detected, the corresponding working condition curve is as follows: Figure 2The curve for condition B, as shown, has a smaller dynamic slope K due to the lower speed. From the curve of condition B, it can be seen that the visual-touch fusion weight information for this conventional grasping condition decreases slowly with increasing contact force, which is beneficial for improving the smoothness of the end effector's operation. When a fine-operation condition is detected, the corresponding curve is as follows: Figure 2 The curve for condition C, as shown, has a relatively small dynamic slope K due to the relatively low speed. From the curve for condition C, it can be seen that the visual-touch fusion weight information decays more slowly with increasing contact force, thus achieving "fine operation" of the end effector. Figure 2 The response effect diagram shown illustrates the core advantages of this solution. By calculating the dynamic slope in real time, the visual-touch fusion weight information and the control target (impact velocity, contact force) have an optimal evolution trajectory that is compatible with each other, thus solving the technical contradiction that the traditional single stiffness curve cannot simultaneously take into account "high-speed impact protection" and "low-speed accuracy".

[0050] Subsequently, based on the view-touch fusion weight information, combined with visual servo speed information and impedance correction speed information, collaborative dissipation processing is performed to obtain collaborative dissipation information. Specifically, the controller determines speed deviation information based on visual servo speed information and impedance correction speed information, and performs collaborative dissipation processing based on the view-touch fusion weight information, speed deviation information, and preset fusion collaboration coefficients to obtain collaborative dissipation information. For example, the controller can calculate collaborative dissipation information according to a preset formula. Cooperative dissipation calculation is performed to obtain cooperative dissipation information. ,in, To coordinate dissipated information, It is a bell-shaped activation function. for Visual servo speed information at any given moment. for Impedance correction velocity information at any given time. The preset fusion and coordination coefficient is used. When there is a significant difference between visual and tactile speed commands, this coordination dissipation information applies a "virtual viscous damping" proportional to the speed difference, actively absorbing and dissipating the acceleration jump energy caused by the change in the topology of the control model. From the perspective of energy conservation, it smooths the control signal waveform and effectively eliminates the non-physical micro-oscillations commonly found in traditional multimodal control.

[0051] Finally, based on the cooperative dissipation information Visual servo speed information Impedance correction speed information and visual-touch fusion weight information Velocity synthesis is performed to obtain the control velocity information of the robot's end effector. Specifically, such as... Figure 3As shown, it can be done according to the preset dynamic coupling equations. Velocity synthesis calculations are performed to obtain the control velocity information of the robot's end effector. Based on visual servoing velocity information and visual-touch fusion weight information, weighted visual velocity information is determined. Based on impedance-corrected velocity information, velocity difference coupling is performed using view-touch fusion weighting information to obtain weighted impedance-velocity information. We employ weighted visual velocity information and weighted impedance velocity information, combined with cooperative dissipation information. Speed ​​synthesis is performed to obtain the final control speed information. ,in, The control speed information at time t, The visual servo speed information at time t. This represents the impedance-corrected velocity information at time t. Based on this control velocity information, the robot's end effector is driven to perform compliant grasping.

[0052] Through the above technical solution, this application effectively solves the problems of insufficient dynamic adaptability, mode switching oscillation, and pseudo force signal caused by contact surface deformation in traditional visual-touch fusion control.

[0053] Compared to the traditional "logic threshold switching" mode, this application introduces dynamic slope modulation information for adaptive stiffness adjustment, enabling the transition curve of the visual-touch fusion weight information to "deform" in real time according to the robot's current speed and contact force changes. For example, when approaching a bunch of grapes at high speed, the visual-touch fusion weight information rapidly decreases, achieving rapid stiffness unloading and avoiding the severe position overshoot and force impact caused by the forced interruption of visual signals and abrupt switching of force control modes in the traditional mode. During low-speed, fine-grained operations, the transition curve of the visual-touch fusion weight information becomes smoother, allowing vision and touch to coexist over a longer stroke. This maintains visual positioning while performing delicate force-position and posture adjustments, avoiding the problem of premature loss of visual guidance leading to decreased positioning accuracy in the traditional mode. This variable slope adaptive mechanism significantly improves the dynamic adaptability of the control system.

[0054] Compared to existing "fixed-proportion linear weighting" schemes, this application introduces cooperative dissipation information in the velocity synthesis process. When a significant difference exists between visual servo velocity information and impedance-corrected velocity information, this cooperative dissipation term applies a "virtual viscous damping" proportional to the velocity difference, actively absorbing and dissipating the acceleration jump energy caused by changes in the control model's topology. For example, when visual guidance velocity and tactile correction velocity conflict, the cooperative dissipation term can eliminate virtual oscillations generated during heterogeneous data fusion from a fundamental dynamic perspective, thereby avoiding non-physical micro-oscillations caused by control law conflicts in traditional fixed-proportion weighting schemes and protecting the fragile results.

[0055] Furthermore, this application introduces soft tissue nonlinear compensation based on contact surface deformation compensation in the impedance correction stage. By obtaining a preset soft tissue relaxation coefficient and reference characteristic force, a nonlinear compensation function is used to process the original contact force signal to obtain the contact force correction signal. For example, when the grape stem epidermis undergoes elastic compression deformation, the compensation function can effectively filter out the "pseudo-rigid force" signal generated by epidermal deformation, preventing the impedance controller from misjudging soft contact as a hard collision and prematurely triggering position retraction. This allows the robot end effector to closely adhere to the surface of the stem before making precise torque compliance adjustments, solving the problem of force feedback signal distortion caused by assuming the contact object to be an ideal rigid body or a simple spring in traditional impedance control models, and improving the accuracy and stability of force control.

[0056] In summary, this application effectively solves the problems of insufficient dynamic adaptability, mode switching oscillation, and pseudo-force signal caused by contact surface deformation in visual-touch fusion control by adaptively adjusting the visual-touch fusion weights, introducing cooperative dissipation processing, and impedance correction, thereby achieving compliant and efficient grasping of the robot end effector in unstructured environments.

[0057] In some implementations, visual processing is performed based on image acquisition information to obtain visual servo speed information, including: Based on image acquisition information, the object to be captured and its corresponding feature state information are determined; the feature state information includes the predicted velocity magnitude of the object to be captured and its predicted coordinates at the next moment. When the signal value of the original contact force signal is greater than the preset signal threshold, the response sensitivity of the robot end is dynamically adjusted according to the characteristic state information to obtain the visual servo control gain of the robot. Based on the visual servo control gain, combined with the basic gain information, visual servo speed information is generated.

[0058] Obtained by processing image acquisition information (e.g., using Kalman filters for target tracking and motion prediction), it is used to dynamically assess the motion trend of the target object, thereby providing forward-looking data for the robot's end effector response. The raw contact force signal refers to the mechanical data directly measured by force sensors (e.g., six-dimensional force / torque sensors or tactile sensor arrays) mounted on the robot's end effector or gripper. This raw contact force signal reflects the interaction force between the robot's end effector and the environment or the object to be grasped, and is an important basis for tactile perception. The preset signal threshold is a pre-defined force value.

[0059] The object to be grasped refers to the target object that the robot's end effector needs to grasp, such as fruit in crops. Identifying the object to be grasped is the primary task of vision processing, which is usually achieved through computer vision algorithms such as image recognition and object detection.

[0060] When the signal value of the original contact force signal exceeds this threshold, the system determines that the robot end effector has made contact with the target, or that the contact force has reached a level requiring specific processing. This threshold can be set empirically based on the actual application scenario and the fragility of the object to be grasped, or obtained through experimental calibration. In this context, response sensitivity refers to the degree of response of the visual servo controller to changes in the target state. Dynamically adjusting the response sensitivity means that the system can change its response intensity to visual errors in real time according to the motion state of the object to be grasped (especially the predicted velocity modulus) to adapt to different operating conditions. Visual servo control gain refers to a coefficient used to adjust the control output intensity in the visual servo control law. By dynamically adjusting the response sensitivity, an adaptive visual servo control gain can be obtained. This visual servo control gain directly affects the speed at which the robot end effector corrects visual errors, thus determining the speed and accuracy of visual guidance. Base gain information refers to a preset base gain information. This base gain information provides basic visual servo control intensity when there is no need for dynamic adjustment (e.g., when the target is stationary or moving at low speed), ensuring the stability and accuracy of the system under normal conditions. It serves as the benchmark for dynamically adjusting the gain, and together with the dynamic adjustment section, they constitute the complete visual servo control gain. Specifically, the visual servo speed information can be calculated based on a preset visual servo speed information calculation formula: According to the visual servo control gain and basic gain information Calculations are performed to obtain visual servo speed information. Visual servo speed information refers to the desired motion speed command of the robot's end effector under visual guidance, calculated based on the visual servo control gain and fundamental gain information. This visual servo speed information is used to drive the robot's end effector to move towards the object to be grasped, thereby achieving precise visual tracking and positioning.

[0061] The core of this application's solution lies in introducing a dynamic adjustment mechanism to address the limitations of traditional visual servoing in unstructured environments and contact conditions. This method first uses advanced computer vision technology, based on image acquisition information, to accurately identify and determine the object to be grasped, and further extracts its characteristic state information, including the predicted velocity modulus of the object and its predicted coordinates at the next moment. This predictive information provides the system with prior knowledge of the target's motion, enabling the control system to predict the target's future state rather than relying solely on current observations. Subsequently, the system continuously monitors the original contact force signal. When the signal value of the original contact force signal exceeds a preset signal threshold, it indicates that the robot's end effector has made contact with the target or is about to make contact, at which point the system determines that the preset contact conditions are met. At this critical moment, the system no longer uses a fixed visual servoing gain, but instead dynamically adjusts the response sensitivity of the robot's end effector based on the previously determined characteristic state information. Specifically, this adjustment mechanism introduces a dynamic response sensitivity factor and a predicted velocity modulus, allowing the visual servoing control gain to adaptively increase as the speed of the object to be grasped increases. This means that when the target is moving at high speed, the system can improve its response speed to correct visual errors at a faster frequency, effectively compensating for the inherent time delay in visual acquisition and processing, thereby achieving "zero phase difference" tracking and avoiding collisions caused by visual lag. Conversely, when the target tends to be stationary or moving at low speed, the gain will automatically fall back to the baseline value, effectively suppressing system self-oscillations caused by high-frequency observation noise and ensuring the stability of precise operation. Finally, based on the dynamically adjusted visual servo control gain and combined with the preset baseline gain information, the final visual servo speed information is generated. This visual servo speed information incorporates dynamic gain characteristics, ensuring the smoothness and adaptability of the visual guidance signal throughout the approach and contact process. In this way, the solution of this application can optimize the response according to real-time environmental changes, effectively avoiding the problems of severe position overshoot, force impact, and decreased positioning accuracy that may occur in traditional "logic threshold switching" or "fixed ratio linear weighting" modes, thus achieving significant progress in balancing the speed of approaching the target and the compliance of contacting the fruit. This dynamically adjusted vision processing module, combined with the overall robot end-effector control method, can provide more accurate and adaptive visual guidance speed for subsequent vision-touch fusion collaborative dissipation processing, thereby improving the robustness and compliance of the entire control system.

[0062] In some implementations, impedance correction is performed based on the original contact force signal and combined with preset contact surface deformation compensation information to obtain impedance correction speed information, including: Based on the preset contact surface deformation compensation information, the original contact force signal is subjected to soft tissue nonlinear compensation to obtain the contact force correction signal; Obtain the current position of the robot's end effector and the desired position output by the servo controller of the robot's end effector; The current acceleration of the robot end effector is determined based on the contact force correction signal, the current position and desired position of the robot end effector, and the preset impedance model corresponding to the robot end effector. Based on the current acceleration, the velocity is predicted to obtain the expected velocity at the next moment; The expected velocity at the next moment is used as the impedance-corrected velocity information.

[0063] This application's solution addresses the issue of spurious force signals caused by nonlinear deformation of biological soft tissue during impedance correction by introducing a soft tissue nonlinear compensation mechanism, thereby improving the accuracy and stability of force control. Specifically, upon receiving the original contact force signal from the robot's end effector, the system first performs soft tissue nonlinear compensation based on preset contact surface deformation compensation information. This compensation process effectively filters out spurious force signals generated by soft tissue deformation such as fruit skin, ensuring that the obtained contact force correction signal accurately reflects the actual contact state between the robot's end effector and the target object. This avoids misjudging soft contact as hard collision, preventing premature triggering of anti-collision logic or force control instability.

[0064] Specifically, the current position of the robot end effector is obtained, and the desired position output by the servo controller of the robot end effector is also obtained. The desired position is the target pose preset by the robot end effector, which can be obtained by the vision servo controller directly generating the target pose based on the features of the object to be grasped, and then determining the generated target pose as the desired position. This application embodiment does not impose specific restrictions on the method of obtaining the desired position. The current position is the current actual pose of the robot end effector. Specifically, the current actual pose of the robot end effector can be obtained by reading the positions of each joint through the joint encoder and combining the robot DH parameters with forward kinematics calculation, and then determining the calculated actual pose as the current position of the robot end effector. This application embodiment does not impose specific restrictions on the method of obtaining the current position of the robot end effector.

[0065] Subsequently, based on the corrected contact force correction signal, the current position and desired position of the robot's end effector, and the corresponding preset impedance model, the current acceleration of the robot's end effector is determined. Specifically, the corrected contact force correction signal, the current position and desired position of the robot's end effector can be input into the corresponding preset impedance model, causing the impedance model to be calculated according to a preset expression for a second-order impedance model. For example, according to the preset expression for a second-order impedance model: Calculations are performed to obtain the current acceleration. ,in, For the desired inertia of the impedance model, Damping for the impedance model, Here is the stiffness matrix of the impedance model. This represents the current position in Cartesian space within the end effector. The desired position of the robot's end effector. The current speed of the robot's end effector. The desired velocity corresponding to the current velocity of the robot's end effector. The current acceleration of the robot's end effector. The desired acceleration corresponding to the current acceleration of the robot's end effector. The contact force correction signal is used; specifically, the desired velocity is the first-order time rate of change of the desired position. It can be obtained by first low-pass filtering the desired position sequence to suppress planning noise, and then using the first-order numerical difference method to calculate the desired position after filtering from adjacent control cycles and the control sampling period. This application embodiment does not limit the specific method for obtaining the desired velocity. The current velocity is a scalar determined based on the current velocity information of the robot's end effector. The current velocity is the first-order time rate of change of the current position. It can be obtained by first low-pass filtering the actual position sequence to suppress high-frequency sensor noise, and then using the first-order numerical difference method to calculate the actual position after filtering from adjacent control cycles and the control sampling period, or by... The robot controller directly feeds back the current velocity; this process is existing technology and will not be detailed here. The desired acceleration is the second-order time rate of change of the desired position (or the first-order time rate of change of the desired velocity). It can be obtained by first low-pass filtering the desired position or velocity sequence and then using the numerical difference method. Two paths can be used: one is to calculate it using the square of the filtered desired position and the sampling period over three adjacent control cycles; the other is to calculate it using the filtered desired velocity and the sampling period over two adjacent control cycles. Both paths can stably obtain the desired acceleration. Alternatively, the trajectory planning module in the robot's end effector can directly output the desired acceleration at each moment. This application does not limit the specific method for obtaining the desired acceleration. By simplifying the expression of the preset second-order impedance model, the explicit equation for the current acceleration can be obtained: Then, the current acceleration can be calculated based on the explicit equation of the current acceleration; then, the velocity can be predicted based on the current acceleration to obtain the expected velocity at the next moment. For example, the expected velocity at the next moment can be predicted based on the current acceleration by discretizing and recursively calculating the current acceleration.

[0066] As a specific example of this application, the Euler method can be used to discretize and recursively calculate the current acceleration, assuming the control period is... Therefore, according to the control cycle and current acceleration Combined with the current speed The formula for the expected velocity at the next moment is preset. Calculations are performed to obtain the expected velocity at the next moment. And the expected velocity at the next moment is used as the impedance-corrected velocity information. At the current speed, This represents the current acceleration.

[0067] Through the above process, the solution in this application ensures that the impedance-corrected velocity information is generated based on real and valid contact force signals, rather than being interfered with by spurious force signals. This enables the robot end effector to achieve more precise and compliant force control when in contact with fragile biological soft tissues (such as fruit stalks), effectively avoiding damage to the target object. This precisely corrected impedance-corrected velocity information, as a key input in the aforementioned robot end effector control method, can be more reliably combined with visual servo velocity information for dissipative processing and velocity synthesis, thereby significantly improving the robustness and adaptability of the entire visual-touch fusion compliant grasping control method. Especially when handling fragile target grasping tasks in unstructured environments, it can provide more stable and safer control performance.

[0068] In some implementations, based on preset contact surface deformation compensation information, soft tissue nonlinear compensation is performed on the original contact force signal to obtain a contact force correction signal; including: Obtain preset contact surface deformation compensation information, which includes the preset soft tissue relaxation coefficient and reference characteristic force corresponding to the robot end effector; By using a preset soft tissue relaxation coefficient and combining it with a reference characteristic force, the original contact force signal is nonlinearly compensated to obtain a corrected contact force signal.

[0069] Obtaining preset contact surface deformation compensation information refers to determining the parameters needed to compensate for contact surface deformation before the robot's end effector contacts the target object or during system initialization, through pre-setting or experimental calibration. For example, force-deformation tests can be conducted on different biological soft tissues to establish their mechanical models, thereby extracting the corresponding compensation parameters; alternatively, matching compensation information can be retrieved from a preset database based on the known material properties of the object to be grasped. The soft tissue relaxation coefficient characterizes the maximum relative deformation rate or compliance of biological soft tissue under external force. This coefficient can be determined by conducting compression experiments on specific biological tissues (such as grape stems) and measuring their deformation response under different forces; alternatively, it can be estimated based on empirical values ​​or through simulation methods such as finite element analysis. The reference characteristic force defines the characteristic force range for when the soft tissue undergoes significant nonlinear deformation or enters a specific deformation stage. This force value can be determined by experimentally observing the inflection point of the soft tissue deformation curve. For example, when the contact force reaches a certain threshold, the deformation behavior of the soft tissue transitions from the elastic stage to the plastic stage, and this threshold can be used as a reference characteristic force; or, it can be set according to the typical range of the expected contact force of the robot end effector.

[0070] Among them, the exponent term This makes the compensation effect more significant when the original contact force signal is small, and the compensation effect gradually weakens as the original contact force signal increases, thus simulating the nonlinear response of soft tissue at different stress stages.

[0071] This application addresses the control problem caused by spurious force signals during contact with biological soft tissues by introducing a nonlinear compensation mechanism for the original contact force signal. Specifically, when the robot's end effector contacts the target object (a fruit stalk), the force sensor collects the original contact force signal. Because the fruit stalk surface has viscoelastic soft tissue, even if the actual contact force is small in the initial stage of contact, the instantaneous deformation of the soft tissue can cause the force sensor to output a falsely high "spurious force" signal. To accurately reflect the true contact state, this application first obtains preset contact surface deformation compensation information, which includes a soft tissue relaxation coefficient and a reference characteristic force determined for specific soft tissue characteristics. These parameters are pre-calibrated or set based on the mechanical properties of the soft tissue, quantifying the compliance and deformation characteristics of the soft tissue. Subsequently, the system uses these preset parameters to perform nonlinear compensation processing on the original contact force signal, specifically through the formula... To achieve this, in the formula, This is the soft tissue relaxation coefficient. This is the original contact force signal. For reference characteristic force, For the contact force correction signal; the exponential decay term in this formula When the original contact force signal is small, a large compensation factor is generated, effectively reducing the spurious force component caused by soft tissue deformation in the original force signal. As the original contact force signal increases, this compensation factor gradually decreases, making the corrected contact force signal closer to the original force signal, reflecting the characteristics of soft tissue deformation tending to saturate or entering a more rigid stage. In this way, the original contact force signal is transformed into a more realistic contact force correction signal. This contact force correction signal is then used in the subsequent impedance correction module to calculate impedance correction speed information. This processing ensures that the impedance controller can perform compliant control based on the real contact force, avoiding misjudgments and control instability caused by spurious force signals. This enables the robot end effector to achieve a more accurate and compliant force control response when in contact with biological soft tissue, significantly improving the reliability and safety of grasping operations.

[0072] In some implementations, adaptive stiffness adjustment is performed based on the current velocity information and contact force change information of the robot's end effector, combined with preset dynamic slope modulation information, to obtain view-touch fusion weight information, including: The dynamic slope is calculated based on the current speed information and contact force change information of the robot end effector, combined with the preset dynamic slope modulation information. Based on the dynamic slope, the weighting is performed by combining the magnitude of the contact force correction signal and the preset transition threshold to obtain the visual-touch fusion weighting information.

[0073] The current velocity information of the robot's end effector refers to the velocity vector or magnitude of the robot's end effector moving towards the target object in space. This current velocity information is one of the key parameters for evaluating the robot's current motion state. Methods for obtaining this information may include, but are not limited to: real-time calculation using joint encoder data from the robot body combined with the robot's kinematic model; or indirect acquisition by using external sensors, such as LiDAR, depth cameras, or ultrasonic sensors, to measure the rate of change of distance between the robot's end effector and the target object. The contact force change information refers to the rate of change of contact force over time when the robot's end effector comes into contact with the target object. This contact force change information reflects the dynamic characteristics of the contact process, such as the suddenness or smoothness of the contact. Methods for obtaining this contact force change information may include, but are not limited to: performing first-order differential processing on the raw contact force signal collected by the force sensor to obtain the instantaneous rate of change of force; or filtering the raw contact force signal to eliminate high-frequency noise and extract the trend of force change.

[0074] The preset dynamic slope modulation information refers to a set of pre-defined parameters used to adjust the dynamic slope calculation process. For example, this dynamic slope modulation information may include a base steepness coefficient, an impact velocity sensitivity factor, and a force rate sensitivity factor. These parameters can be pre-calibrated or stored in a lookup table according to different application scenarios, target object characteristics, or desired control response characteristics. The dynamic slope is a real-time changing parameter calculated based on the robot end effector's current velocity information, contact force change information, and the preset dynamic slope modulation information. Specifically, nonlinear dynamic slope modulation can be performed according to the following formula: In the formula, For dynamic slope, Based on the basic steepness coefficient, As an impact velocity sensitive factor, The current speed of the end effector. It is an exponent of speed, and Greater than 1, As a force-sensitive factor, This refers to the contact force change information. The dynamic slope directly determines the rate or steepness of the transition from visual dominance to tactile dominance in the visual-tactile fusion weight information when the magnitude of the contact force correction signal changes. Real-time adjustment of the dynamic slope allows the visual-tactile fusion transition process to adaptively match the robot's current operating state. The magnitude of the contact force correction signal refers to the numerical value of the contact force signal after soft tissue nonlinear compensation processing. This magnitude reflects the actual and effective contact force between the robot's end effector and the target object, eliminating "pseudo-force" signals caused by soft tissue deformation of the target object. The preset transition threshold is a preset force value used to define the visual-tactile switching center point during the visual-tactile fusion weight processing. This threshold typically represents a critical contact force; when the magnitude of the actual contact force correction signal reaches or exceeds this threshold, the visual-tactile fusion weight information will begin to significantly shift from visual dominance to tactile dominance. This threshold can be set according to the physical characteristics of the target object, the requirements of the grasping task, or experience. Weight processing refers to the calculation of weights based on the dynamic slope, the magnitude of the contact force correction signal, and the preset transition threshold, according to a preset weight calculation formula. Weights are calculated, and the result is determined as the view-touch fusion weight information. In the formula, For visual-touch fusion weight information, This is the transition threshold. To correct the magnitude of the contact force correction signal, This is the dynamic slope. The processing maps the input parameters to a weight value between 0 and 1, reflecting the relative contribution of visual servo speed information and impedance-corrected speed information to the final control speed information. The visual-tactile fusion weight information is a scalar value between 0 and 1, dynamically representing the relative importance of visual servo speed information and impedance-corrected speed information in the robot's end-effector control speed information. When the visual-tactile fusion weight information is close to 1, it indicates that visual control is dominant at the current stage; when it is close to 0, it indicates that tactile (impedance) control is dominant. This weight information is a key parameter for achieving compliant fusion control of vision and touch.

[0075] The proposed solution calculates the dynamic slope based on the robot end effector's current velocity and contact force change information, combined with preset dynamic slope modulation information. Specifically, when the robot end effector approaches the target at a higher current velocity, or when the contact force change information indicates a more intense contact process, the preset dynamic slope modulation information significantly increases the calculated dynamic slope, making the transition curve of the view-touch fusion weight information steeper. Conversely, when the robot end effector approaches at a lower velocity or the contact process is smooth, the dynamic slope decreases accordingly, making the weight transition curve flatter. Subsequently, based on this dynamic slope, weight processing is performed using the magnitude of the contact force correction signal and a preset transition threshold to finally obtain the view-touch fusion weight information. In this way, the view-touch fusion weight information can be adjusted in real time and adaptively according to the actual motion state and contact force feedback of the robot end effector. This adaptive stiffness adjustment mechanism works closely with the cooperative dissipation processing and velocity synthesis processing in the aforementioned robot end effector control method. The view-touch fusion weight information not only directly participates in the synthesis of the final control velocity information but also indirectly adjusts the energy absorption intensity of the system during mode switching by influencing the calculation of cooperative dissipation information. This dynamically adjusted weight information enables the entire control system to smoothly and efficiently switch between the speed of visual guidance and the softness of tactile feedback according to the actual working conditions, effectively avoiding the control instability and shock problems caused by fixed weights or abrupt switching in traditional methods.

[0076] In some implementations, collaborative dissipation information is obtained by combining visual servo speed information and impedance correction speed information with visual fusion weight information and performing collaborative dissipation processing. Based on the visual servo speed information and the impedance-corrected speed information, determine the speed deviation information; Based on the visual-touch fusion weight information, velocity deviation information, and preset fusion coordination coefficients, coordination dissipation processing is performed to obtain coordination dissipation information.

[0077] Cooperative dissipation processing is a process designed to manage and eliminate potential energy conflicts or inconsistencies that may arise when fusing different control modes. This process actively absorbs and dissipates transient energy caused by differences in control commands by introducing a compensation term, thereby ensuring the smoothness and stability of the control system during mode switching or fusion transitions. Cooperative dissipation information, the output of cooperative dissipation processing, represents the amount of energy or velocity compensation required to ensure the smooth fusion of visual servo velocity information and impedance-corrected velocity information. This cooperative dissipation information is typically represented as a velocity vector whose direction and magnitude are designed to counteract conflicting velocity components between the two control modes, thus preventing non-physical micro-oscillations in the end effector.

[0078] This application's solution effectively addresses the virtual energy and micro-oscillation issues arising from the inherent differences between visual servo control and impedance control during the fusion process by introducing a cooperative dissipation processing mechanism. This mechanism first accurately determines the speed deviation between visual servo speed information and impedance-corrected speed information. This speed deviation directly quantifies the degree of conflict between the two heterogeneous control commands, providing a clear basis for subsequent energy dissipation. Based on this, the cooperative dissipation processing utilizes visual-touch fusion weight information, the aforementioned speed deviation information, and a preset fusion cooperation coefficient, calculated through a bell-shaped activation function to obtain the cooperative dissipation information, which can be calculated according to a preset cooperative dissipation information calculation formula. Cooperative dissipation calculation is performed to obtain cooperative dissipation information. The fusion coordination coefficient determines the strength of the system's absorption of conflict energy during mode switching, while the bell-shaped activation function A(α) dynamically adjusts the activity of dissipation based on the current visual-tactile fusion weight α. This function is designed to provide the strongest dissipation effect when the contributions of visual and tactile control modes are relatively balanced (i.e., when α is at an intermediate value), and to weaken the dissipation effect when one mode completely dominates (i.e., when α is close to 0 or 1). This dynamic cooperative dissipation mechanism essentially introduces a "virtual viscous damping" into the control law, which actively absorbs and dissipates the transient energy generated by the difference between visual servo speed information and impedance-corrected speed information. By integrating the cooperative dissipation information into the final robot end-effector control speed information, the system can smooth the control signal waveform from a fundamental dynamic perspective, effectively avoiding non-physical micro-oscillations that may occur during the transition between visual and tactile control modes or in the fusion transition zone. This enables the robot end-effector to achieve a more compliant and stable dynamic response when performing grasping tasks, especially when contacting fragile targets, thereby significantly reducing the risk of damage to the target. This collaborative dissipation processing, in conjunction with the aforementioned adaptive stiffness adjustment mechanism, enhances the robot's compliant grasping ability in complex environments. Adaptive stiffness adjustment ensures that the system dynamically adjusts the weighting of vision and touch under varying current speeds and contact forces, enabling rapid stiffness unloading or maintaining fine manipulation. Furthermore, the collaborative dissipation processing, building upon this dynamic weighting adjustment, addresses the virtual energy and non-physical micro-oscillations caused by the mathematical and topological differences between the two control modes, ensuring the physical stability of the fusion process. Through this approach, the robot can not only adaptively adjust its control strategy according to the task status but also maintain smooth, shock-free movement during strategy switching, achieving significant progress in balancing the speed of approaching the target with the compliance of contacting the fruit.

[0079] In some implementations, velocity synthesis processing is performed based on cooperative dissipation information, visual servoing velocity information, impedance correction velocity information, and visual-touch fusion weight information to obtain the control velocity information of the robot end effector, including: Based on visual servo speed information and visual-touch fusion weight information, the weighted visual speed information is determined; Based on impedance-corrected velocity information, velocity difference coupling is performed by combining visual-touch fusion weight information to obtain weighted impedance velocity information; We synthesize the control velocity information of the robot end effector by using weighted visual velocity information and weighted impedance velocity information, combined with cooperative dissipation information.

[0080] This application aims to address the virtual energy and micro-oscillation problems caused by the fundamental differences in control laws between visual servo speed information and impedance-corrected speed information. To this end, this solution proposes an energy-coordinated hybrid speed control law. First, the system dynamically determines weighted visual speed information based on visual servo speed information and visual-tactile fusion weight information, allowing the contribution of visual guidance to be adjusted according to the current fusion weights. Simultaneously, based on impedance-corrected speed information and combined with weights complementary to the visual-tactile fusion weight information, speed difference coupling processing is performed to obtain weighted impedance speed information, thereby balancing the proportion of tactile control. This weighting mechanism ensures a smooth transition between visual and tactile control, avoiding hard switching. More importantly, this solution introduces cooperative dissipation information when synthesizing the final robot end-effector control speed information. This collaborative dissipation information, acting as an additional compensation term, functions like a "virtual damper." When there are significant differences between visual and tactile speed commands, it can actively absorb and dissipate the virtual energy generated by changes in the control model's topology. This eliminates potential acceleration jumps and non-physical micro-oscillations that may occur during heterogeneous data fusion from a fundamental dynamic perspective, through a pre-defined dynamic coupling equation. Based on cooperative dissipation information Visual servo speed information Impedance correction speed information and visual-touch fusion weight information Calculations are performed to obtain the control speed information of the robot's end effector. In this way, the solution not only achieves dynamic weighted fusion of visual and tactile control, but also ensures the physical rationality and stability of control commands from an energy perspective. This enables the robot end effector to exhibit high compliance and stability when approaching and contacting fragile objects, effectively avoiding damage to the grasped object.

[0081] In some implementations, it also includes: During the gripping and holding phase, the ratio of tangential force to normal force is monitored in real time. When the ratio exceeds the preset safe friction threshold, the ratio is corrected based on the safe friction threshold, combined with the slip risk index information and displacement compensation step information of the robot end effector, to obtain the position correction information. A new reference position is obtained by superimposing the position correction information with the current reference position of the robot's end effector. The normal gripping force at the robot's end effector is updated based on the new reference position.

[0082] The real-time monitoring of the ratio of tangential force to normal force aims to continuously acquire the frictional state of the contact surface between the robot's end effector and the grasped object, in order to determine whether there is a potential slippage tendency. This monitoring can be achieved by integrating multi-axis force / torque sensors into the robot's end effector or the gripper's fingertips. For example, a six-dimensional force sensor can be used to directly measure the tangential and normal forces at the contact point. After necessary filtering and coordinate transformation of the sensor data, the ratio can be calculated.

[0083] When the ratio exceeds the preset safe friction threshold, it indicates that the actual friction state has exceeded the safety boundary and there is a risk of slippage. At this time, a correction mechanism needs to be triggered. The safe friction threshold can be pre-calibrated based on the material properties of the object being grasped (such as the coefficient of friction of the fruit's skin), the grasping task requirements, and experimental data. For example, the static friction coefficient of different fruit surfaces can be measured offline and multiplied by a safety factor (such as 0.6 to 0.8, but not limited to this) as the safe friction threshold.

[0084] Based on the safety friction threshold, and combined with the slip risk index information and displacement compensation step size information of the robot end effector, the comparison value is corrected to obtain the position correction amount information. For example, it can be calculated according to a preset correction formula. Calculations are performed to obtain the position correction information. ;in, This is the position correction amount. For displacement compensation step size, The preset safety friction threshold, This is the slip risk index (obtained through pre-calibration). For tangential force, The normal force is used. The slip risk index reflects the probability or severity of slippage and is used to adjust the nonlinear amplification effect of the correction amount. The slip risk index can be a preset constant, determined through experimental optimization. For example, for easily slipping objects, the slip risk index can be set larger, making the correction amount more sensitive to changes in the friction ratio. The slip risk index can also be dynamically changed, for example, adjusted in real time based on factors such as the fragility of the grasped object, the grasping speed, or environmental vibration. The displacement compensation step size controls the magnitude of each position correction, preventing over-compensation or under-compensation. The displacement compensation step size can be a preset fixed value, determined based on the motion accuracy of the robot's end effector and the precision of the grasping task. The displacement compensation step size can also be adaptive, for example, dynamically adjusted based on the magnitude of the current grasping force, the size of the target object, or the urgency of the slippage risk. The correction processing uses a power-law relationship for nonlinear correction, so that when the ratio approaches the safety threshold, the correction amount increases exponentially, thereby achieving a sensitive response to slippage risk.

[0085] A new reference position is obtained by superimposing the position correction information with the current reference position of the robot's end effector. Upon receiving the new reference position, the robot controller can generate a corresponding trajectory through its motion planning module, driving the robot's end effector to move to the new position. Alternatively, within the control loop of the robot's end effector, the position correction amount can be added as an increment to the current position command to achieve real-time position adjustment.

[0086] Based on the new reference position, the normal gripping force of the robot's end effector is updated. If the robot's end effector is a force-controlled gripper, the new reference position can be directly used as the desired position of the gripper's fingertips, and the controller will adjust the gripping force accordingly. If the robot's end effector is a position-controlled gripper, the adjustment of the new reference position will cause the gripper's fingertips to move slightly inward, thereby increasing the squeezing force on the object, i.e., the normal gripping force.

[0087] This application's solution effectively prevents fruit slippage or damage by introducing a real-time monitoring and adaptive compensation mechanism for the ratio of tangential force to normal force during the gripping and holding phase, ensuring the stability and reliability of the gripping process. Real-time monitoring of the tangential force to normal force ratio during the gripping and holding phase allows the system to dynamically perceive slippage risks and avoid accidental drop due to force control instability. When the ratio exceeds a preset safe friction threshold, a correction mechanism is triggered, ensuring intervention only when the slippage risk is significant, preventing unnecessary adjustments from interfering with normal operation. Based on the safe friction threshold, correction processing is performed using slippage risk index information and displacement compensation step size information. The slippage risk index information reflects the slippage probability, and the displacement compensation step size information controls the adjustment range, making the correction amount adaptive to the current risk level and avoiding over- or under-compensation. The resulting position correction information ensures more significant compensation when the slippage risk is high, improving response accuracy. The position correction information is superimposed with the current reference position to obtain a new reference position, thereby adjusting the position of the robot's end effector to compensate for potential slippage and maintain the stability of the gripping posture. The normal clamping force is updated based on the new reference position, directly enhancing the clamping force to counteract the slippage force and ensure that the fruit is firmly held.

[0088] The solution presented in this application, combined with the aforementioned robot end-effector control method, further addresses the slippage problem that may occur during the grasping and holding phase while achieving compliant approach and contact. By introducing a slippage risk monitoring and adaptive compensation mechanism for the grasping and holding phase based on visual-touch fusion control, the entire grasping process (from approach and contact to holding) exhibits higher robustness and safety. This combination is particularly effective in handling easily slipped and damaged objects such as clustered fruits, significantly reducing the grasping failure rate and fruit damage rate. The aforementioned robot end-effector control method ensures that the robot end-effector can compliantly contact the target and initiate initial grasping. Furthermore, the anti-slip module in this application continuously monitors after grasping stabilizes. Once slippage is detected, it immediately increases the normal gripping force by adjusting the reference position, thereby enhancing the reliability of the grasping process without affecting the initial compliance. This phased, multimodal control strategy solves the problem of traditional methods lacking dynamic adaptability during the grasping and holding phase.

[0089] like Figure 4 As shown, this application provides a control device for a robot end effector, comprising: Acquisition module 21 is used to acquire image acquisition information and raw contact force signals at the robot end effector; The vision processing module 22 is used to perform vision processing based on image acquisition information to obtain vision servo speed information; Impedance correction module 23 is used to perform impedance correction based on the original contact force signal and combined with preset contact surface deformation compensation information to obtain impedance correction speed information. The adjustment module 24 is used to adaptively adjust the stiffness based on the current speed information and contact force change information of the robot end, combined with the preset dynamic slope modulation information, to obtain the visual-touch fusion weight information. The collaborative module 25 is used to perform collaborative dissipation processing based on the visual-touch fusion weight information, combined with visual servo speed information and impedance correction speed information, to obtain collaborative dissipation information. Synthesis module 26 is used to perform velocity synthesis processing based on cooperative dissipation information, visual servo velocity information, impedance correction velocity information and visual-touch fusion weight information to obtain the control velocity information of the robot end effector. The drive module 27 is used to drive the end of the machine according to the control speed information.

[0090] The robot end effector provided in this embodiment is used to execute the steps in the robot end effector control method provided in the first aspect above. The principle of the robot end effector control device provided in this embodiment is the same as that of the robot end effector control method provided in the first aspect above, and will not be discussed in detail here.

[0091] The core innovation of this embodiment lies in the adaptive stiffness adjustment achieved by combining dynamic slope modulation information with current velocity information and contact force change information. This enables dynamic adaptive adjustment of the visual-touch fusion weight information, allowing the system to automatically adjust the fusion rate based on impact energy. This allows for rapid stiffness unloading to avoid hard collisions during high-speed approach and maintains visual accuracy for fine adjustments during low-speed operation. Simultaneously, collaborative dissipation processing eliminates the dynamic conflict between visual servo velocity information and impedance correction velocity information, physically eliminating mode-switching oscillations and preventing damage to delicate fruits from non-physical micro-oscillations. Furthermore, nonlinear correction of biological soft tissue using contact surface deformation compensation information effectively filters out spurious force signals generated by soft tissue deformation, improving the accuracy and stability of force control. Through these technical solutions, this application effectively solves key problems in visual-touch fusion control, achieving compliant and efficient grasping of the robot end effector in unstructured environments, significantly improving fruit integrity and system robustness during harvesting operations.

[0092] like Figure 5 As shown, this application provides a robot device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112 and the memory 113 communicate with each other through the communication bus 114, and the memory 113 is used to store computer programs. In one embodiment of this application, the processor 111, when executing a program stored in a memory, implements the steps of the robot end-effector control method described in any of the first aspects.

[0093] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the robot end-effector control method provided in any of the foregoing method embodiments.

[0094] It should be noted that the system, computer room air conditioning, and storage media embodiments are basically similar to the method embodiments, so the descriptions are relatively simple. For relevant details, please refer to the descriptions in the method embodiments.

[0095] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0096] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A control method of a robot end, characterized by, include: Acquire the raw contact force signal at the robot's end effector and the image acquisition information corresponding to the object to be grasped; Visual processing is performed based on the image acquisition information to obtain visual servo speed information; Based on the original contact force signal, impedance correction is performed in combination with preset contact surface deformation compensation information to obtain impedance correction speed information. Based on the current speed information and contact force change information of the robot end effector, and combined with the preset dynamic slope modulation information, adaptive stiffness adjustment is performed to obtain the visual-touch fusion weight information. Based on the visual-touch fusion weight information, combined with the visual servo speed information and impedance correction speed information, collaborative dissipation processing is performed to obtain collaborative dissipation information. Based on the cooperative dissipation information, the visual servo speed information, the impedance correction speed information, and the visual-touch fusion weight information, speed synthesis processing is performed to obtain the control speed information of the robot end effector; Drive the robot end effector according to the control speed information; The process of performing collaborative dissipation processing based on the visual-touch fusion weight information, combined with the visual servo speed information and the impedance correction speed information, yields collaborative dissipation information, including: Based on the visual servo speed information and the impedance correction speed information, determine the speed deviation information; Based on the visual-touch fusion weight information, the speed deviation information, and the preset fusion coordination coefficient, a coordination dissipation process is performed to obtain coordination dissipation information; The step of performing velocity synthesis processing based on the cooperative dissipation information, the visual servoing velocity information, the impedance correction velocity information, and the visual-touch fusion weight information to obtain the control velocity information of the robot end effector includes: Based on the visual servo speed information and the visual-touch fusion weight information, the weighted visual speed information is determined; Based on the impedance-corrected velocity information, velocity difference coupling processing is performed in conjunction with the visual-touch fusion weight information to obtain weighted impedance velocity information. The control speed information of the robot end effector is obtained by combining the weighted visual speed information and the weighted impedance speed information with the cooperative dissipation information to perform speed synthesis.

2. The control method of a robot end according to claim 1, characterized in that, The step of performing visual processing based on the image acquisition information to obtain visual servo speed information includes: Based on the image acquisition information, the object to be captured and the corresponding feature state information of the object to be captured are determined; When the signal value of the original contact force signal is greater than a preset signal threshold, the response sensitivity of the robot end effector is dynamically adjusted according to the characteristic state information to obtain the visual servo control gain of the robot. Based on the visual servo control gain and combined with the basic gain information, the visual servo speed information is generated.

3. The control method of a robot end according to claim 1, wherein, The impedance correction based on the original contact force signal, combined with preset contact surface deformation compensation information, yields impedance correction speed information, including: Based on the preset contact surface deformation compensation information, the original contact force signal is subjected to soft tissue nonlinear compensation to obtain the contact force correction signal; Obtain the current position of the robot end effector and the desired position output by the servo controller of the robot end effector; The current acceleration of the robot end effector is determined based on the contact force correction signal, the current position and desired position of the robot end effector, and the preset impedance model corresponding to the robot end effector. Based on the current acceleration, the velocity is predicted to obtain the expected velocity at the next moment; The expected velocity at the next moment is used as the impedance-corrected velocity information.

4. The control method of a robot end according to claim 3, wherein, The step of performing soft tissue nonlinear compensation on the original contact force signal based on preset contact surface deformation compensation information to obtain a contact force correction signal includes: Obtain preset contact surface deformation compensation information, wherein the contact surface deformation compensation information includes the preset soft tissue relaxation coefficient and reference characteristic force corresponding to the robot end effector; The original contact force signal is nonlinearly compensated using the soft tissue relaxation coefficient and the reference characteristic force to obtain the corrected contact force signal.

5. The control method of a robot end according to claim 3, wherein, The adaptive stiffness adjustment, based on the current velocity information and contact force change information of the robot end effector, combined with preset dynamic slope modulation information, yields view-touch fusion weight information, including: The dynamic slope is calculated based on the current speed information and contact force change information of the robot end effector, combined with preset dynamic slope modulation information. Based on the dynamic slope, weighting is performed using the magnitude of the contact force correction signal and a preset transition threshold to obtain visual-touch fusion weight information.

6. The control method of a robot end according to claim 1, wherein, Also includes: During the gripping and holding phase, the ratio of tangential force to normal force is monitored in real time. When the ratio exceeds the preset safe friction threshold, the ratio is corrected based on the safe friction threshold and the slip risk index information and displacement compensation step information corresponding to the robot end effector to obtain position correction information. A new reference position is obtained by superimposing the position correction information with the current reference position of the robot's end effector. Based on the new reference position, the normal gripping force of the robot end effector is updated.

7. A control device for a robot end, characterized in that include: The acquisition module is used to acquire the original contact force signal at the robot's end effector and the image acquisition information corresponding to the object to be grasped; The vision processing module is used to perform vision processing based on the image acquisition information to obtain vision servo speed information; The impedance correction module is used to perform impedance correction based on the original contact force signal and combined with preset contact surface deformation compensation information to obtain impedance correction speed information. The adjustment module is used to adaptively adjust the stiffness based on the current speed information and contact force change information of the robot end effector, combined with preset dynamic slope modulation information, to obtain visual-touch fusion weight information. The collaboration module is used to perform collaborative dissipation processing based on the visual-touch fusion weight information, combined with the visual servo speed information and impedance correction speed information, to obtain collaborative dissipation information. The synthesis module is used to perform velocity synthesis processing based on the cooperative dissipation information, the visual servo velocity information, the impedance correction velocity information, and the visual-touch fusion weight information to obtain the control velocity information of the robot end effector. A drive module is used to drive the robot end effector according to the control speed information; When the collaborative module performs collaborative dissipation processing based on the visual-touch fusion weight information, combined with the visual servo speed information and the impedance correction speed information, to obtain collaborative dissipation information, the specific execution is as follows: Based on the visual servo speed information and the impedance correction speed information, determine the speed deviation information; Based on the visual-touch fusion weight information, the speed deviation information, and the preset fusion coordination coefficient, a coordination dissipation process is performed to obtain coordination dissipation information; When the synthesis module performs velocity synthesis processing based on the cooperative dissipation information, the visual servo velocity information, the impedance correction velocity information, and the visual-touch fusion weight information to obtain the control velocity information of the robot end effector, it specifically performs the following: Based on the visual servo speed information and the visual-touch fusion weight information, the weighted visual speed information is determined; Based on the impedance-corrected velocity information, velocity difference coupling processing is performed in conjunction with the visual-touch fusion weight information to obtain weighted impedance velocity information. The control speed information of the robot end effector is obtained by combining the weighted visual speed information and the weighted impedance speed information with the cooperative dissipation information to perform speed synthesis.

8. A robotic device, characterized by It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in a memory, implements the control method of the robot end effector as described in any one of claims 1-6.

Images