A human-machine collaborative carrying method and system based on visual-haptic hybrid perception

Abstract

The present application relates to the technical field of robot control and physical human-computer interaction, and particularly relates to a human-robot collaborative carrying method and system based on visual-haptic hybrid perception. The method comprises the following steps: acquiring human motion information through a visual sensor, and combining with an end force sensor to obtain interactive force information, so as to realize comprehensive perception of human motion state and human-computer interaction state; based on human motion visual measurement, robot end interactive force and object dynamics model, the control input of the human is estimated online to infer the motion intention of the human, and the desired trajectory of the robot is dynamically generated; the control effect and long-term cost are evaluated through a Critic network, and the system uncertainty and external disturbance are compensated through an Actor network, so that the control input can be adaptively optimized according to the collaborative carrying state. The active identification and coordinated response of the robot to the carrying intention of the operator are realized, the trajectory tracking accuracy and dynamic adaptability of the robot are improved, and the human-robot collaborative carrying task is smoothly, safely and efficiently completed.

Description

Technical Field

[0001] This invention relates to the field of robot control and physical human-machine interaction technology, and in particular to a human-machine collaborative handling method and system based on visual-force hybrid perception. Background Technology

[0002] Existing technologies mainly fall into the following categories: one is passive impedance control based on end-effector force sensors, where the robot achieves compliant movement based on the interactive force applied by the human; another is human motion intention recognition methods based on vision or multimodal information, which generate a robot reference trajectory by recognizing the human hand trajectory or human posture; and the third is control methods that use approximators such as RBF neural networks to compensate for the uncertainty of robot dynamics. Although the above technologies can accomplish cooperative handling tasks to a certain extent, there is still a lack of a unified method that can simultaneously achieve "online intention estimation—desired trajectory generation—uncertainty adaptive compensation" when human motion intentions change in real time, the coupled dynamics of objects are complex, and the robot has unmodeled dynamics and external disturbances.

[0003] Existing technologies have at least the following shortcomings: First, passive impedance control relying solely on end-effector force sensors typically leaves the robot in a passive following state, only responding after significant changes in the interaction force. This leads to large initial position deviations and response lags during the initial stages of collaborative handling. Second, while vision-based trajectory-based following methods can observe human hand movements, they fail to fully reflect the contact dynamics between humans, objects, and the robot, making them prone to trajectory deviations and oscillations during speed changes, direction switching, or load changes. Third, traditional model-driven control methods rely on relatively accurate robot dynamic models, but real-world systems commonly experience parameter perturbations, friction, external disturbances, and unmodeled dynamics, making it difficult for fixed-parameter controllers to maintain high accuracy and stability throughout the collaborative process. Fourth, while compensation methods such as RBF neural networks can mitigate some uncertainties, their global evaluation capabilities for long-term performance are insufficient, and significant errors and delays may still exist in the transient phase. Fifth, many existing solutions do not directly estimate human control inputs, making it difficult for the robot to proactively update the reference trajectory based on the human's true collaborative intentions. Therefore, there is still room for improvement in human-robot synchronization, smoothness, and reducing human workload. Summary of the Invention

[0004] To address the lack of a unified method in existing technologies capable of simultaneously achieving "online intent estimation—desired trajectory generation—uncertainty adaptive compensation," this invention provides a human-machine collaborative handling method and system based on a hybrid vision-force perception system. The technical solution is as follows: On the one hand, a human-machine collaborative handling method based on visual-force hybrid perception is provided, characterized in that the method includes: S1. Collect human motion information based on visual sensors; collect interactive force information during human-robot collaborative handling based on robot end effector force sensors; S2. Based on human motion information and interaction force information, establish a robot dynamics model, a transported object dynamics model, and a robot composite control input structure; S3. Establish the transformation relationship between the vision sensor and the robot coordinate system through the hand-eye calibration method; S4. Based on the dynamic model of the object being transported, establish a state-space model of the object being transported; S5. Design an observer for human control input, as well as a human intention parameter update law and robot desired trajectory, to perform online estimation of human control input; S6. Construct a long-term cost evaluation mechanism for the Critic neural network to evaluate the impact of the current control input on the future cooperative transport performance; design an uncertainty compensation mechanism for the Actor neural network, and continuously adjust the compensation amount for system uncertainty based on the evaluation results of the Critic neural network and its own state error; obtain an adaptive control law based on the Actor-Critic network. S7. Repeat S1-S6 to form a closed-loop human-machine collaborative handling control process based on visual perception, force feedback, human intention estimation, expected trajectory generation, Actor-Critic compensation, and robot control execution.

[0005] Optionally, in S2, based on human motion information and interaction force information, a robot dynamics model, a transported object dynamics model, and a robot composite control input structure are established, including: Based on human motion information and interaction force information, and combined with the dynamic coupling relationship between the robot end effector, the object being transported, and the human motion, a robot dynamics model is established as shown in the following formula:

[0006] All variables are in the robot reference coordinate system. The following indicates; Represents the robot's inertia matrix. Represents the Coriolis force and centrifugal force matrix of the robot. Represents the robot's gravity term. , , These represent the robot's end-effector position, velocity, and acceleration, respectively. This indicates the robot's control input. This represents the interaction force measured by the robot's end effector force sensor; Establish a dynamic model for the rigid object being transported, as shown in the following formula:

[0007] Among them, the position, velocity, and acceleration of the center of mass of the object being transported are respectively , , The control input applied by the human body to the object being moved is ; The inertia matrix of the object being transported. This represents the matrix of Coriolis force and centrifugal force of the object being transported. The term representing the weight of the object being moved. This refers to external control input applied by the human body. This represents the interaction force exerted by the robot's end effector on the object being transported; Robot control input It is decomposed into three parts, namely the robot dynamics compensation term. Expected trajectory tracking item and interaction force compensation item The expression for the robot's composite control input structure is obtained as follows:

[0008] in, Used to compensate for the robot's own dynamics Used to track the robot's desired trajectory generated by human intent. Used to compensate for the effects of human-computer interaction forces and changes in human movement on the system.

[0009] Optionally, in S3, the transformation relationship between the vision sensor and the robot coordinate system is established using the hand-eye calibration method, including: The human motion information collected by the visual sensor is performed in the camera coordinate system {A}, while the robot control is performed in the robot reference coordinate system {R}. Based on coordinate systems {A} and {R}, a spatial mapping relationship between the camera coordinate system and the robot reference coordinate system is established through hand-eye calibration, resulting in the following transformation relationship between the vision sensor and the robot coordinate system:

[0010] in, This represents human motion information in the robot's reference coordinate system; This represents the transformation matrix from the camera coordinate system to the robot coordinate system; This represents human motion information collected in the camera coordinate system.

[0011] Optionally, in S4, based on the dynamic model of the object being transported, a state-space model of the object being transported is established, including: Based on the dynamic model of the object being transported and the transformation relationship between the visual sensor and the robot coordinate system, the dynamic model of the object being transported is converted into a state-space form, resulting in the state-space model of the object being transported:

[0012] in, Indicates the location of the object being moved; Indicates the position of the human body during movement; This indicates the position of the robot's end effector.

[0013] Optionally, in S5, an observer for human control input is designed, along with a human intention parameter update law and the robot's desired trajectory, to perform online estimation of the human control input, including: The observer for human body control input is shown in the following formula:

[0014] in, This represents the estimated state of the object being moved. This represents the estimated value of human control input. Here is the gain matrix of the positive definite observer. This indicates the state estimation error; Based on the design, the observer estimates the human control input online by using visually measured human motion information, robot end-effector force information, and the dynamic state of the object being transported, enabling the robot to obtain the operator's implicit force application intention during collaborative transport. Based on the human intention parameter update law according to the following formula, the estimation results of human control input are... Calibrate for accuracy:

[0015]

[0016] in, It is a positive parameter. Indicates the speed of human movement. This indicates the robot's desired trajectory velocity; Obtain human control input estimates Subsequently, the robot's desired trajectory is updated in the following way:

[0017] The robot dynamically adjusts its desired trajectory based on changes in the force applied by the human body and the trend of human movement.

[0018] Optionally, in S6, a long-term cost evaluation mechanism based on a Critic neural network is constructed to evaluate the impact of the current control input on future cooperative transport performance, including: The Critic neural network approximates the long-term cost function as shown in the following formula:

[0019] in, A normal number representing the degree of impact of future costs. Represents the instantaneous cost function; Critic Network Weight Update Law As shown in the following formula:

[0020] in, The learning rate for the Critic network; express; express; The impact of current control inputs on the cooperative transport effect is evaluated using a Critic network.

[0021] Optionally, in S6, an uncertainty compensation mechanism for the Actor neural network is designed. Based on the evaluation results of the Critic neural network and its own state error, the compensation amount for system uncertainty is continuously adjusted, including: The uncertainty in the robot dynamics compensation term is approximated using an Actor neural network, and the ideal compensation term is defined as:

[0022] in, For the ideal weights of the Actor network, For Actor network basis functions, Input to the Actor network, This represents the approximation error of the neural network. The Actor network update mechanism is as follows:

[0023] The robot can continuously adjust the compensation amount for system uncertainties based on the evaluation results of the Critic network and its own state error.

[0024] Optionally, in S6, the adaptive control law based on the Actor-Critic network includes: By combining the Actor neural network compensation term with the robot's composite control input structure, the robot's adaptive control law is obtained:

[0025]

[0026]

[0027] in, This is the uncertainty compensation term output by the Actor network; This is the position error feedback term; For speed error feedback; Used to enhance control robustness; This is the end-effector interaction force compensation term; This is a visual-mechanical fusion compensation term formed by combining the estimation results of human movement speed and human control input; Calculate the robot's end-effector control input based on the robot's adaptive control law, and then:

[0028] By obtaining the control torques of each joint of the robot, the robot can perform cooperative transport actions.

[0029] On the other hand, a human-machine collaborative handling system based on vision-force hybrid perception is provided. This system is applied to a human-machine collaborative handling method based on vision-force hybrid perception. The system includes: a robot body, a vision sensor, a robot end force sensor, a controller, a rigid object to be handled, and a human-machine collaborative handling control software platform. Sensors are installed in each joint of the robot body. The sensors in each joint are used to acquire joint angles, joint angular velocities and joint torques in real time, and further obtain the position, velocity and acceleration of the robot end effector. Visual sensors are used to collect real-time motion information of key points on the operator's hand or body; Robot end effector force sensors are used to collect the interaction forces between the robot end effector and the object being transported during human-robot collaborative handling processes; The controller receives visual sensor, robot end-effector force sensor and robot state feedback information, and performs visual coordinate transformation, human control input estimation, robot desired trajectory generation, Actor-Critic network adaptive compensation and robot control input calculation; it outputs robot end-effector control input and maps it to joint control torque through robot Jacobian matrix transpose. The above system structure forms a closed-loop control framework of "visual perception - force feedback - human intention estimation - expected trajectory generation - neural network compensation - robot execution".

[0030] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, the at least one instruction being loaded and executed by a processor to implement any of the above-described human-machine collaborative handling methods based on visual-force hybrid perception.

[0031] The beneficial effects of the technical solutions provided by the embodiments of the present invention include at least the following: (1) Existing human-robot collaborative handling methods mostly rely on single force feedback or passive impedance control. The robot mainly follows the changes in the force at the end effector and lacks active perception of human movement trends and operational intentions, which easily leads to problems such as motion lag, human-robot asynchrony, and unnatural collaborative processes. This invention proposes a visual-force hybrid perception method, which collects human motion information through a visual sensor and combines it with an end effector force sensor to obtain interactive force information, thereby achieving comprehensive perception of human motion state and human-robot interaction state. This enables the robot to generate motion trajectories that better meet the operator's expectations, improving the synchronization, stability, and comfort of collaborative handling.

[0032] 2) Existing methods struggle to directly obtain the operator's actual control input and motion intentions, especially when there is dynamic coupling between the human, robot, and the object being transported. Relying solely on end-effector forces or positional changes makes it difficult to accurately determine the operator's desired movement, leading to the robot passively following and resulting in low collaborative efficiency. This invention proposes an observer design method for human-robot collaborative transport. Based on human motion visual measurement, robot end-effector interaction forces, and the dynamic model of the object being transported, it estimates the human's control input online and further infers the operator's motion intentions, thereby dynamically generating the robot's desired trajectory. This enables the robot to actively recognize and coordinate its response to the operator's transport intentions.

[0033] 3) Existing passive impedance control or traditional neural network compensation methods are insufficient in adapting to robot dynamic uncertainties, unmodeled disturbances, and rapid interactive changes, easily leading to problems such as large trajectory tracking errors, response lag, and motion oscillations. This invention proposes an adaptive control method based on an Actor-Critic network. The Critic network is used to evaluate the control effect and long-term cost, while the Actor network is used to compensate for system uncertainties and external disturbances. This allows the control input to adaptively optimize according to the collaborative transport state, thereby improving the robot's trajectory tracking accuracy, robustness, and dynamic adaptability, ensuring the smooth, safe, and efficient completion of human-robot collaborative transport tasks. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1This is a flowchart of a human-machine collaborative handling method based on visual-force hybrid perception provided by an embodiment of the present invention. Figure 2 These are hand pose, actual pose and desired pose diagram of the robotic arm end effector provided in the embodiments of the present invention; Figure 3 This is a pose diagram with unsuitable parameters provided in the embodiments of the present invention; Figure 4 This is a human hand control input diagram provided in an embodiment of the present invention; Figure 5 This is a diagram of collaborative transport trajectory tracking (trajectory 1) using the technical solution provided in this embodiment of the invention; Figure 6 This is a diagram of collaborative transport trajectory tracking (trajectory 2) using the technical solution provided in this embodiment of the invention; Figure 7 This is a collaborative transport trajectory tracking diagram using a PI control scheme provided in an embodiment of the present invention; Figure 8 This is a cooperative transport trajectory tracking diagram using the RBFNN control scheme provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of a human-machine collaborative handling system based on a visual-force hybrid perception according to an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0036] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0037] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0038] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0039] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0040] This invention provides a human-machine collaborative handling method based on visual-force hybrid perception. This method can be implemented by a human-machine collaborative handling device based on visual-force hybrid perception, which can be a terminal or a server. Figure 1 The flowchart shown is a human-machine collaborative handling method based on visual-force hybrid perception. The processing flow of this method may include the following steps: S1. Collect human motion information based on visual sensors; collect interactive force information of human-machine collaborative handling process based on robot end effector force sensors.

[0041] In one feasible implementation, the present invention first establishes a human-machine collaborative handling system based on vision-force hybrid perception. This system includes a robot body, a vision sensor, a robot end effector force sensor, a controller, a rigid object to be handled, and a human-machine collaborative handling control software platform. The vision sensor is used to collect real-time motion information of the operator's hand or key points on the human body, denoted as... The robot end effector force sensor is used to collect the interaction force between the robot's end effector and the object being transported during human-robot collaborative handling, denoted as ; The sensors at each joint of the robot body acquire joint angles, joint angular velocities, and joint torques in real time, and further obtain the robot's end-effector position, velocity, and acceleration, denoted as follows: , , .

[0042] The controller receives information from vision sensors, force sensors, and robot state feedback, and performs visual coordinate transformation, human control input estimation, robot desired trajectory generation, Actor-Critic network adaptive compensation, and robot control input calculation. The controller ultimately outputs the robot's end-effector control input. And it is mapped to joint control torque through the transpose of the robot's Jacobian matrix:

[0043] in, This represents the control torque of the robot's joints. Represents the Jacobian matrix of the robot. This represents the transpose of the Jacobian matrix. This represents the control input for the robot's end effector. Through the above system structure, a closed-loop control framework can be formed, consisting of "visual perception—force feedback—human intention estimation—desired trajectory generation—neural network compensation—robot execution."

[0044] S2. Based on human motion information and interaction force information, establish a robot dynamics model, a transported object dynamics model, and a robot composite control input structure; In one feasible implementation, based on human motion information and interaction force information, and combined with the dynamic coupling relationship between the robot end effector, the object being transported, and the human motion, a robot dynamics model is established as follows to describe the robot's motion state during the transport process:

[0045] All variables are in the robot reference coordinate system. The following indicates; Represents the robot's inertia matrix. Represents the Coriolis force and centrifugal force matrix of the robot. Represents the robot's gravity term. , , These represent the robot's end-effector position, velocity, and acceleration, respectively. This indicates the robot's control input. This represents the interaction force measured by the robot's end effector force sensor; this model is used to describe the robot's own dynamic characteristics and to provide a dynamic compensation basis for subsequent control input design.

[0046] To further describe the interaction between the human, the robot, and the object being transported, a dynamic model of the rigid object being transported is established as follows:

[0047] Among them, the position, velocity, and acceleration of the center of mass of the object being transported are respectively , , The control input applied by the human body to the object being moved is ; The inertia matrix of the object being transported. This represents the matrix of Coriolis force and centrifugal force of the object being transported. The term representing the weight of the object being moved. This refers to external control input applied by the human body. This model represents the interaction force exerted by the robot's end effector on the object being transported. It can describe the motion changes of the object being transported under the combined action of human force and robot end effector force, providing a basis for the design of human control input observers.

[0048] This invention will integrate robot control input. It is decomposed into three parts, namely the robot dynamics compensation term. Expected trajectory tracking item and interaction force compensation item The expression for the robot's composite control input structure is obtained as follows:

[0049] in, Used to compensate for the robot's own dynamics Used to track the robot's desired trajectory generated by human intent. Used to compensate for the effects of human-computer interaction forces and changes in human movement on the system.

[0050] The specific design of each part is as follows:

[0051]

[0052]

[0053]

[0054] in, , , These represent the position, velocity, and acceleration of the robot's desired trajectory, respectively. This represents the proportional gain matrix, which corresponds to the stiffness parameter in impedance control. This represents the differential gain matrix, which corresponds to the damping parameter in impedance control. It is a positive control parameter; It is a symbolic function; express Moore-Penrose generalized inverse; This represents the human control input estimated by the observer. This composite control structure unifies robot dynamics compensation, impedance tracking control, and human interaction intent compensation into a single control input, thereby enhancing the robot's compliance, stability, and active following ability during collaborative handling.

[0055] S3. Establish the transformation relationship between the vision sensor and the robot coordinate system through the hand-eye calibration method; In one feasible implementation, the human motion information collected by the visual sensor is performed in the camera coordinate system {A}, while the robot control is performed in the robot reference coordinate system {R}. like Figure 2 The diagram shows the hand pose, the actual pose of the robotic arm's end effector, and the desired pose, according to an embodiment of the present invention. Figure 3 For pose graphs with unsuitable parameters; based on coordinate systems {A} and {R}, a spatial mapping relationship between the camera coordinate system and the robot reference coordinate system is established through hand-eye calibration. This represents the pose transformation matrix of the camera coordinate system relative to the robot reference coordinate system. This represents the pose transformation matrix of the robot's end-effector coordinate system relative to the robot's reference coordinate system. This represents the pose transformation matrix relative to the robot's end effector coordinate system. This represents the pose transformation matrix of the camera coordinate system relative to the calibration board coordinate system. Let represent the pose transformation matrix of the calibration board coordinate system relative to the camera coordinate system, then we have:

[0056] From the above formula, we can obtain:

[0057] because To maintain the robot's position under different robot postures, select two different robot postures. and We can obtain:

[0058] Further analysis revealed:

[0059] make:

[0060]

[0061]

[0062] The hand-eye alignment problem can then be transformed into:

[0063] The transformation matrix from the camera coordinate system to the robot coordinate system is obtained by solving the above equations. Subsequently, the human motion information collected in the camera coordinate system was... Converting human motion information into robot reference coordinates The transformation relationship between the vision sensor and the robot coordinate system is obtained by the following formula:

[0064] This step ensures that the human motion state obtained from visual measurements can be calculated in the same coordinate system as the robot dynamics model, the dynamics model of the object being transported, and the robot controller.

[0065] S4. Based on the dynamic model of the object being transported, establish a state-space model of the object being transported; In one feasible implementation, in order to design a human control input observer, this invention converts the dynamic model of the object being transported into a state-space form based on the dynamic model of the object being transported and the transformation relationship between the visual sensor and the robot coordinate system, and defines the state variables of the object being transported:

[0066] The equation of state for the object being transported is then expressed as:

[0067] in:

[0068]

[0069] in, Represents the zero matrix. Represents the identity matrix. Position of the object being transported. Based on the position of human movement With the robot's end position The kinematic relationships were obtained as follows:

[0070] This state-space model incorporates human control input. Robot Interaction Linking this to changes in the state of the object being moved provides a basis for online estimation of human control inputs; such as Figure 4 The diagram shown is a manual input control diagram.

[0071] S5. Design an observer for human control input, as well as a human intention parameter update law and robot desired trajectory, to perform online estimation of human control input; In one feasible implementation, since the actual human control input fγ, the human motion intention xe, and the human equivalent stiffness Kp are difficult to measure directly, the present invention designs an observer for the human control input as shown in the following formula:

[0072] in, This represents the estimated state of the object being moved. This represents the estimated value of human control input. Here is the gain matrix of the positive definite observer. The state estimation error is represented as: Human control input is represented as:

[0073] Expanding, we get:

[0074] make:

[0075] Human body control input can then be represented as:

[0076] because and Since all parameters are unknown to the robot, this invention employs estimated parameters. and Indicates the estimated value of human control input:

[0077] Define the parameter estimation error:

[0078]

[0079] The estimation error of human control input is:

[0080] The observation error system can be obtained from the state equation and the observer equation:

[0081] It can also be written as:

[0082] in:

[0083] Based on the design, the observer estimates the human control input online by using visually measured human motion information, robot end-effector force information, and the dynamic state of the object being transported, enabling the robot to obtain the operator's implicit force application intention during collaborative transport. To enable the observer to continuously update human intention-related parameters, this invention designs a human intention parameter update law based on the following formula to estimate the human control input results. Calibrate for accuracy:

[0084]

[0085] in, It is a positive parameter. Indicates the speed of human movement. This indicates the robot's desired trajectory velocity; The above update law can be used to correct the estimated value of human body equivalent stiffness online. Estimates of parameters related to intent This improves the estimation results of human control input. The accuracy.

[0086] When the estimated human control input is obtained Subsequently, the robot's desired trajectory is updated in the following way:

[0087] To be consistent with the symbols used in control inputs, it can also be written as:

[0088] Using this trajectory generation method, the robot can dynamically adjust its desired trajectory based on changes in the force applied by the human body and the trend of human movement. When the robot's desired trajectory... With human movement intention When kept in line, the robot can form a motion coupling relationship with the operator, enabling active following and expected collaboration in collaborative handling.

[0089] S6. Construct a long-term cost evaluation mechanism for the Critic neural network to evaluate the impact of the current control input on the future cooperative transport performance; design an uncertainty compensation mechanism for the Actor neural network, and continuously adjust the compensation amount for system uncertainty based on the evaluation results of the Critic neural network and its own state error; obtain an adaptive control law based on the Actor-Critic network. In one feasible implementation, in order to evaluate the impact of the current control input on the cooperative transport effect, the present invention designs a Critic neural network to approximate the long-term cost function as shown in the following formula:

[0090] in, A normal number representing the degree of impact of future costs. Represents the instantaneous cost function; The instantaneous cost function is designed as follows:

[0091] in, Indicates the trajectory error or cooperative error status. It is a positive definite weight matrix. To control the input weight matrix, This serves as the robot's control input. The cost function considers both trajectory tracking error and control input magnitude, enabling the controller to improve tracking accuracy while avoiding excessive control force.

[0092] The Critic neural network approximates the long-term cost function, and the ideal long-term cost function is defined as:

[0093] Its estimated value is:

[0094] in, For the ideal weights of the Critic network, Estimate the weights for the Critic network. These are the basis functions for the Critic network. Input for the Critic network, This represents the approximation error. A suitable value is:

[0095] Define the Critic network approximation error as:

[0096] when When the above formula is simplified, it can be represented as:

[0097] Further written as:

[0098] in, Indicates to gradient calculation, This represents the rate of change of the input to the Critic network.

[0099] Define the Critic network error function:

[0100] The Critic network weight update law is designed as follows:

[0101] in, Let be the learning rate of the Critic network. Substituting the error function, we get:

[0102] To elaborate further:

[0103] Right now:

[0104] Finally, the Critic network weight update law is obtained. As shown in the following formula:

[0105] in, The learning rate for the Critic network; express; express; in:

[0106] Through this Critic network, the controller can evaluate the impact of the current control input on future cooperative transport performance in real time, providing an evaluation basis for the Actor network's adaptive compensation.

[0107] In one feasible implementation, considering the dynamic uncertainties, unmodeled dynamics, and external disturbances in the robot system, relying solely on traditional dynamic compensation terms is insufficient to guarantee tracking accuracy in complex cooperative transport processes. Therefore, this invention employs an Actor neural network to approximate the uncertainties in the robot's dynamic compensation terms, defining the ideal compensation term as:

[0108] in, For the ideal weights of the Actor network, For Actor network basis functions, Input to the Actor network, This represents the approximation error of the neural network. And satisfy:

[0109] In actual control, the output of the Actor network is represented as:

[0110] in:

[0111] The Actor network input is defined as:

[0112] in, and These are the robot's end-effector position and velocity, respectively. For the robot's desired trajectory velocity, Let the acceleration be the desired trajectory acceleration for the robot. The general estimation error of the Actor network is defined as:

[0113] The Actor network error design is as follows:

[0114] in, It is a positive parameter matrix or a positive parameter vector. This represents the long-term cost estimate output by the Critic network. Let be the desired long-run cost function. To minimize the long-run cost, we take:

[0115] therefore:

[0116] The error function of the Actor network is:

[0117] The Actor network weight update law is:

[0118] in, This represents the learning rate of the Actor network. Further elaborated as follows:

[0119] We can obtain:

[0120] because Since it is not directly available, it is replaced with the current output of the Actor network, resulting in the Actor network update mechanism as follows:

[0121] Through the aforementioned Actor network update mechanism, the robot can continuously adjust the compensation amount for system uncertainties based on the Critic network evaluation results and its own state errors, thereby improving the trajectory tracking accuracy and robustness in the human-robot collaborative handling process.

[0122] In one feasible implementation, the Actor neural network compensation term is combined with the robot's composite control input structure to obtain the robot's adaptive control law:

[0123]

[0124]

[0125] in, This is the uncertainty compensation term output by the Actor network; This is the position error feedback term; For speed error feedback; Used to enhance control robustness; This is the end-effector interaction force compensation term; This is a visual-mechanical fusion compensation term formed by combining the estimation results of human movement speed and human control input; Substituting the above control law into the robot dynamics model, we can obtain the closed-loop error dynamics:

[0126]

[0127]

[0128] in, The tracking error between the actual trajectory and the desired trajectory of the robot's end effector can be defined as:

[0129] or:

[0130] When the expected trajectory Set as human motion intention At that time, that is:

[0131] Robots can establish a motion coupling relationship with operators by estimating human intentions, thereby achieving predictive coordination and compliant control during the handling process.

[0132] S7. Repeat S1-S6 to form a closed-loop human-machine collaborative handling control process based on visual perception, force feedback, human intention estimation, expected trajectory generation, Actor-Critic compensation, and robot control execution.

[0133] In one feasible implementation, during the execution of the human-machine collaborative handling task, the controller cyclically executes the following process according to a fixed sampling period. First, it collects human motion information obtained from the visual sensor. , And through the coordinate transformation formula:

[0134] Human motion information is converted to the robot's reference coordinate system; then, the interaction force is acquired from the robot's end effector force sensor. And obtain the robot's end state. , , Next, according to:

[0135] Calculate the position of the center of mass of the object being transported, and use the state-space model:

[0136] And the observer:

[0137] Estimate human control input Then utilize:

[0138] Generate the robot's desired trajectory and calculate the long-term cost estimate using the Critic network:

[0139] Then, the Actor network bases its decisions on:

[0140] Update the network weights to obtain the adaptive compensation term:

[0141] Ultimately, according to the complete control law:

[0142]

[0143]

[0144] Calculate the robot end-effector control input, and through:

[0145] The control torques of each joint of the robot are obtained, enabling the robot to perform cooperative transport actions. This closed-loop process is repeated continuously, forming a closed-loop human-robot cooperative transport control process of "visual perception - force feedback - human intention estimation - expected trajectory generation - Actor-Critic compensation - robot control execution".

[0146] In this embodiment of the invention, experimental results show that the method can accurately estimate human control input based on visual and force information, generate the robot's desired trajectory that conforms to the human's movement intention, and compensate for the uncertainty of the robot system through an Actor-Critic network. This enables the robot to maintain high trajectory tracking accuracy, motion stability, and cooperative compliance under different desired trajectories, different human movement speeds, and different interaction states. Compared with passive impedance control and RBF neural network compensation methods, this invention can effectively reduce trajectory deviation, response lag, and motion oscillation, improving the safety, comfort, and efficiency of human-robot collaborative handling. Figure 5This is a diagram of collaborative transport trajectory tracking (trajectory 1) using the technical solution provided in this embodiment of the invention; Figure 6 This is a diagram of collaborative transport trajectory tracking (trajectory 2) using the technical solution provided in this embodiment of the invention; Figure 7 This is a collaborative transport trajectory tracking diagram using a PI control scheme provided in an embodiment of the present invention; Figure 8 This is a collaborative transport trajectory tracking diagram using the RBFNN control scheme provided in an embodiment of the present invention.

[0147] This invention proposes a human-machine collaborative handling control strategy based on a hybrid vision-force perception system. This strategy uses a visual sensor to collect real-time human hand movement information and combines it with a robot end effector force sensor to acquire human-machine interaction force information, achieving synchronous perception of the human's movement state and the interactive force state. This strategy overcomes the limitations of traditional methods that rely solely on force feedback and where the robot passively follows the human. It enables the robot to simultaneously utilize the human's movement trends and changes in interactive forces to generate collaborative handling control inputs, improving human-machine motion synchronization and handling stability.

[0148] A method for estimating human intent based on visual coordinate transformation and an observer is proposed. This method transforms human motion information from the camera coordinate system to the robot's reference coordinate system through hand-eye calibration, and establishes an observer based on the dynamic model of the object being transported to estimate the human's control input online. This method can infer the operator's desired motion direction and target trajectory even when human stiffness parameters and motion intent are unknown, enabling the robot to shift from passive following to active collaboration.

[0149] A composite control input design method for human-robot collaborative handling is proposed. This method decomposes the robot control input into a dynamic compensation term, a desired trajectory tracking term, and an interaction force compensation term. Proportional stiffness, damping feedback, and human control input estimation results are introduced to achieve stable tracking of the desired human trajectory by the robot's end effector. This control structure effectively handles the dynamic coupling relationship between the human, the robot, and the object being handled, ensuring compliance, continuity, and safety during the handling process.

[0150] An adaptive compensation method based on an Actor-Critic network is proposed. The Critic network evaluates the long-term cost corresponding to the control input, while the Actor network performs online compensation for robot dynamic uncertainties, unmodeled dynamics, and external disturbances. This enables the controller to continuously optimize the control input based on trajectory errors and interaction states. Compared with passive impedance control and traditional RBF neural network compensation methods, this method improves trajectory tracking accuracy, robustness, and dynamic adaptability, and reduces oscillations and hysteresis during rapid collaborative transport.

[0151] Figure 9This is a block diagram illustrating a human-machine collaborative handling system based on a visual-force hybrid perception according to an exemplary embodiment. This system is used in a human-machine collaborative handling method based on visual-force hybrid perception. (Refer to...) Figure 9 The system includes: a robot body, a vision sensor, a robot end effector force sensor, a controller, a rigid object to be transported, and a human-robot collaborative transport control software platform; Sensors are installed in each joint of the robot body. The sensors in each joint are used to acquire joint angles, joint angular velocities and joint torques in real time, and further obtain the position, velocity and acceleration of the robot end effector. Visual sensors are used to collect real-time motion information of key points on the operator's hand or body; Robot end effector force sensors are used to collect the interaction forces between the robot end effector and the object being transported during human-robot collaborative handling processes; The controller receives visual sensor, robot end-effector force sensor and robot state feedback information, and performs visual coordinate transformation, human control input estimation, robot desired trajectory generation, Actor-Critic network adaptive compensation and robot control input calculation; it outputs robot end-effector control input and maps it to joint control torque through robot Jacobian matrix transpose. The above system structure forms a closed-loop control framework of "visual perception - force feedback - human intention estimation - expected trajectory generation - neural network compensation - robot execution".

[0152] In this embodiment of the invention, visual perception and end-effector force feedback are fused, and human control input is estimated online through an observer. This enables the robot to recognize the human's collaborative carrying intention without the need to install a dedicated force measurement device on the human side, thus reducing the complexity of system deployment.

[0153] By utilizing a desired trajectory update mechanism based on human control input estimation, the robot can transform from a passive responder into an intelligent follower with active collaborative capabilities, thereby improving the synchronization between the robot and the human and reducing positional deviation and response lag in the initial stage.

[0154] By employing an Actor-Critic network to compensate for robot dynamic uncertainties, unmodeled dynamics, and external disturbances online, compared to methods that only use fixed-parameter impedance control or RBF neural network compensation, the trajectory tracking accuracy, transient response performance, and overall stability can be further improved.

[0155] In collaborative handling, the robot's movement can be more smoothly coordinated with the human's operational intentions, reducing the physical burden on the human and improving the comfort of collaboration and the efficiency of task completion.

[0156] It exhibits good adaptability and robustness to different desired handling trajectories, speed variations, and interaction modes, and can be widely applied to various human-machine collaborative operation scenarios such as manufacturing assembly, rehabilitation assistance, warehouse handling, and service robots.

[0157] Figure 10 This is a schematic diagram of the structure of a human-machine collaborative handling device based on a visual-force hybrid perception according to an embodiment of the present invention, as shown below. Figure 10 As shown, a human-machine collaborative handling device based on vision-force hybrid perception may include the above-mentioned... Figure 9 The diagram shows a human-machine collaborative handling system based on a visual-force hybrid perception system.

[0158] Optionally, a human-machine collaborative handling device 410 based on visual-force hybrid perception may include a first processor 2001.

[0159] Optionally, a human-machine collaborative handling device 410 based on vision-force hybrid perception may further include a memory 2002 and a transceiver 2003.

[0160] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0161] The following is combined Figure 10 A detailed description of each component of a human-machine collaborative handling device 410 based on vision-force hybrid perception is provided below: The first processor 2001 is a control center of a human-machine collaborative handling device 410 based on vision-force hybrid perception. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0162] Optionally, the first processor 2001 can perform various functions of a human-machine collaborative handling device 410 based on vision-force hybrid perception by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0163] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 10 CPU0 and CPU1 are shown in the diagram.

[0164] In a specific implementation, as one example, a human-machine collaborative handling device 410 based on vision-force hybrid perception may also include multiple processors, for example... Figure 10 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0165] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0166] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently, and may be connected via an interface circuit of a human-machine collaborative handling device 410 based on a vision-force hybrid perception system. Figure 10 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0167] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0168] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 10(Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0169] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected via an interface circuit of a human-machine collaborative handling device 410 based on vision-force hybrid perception. Figure 10 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0170] It should be noted that, Figure 10 The structure of a human-machine collaborative handling device 410 based on visual-force hybrid perception shown in the figure does not constitute a limitation on the router. Actual knowledge structure recognition devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0171] Furthermore, the technical effects of a human-machine collaborative handling device 410 based on visual-force hybrid perception can be referred to the technical effects of a human-machine collaborative handling method based on visual-force hybrid perception described in the above method embodiments, and will not be repeated here.

[0172] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0173] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0174] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensors. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0175] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0176] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0177] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0178] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0179] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0180] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0181] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A human-machine collaborative handling method based on visual-force hybrid perception, characterized in that, Includes the following steps: S1. Collect human motion information based on visual sensors; Based on the force sensor at the end of the robot to collect interactive force information during the human-robot collaborative handling process; S2. Based on human motion information and interaction force information, establish a robot dynamics model, a transported object dynamics model, and a robot composite control input structure; S3. Establish the transformation relationship between the vision sensor and the robot coordinate system through the hand-eye calibration method; S4. Based on the dynamic model of the object being transported, establish a state-space model of the object being transported; S5. Design an observer for human control input, as well as a human intention parameter update law and robot desired trajectory, to perform online estimation of human control input; S6. Construct a long-term cost evaluation mechanism for the Critic neural network to evaluate the impact of the current control input on the future cooperative transport performance; design an uncertainty compensation mechanism for the Actor neural network, and continuously adjust the compensation amount for system uncertainty based on the evaluation results of the Critic neural network and its own state error; obtain an adaptive control law based on the Actor-Critic network. S7. Repeat S1-S6 to form a closed-loop human-machine collaborative handling control process based on visual perception, force feedback, human intention estimation, expected trajectory generation, Actor-Critic compensation, and robot control execution.

2. The method according to claim 1, characterized in that, In step S2, based on human motion information and interaction force information, a robot dynamics model, a transported object dynamics model, and a robot composite control input structure are established, including: Based on human motion information and interaction force information, and combined with the dynamic coupling relationship between the robot end effector, the object being transported, and the human motion, a robot dynamics model is established as shown in the following formula: All variables are in the robot reference coordinate system. The following indicates; Represents the robot's inertia matrix. Represents the Coriolis force and centrifugal force matrix of the robot. Represents the robot's gravity term. , , These represent the robot's end-effector position, velocity, and acceleration, respectively. This indicates the robot's control input. This represents the interaction force measured by the robot's end effector force sensor; Establish a dynamic model for the rigid object being transported, as shown in the following formula: Among them, the position, velocity, and acceleration of the center of mass of the object being transported are respectively , , The control input applied by the human body to the object being moved is ; The inertia matrix of the object being transported. This represents the matrix of Coriolis force and centrifugal force of the object being transported. The term representing the weight of the object being moved. This refers to external control input applied by the human body. This represents the interaction force exerted by the robot's end effector on the object being transported; Robot control input It is decomposed into three parts, namely the robot dynamics compensation term. Expected trajectory tracking item and interaction force compensation item The expression for the robot's composite control input structure is obtained as follows: in, Used to compensate for the robot's own dynamics Used to track the robot's desired trajectory generated by human intent. Used to compensate for the effects of human-computer interaction forces and changes in human movement on the system.

3. The method according to claim 1, characterized in that, In step S3, the transformation relationship between the vision sensor and the robot coordinate system is established using the hand-eye calibration method, including: The human motion information collected by the visual sensor is performed in the camera coordinate system {A}, while the robot control is performed in the robot reference coordinate system {R}. Based on coordinate systems {A} and {R}, a spatial mapping relationship between the camera coordinate system and the robot reference coordinate system is established through hand-eye calibration, resulting in the following transformation relationship between the vision sensor and the robot coordinate system: in, This represents human motion information in the robot's reference coordinate system; This represents the transformation matrix from the camera coordinate system to the robot coordinate system; This represents human motion information collected in the camera coordinate system.

4. The method according to claim 1, characterized in that, In step S4, a state-space model of the object being transported is established based on the dynamic model of the object being transported, including: Based on the dynamic model of the object being transported and the transformation relationship between the visual sensor and the robot coordinate system, the dynamic model of the object being transported is converted into a state-space form, resulting in the state-space model of the object being transported: in, Indicates the location of the object being moved; Indicates the position of the human body during movement; This indicates the position of the robot's end effector.

5. The method according to claim 4, characterized in that, In step S5, an observer for human control input is designed, along with a human intention parameter update law and the robot's desired trajectory, to perform online estimation of the human control input, including: The observer for human body control input is shown in the following formula: in, This represents the estimated state of the object being moved. This represents the estimated value of human control input. Here is the gain matrix of the positive definite observer. This indicates the state estimation error; Based on the design, the observer estimates the human control input online by using visually measured human motion information, robot end-effector force information, and the dynamic state of the object being transported, enabling the robot to obtain the operator's implicit force application intention during collaborative transport. Based on the human intention parameter update law according to the following formula, the estimation results of human control input are... Calibrate for accuracy: in, It is a positive parameter. Indicates the speed of human movement. This indicates the robot's desired trajectory velocity; Obtain human control input estimates Subsequently, the robot's desired trajectory is updated in the following way: The robot dynamically adjusts its desired trajectory based on changes in the force applied by the human body and the trend of human movement.

6. The method according to claim 1, characterized in that, In step S6, a long-term cost evaluation mechanism based on a Critic neural network is constructed to evaluate the impact of the current control input on future cooperative transport performance, including: The Critic neural network approximates the long-term cost function as shown in the following formula: in, A normal number representing the degree of impact of future costs. Represents the instantaneous cost function; Critic Network Weight Update Law As shown in the following formula: in, The learning rate for the Critic network; express; express; The impact of current control inputs on the cooperative transport effect is evaluated using a Critic network.

7. The method according to claim 6, characterized in that, In step S6, an uncertainty compensation mechanism for the Actor neural network is designed. Based on the evaluation results of the Critic neural network and its own state error, the compensation amount for system uncertainty is continuously adjusted, including: The uncertainty in the robot dynamics compensation term is approximated using an Actor neural network, and the ideal compensation term is defined as: in, For the ideal weights of the Actor network, For Actor network basis functions, Input to the Actor network, This represents the approximation error of the neural network. The Actor network update mechanism is as follows: The robot can continuously adjust the compensation amount for system uncertainties based on the evaluation results of the Critic network and its own state error.

8. The method according to claim 7, characterized in that, In step S6, the adaptive control law based on the Actor-Critic network includes: By combining the Actor neural network compensation term with the robot's composite control input structure, the robot's adaptive control law is obtained: in, This is the uncertainty compensation term output by the Actor network; This is the position error feedback term; For speed error feedback; Used to enhance control robustness; This is the end-effector interaction force compensation term; This is a visual-mechanical fusion compensation term formed by combining the estimation results of human movement speed and human control input; Calculate the robot's end-effector control input based on the robot's adaptive control law, and then: By obtaining the control torques of each joint of the robot, the robot can perform cooperative transport actions.

9. A human-machine collaborative handling system based on visual-force hybrid perception, characterized in that, The system is used to implement the method according to any one of claims 1-8, and the system includes: a robot body, a vision sensor, a robot end effector force sensor, a controller, a rigid object to be transported, and a human-robot collaborative transport control software platform; Sensors are installed in each joint of the robot body. The sensors in each joint are used to acquire joint angles, joint angular velocities and joint torques in real time, and further obtain the position, velocity and acceleration of the robot end effector. Visual sensors are used to collect real-time motion information of key points on the operator's hand or body; Robot end effector force sensors are used to collect the interaction forces between the robot end effector and the object being transported during human-robot collaborative handling processes; The controller receives visual sensor, robot end-effector force sensor and robot state feedback information, and performs visual coordinate transformation, human control input estimation, robot desired trajectory generation, Actor-Critic network adaptive compensation and robot control input calculation; it outputs robot end-effector control input and maps it to joint control torque through robot Jacobian matrix transpose. The above system structure forms a closed-loop control framework of "visual perception - force feedback - human intention estimation - expected trajectory generation - neural network compensation - robot execution".

10. A computer device, characterized in that, include: At least one processor and memory; The memory and processor are connected via a bus; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, a human-machine collaborative handling method based on visual-force hybrid perception as described in any one of claims 1 to 8 is implemented.

Images