Man-machine collaborative assembly method and system based on augmented reality and force feedback

Abstract

The invention discloses a man-machine collaborative assembly method and system based on augmented reality and force feedback. On one hand, more intuitive error perception is realized through superposition of an assembly path and key operation prompts by the AR equipment; on the other hand, an imitation learning strategy is improved through an assembly scene, an error compensation mapping relation is established in combination with touch feedback, human expert assembly actions are simulated to achieve accurate compensation, a quantitative evaluation mechanism is established according to the assembly scene, and the effectiveness of a compensation result is comprehensively evaluated; and the constructed system is supported to achieve a closed-loop optimization effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of human-computer collaborative assembly, in particular to a human-computer collaborative assembly method and system based on augmented reality and force feedback. BACKGROUND

[0002] Current narrow space assembly error perception and compensation methods mainly involve industrial robot automation assembly, human-computer collaborative assembly, geometric modeling and linear deviation correction technical means.

[0003] The existing industrial robot automation assembly technology mainly relies on industrial robot mechanical arms to complete the assembly task in a relatively open workspace, and cooperates with static clamps and path planning to complete part positioning, movement and connection. When facing narrow space assembly error perception, there are problems such as limited mechanical arm action, inability to accurately reach the target position, and automatic assembly strategy unable to adjust in real time.

[0004] Human-computer collaborative assembly is to introduce operators into the assembly link, and the operator controls the robot or shares the control with the robot, so as to realize task flexibility and improve the adaptability to complex environment, but there are problems such as lack of error intuitive perception, information fragmentation, strong individual dependence, etc., which are embodied in that the operator is difficult to perceive "whether the current assembly is accurate", "where the error is" and "how to adjust the next action", the perception result (visual system) and the manual action (force control) are not fused, the assembly success is highly dependent on the operator's experience, there is lack of unified strategy model, and it cannot be promoted and reused.

[0005] The existing error compensation method is generally used for assembly error prediction and adjustment, but the existing error compensation mostly adopts the geometric modeling and linear deviation correction method, such as using a pose correction model and static error offset correction, which has problems such as too strong model rigidity, lack of feedback mechanism, and low precision requirement, which is embodied in that most of the static models cannot cope with dynamic errors caused by human or environmental changes during operation, the compensation result cannot be verified, only the deviation correction is completed in theory, and real-time closed-loop confirmation cannot be realized. SUMMARY

[0006] Therefore, the present application provides a human-computer collaborative assembly method and system based on augmented reality and force feedback, which realizes the assembly error perception and compensation in a cylindrical cabin with a diameter of 500 mm through an integrated assembly error management system of "error perception-error analysis-touch compensation-result evaluation".

[0007] The human-computer collaborative assembly method based on augmented reality and force feedback of the present application comprises:

[0008] S1, a vision sensor and a force sensor are arranged at the end of the mechanical arm, the vision sensor is used to capture a 3D image of a target assembly area, and the force sensor is used to measure the force and torque of the mechanical arm; an operator operates the mechanical arm through a touch device and wears an AR device, and uses AR augmented reality technology to project a CAD assembly reference datum into an AR field of view to form a virtual contour of an assembly target and extract a virtual assembly point;

[0009] S2, the position and contour of the target in the 3D image of the vision sensor are extracted, and are compared with the virtual contour and virtual assembly point of the assembly target position of the AR augmented reality to obtain an assembly error vector; a guide vector arrow is generated at the target position extracted by the vision sensor in the AR field of view, the direction of the guide vector arrow is the direction of the assembly error vector, and the color of the vector arrow displays the size of the assembly error vector;

[0010] S3, the mechanical arm imitates the action of the operator based on a mimic learning model, compensates for the perceived assembly error under the guidance of the guide vector arrow, and judges whether the compensation is correct by using a touch feedback adjustment mechanism; wherein, the direction and color of the guide vector arrow are updated by using Kalman filtering prediction at key assembly points, and whether the assembly is successfully aligned is determined based on an assembly error compensation effect evaluation model.

[0011] Preferably, in S3, the system observation equation of the Kalman filtering is:

[0012] z t =H·x t +v t

[0013] wherein, the state vector wherein, Δx, Δy, Δz are the assembly error values calculated by the current system, is the error value change rate; the observation vector z t =[Δx′,Δy′,Δz′]T, Δx′, Δy′, Δz′ are the current actual assembly error values obtained by sensor observation, wherein the sensor observation data includes vision sensor, force sensor and mechanical arm end data; the observation noise v t is a Gaussian white noise with a mean of 0 and an observation noise covariance of R;

[0014] The observation matrix H is:

[0015]

[0016] The observation noise covariance R is:

[0017]

[0018] wherein, a variance estimate of sensor observation data;

[0019] a prediction phase:

[0020]

[0021] an update phase:

[0022] K t = P t-1|t-1 · H T · (H · P t|t-1 · H T + R) -1

[0023]

[0024] P t|t = (I - K t · H) · P t|t-1

[0025] wherein P t|t-1 denotes a state covariance prediction matrix of a previous time instant; K t| is a Kalman gain matrix; F is a state transition matrix:

[0026]

[0027] Q is a system noise covariance:

[0028]

[0029] Δt is a system sampling period, and I3denotes a three-order identity matrix.

[0030] More preferably, in the S3, the imitation learning model uses an improved inverse reinforcement learning, wherein,

[0031] a state space S t is:

[0032] S t = [P arm , V arm , R arm , F contact , ΔP, S limit ]

[0033] wherein P arm is a three-dimensional space position (x, y, z) of an end effector of the robot arm, V arm is a velocity vector of the end effector of the robot arm, R arm is an attitude of the robot arm, F contact is a contact force vector of a force sensor, ΔP is an offset vector of an assembly error vector, and S limit is a space restriction factor, V free V is the operable space volume total V is the total space volume of the cabin body

[0034] The action space A is:

[0035] A t = [Δθ'1, …, Δθ'6, I v ]

[0036] where Δθ i ' is the i-th joint angle adjustment, I v is the touch feedback intensity adjustment parameter

[0037] The reward function is

[0038] R(s, a) = -α||ΔP|| - βT adj - γ(1 - S limit ) + δS user - λE effort

[0039] where ||ΔP|| represents the spatial position residual; T adj represents the action adjustment time consumption; S limit represents the space restriction factor; S user represents the human-machine interaction score; E effort represents the robot motion intensity; α, β, γ, δ, λ are weight coefficients

[0040] With the optimal weight coefficients α, β, γ, δ, λ as the optimization strategy, the training of the imitation learning model is completed, and an action feasibility constraint factor is introduced in the strategy optimization process to impose a negative reward penalty on the inaction, and the action feasibility constraint factor is:

[0041]

[0042] where if the action a does not violate the space constraint, it is 1, otherwise it is 0.

[0043] More preferably, the touch feedback adjustment mechanism is used to judge whether the compensation is correct, specifically:

[0044] The touch feedback intensity I v is calculated:

[0045]

[0046] where represents the assembly error change rate; ω represents the absolute error size, the larger the value, the more serious the error and the more intense the feedback; ζ represents the proximity limit degree, which is used to reflect the danger degree; τ represents the error growth trend, the faster the trend, the earlier the warning, and the feedback intensity needs to be improved;

[0047] According to the size of the touch feedback intensity I v , it is determined whether the compensation is correct.

[0048] Preferably, a hierarchical touch feedback adjustment mode is adopted: when the touch feedback intensity I v ∈[0, 1.0], the assembly error ΔP is less than or equal to 0.5 mm, the error change trend is stable, and the space limitation degree S limit is high, the feedback mode is slight vibration; when the feedback intensity I v ∈(1.0, 2.0], the assembly error ΔP is greater than 0.5 mm and less than or equal to 1 mm, the error change trend increases, and the space limitation degree S limit is medium, the feedback mode is medium intensity, and a direction prompt is made; when the feedback intensity I v is greater than 2.0, the assembly error ΔP is greater than 1 mm, the error change trend rapidly increases, and the space limitation degree S limit is low, the feedback mode is strong vibration.

[0049] Preferably, when the assembly error ΔP is less than or equal to 0.5 mm, it indicates that the assembly is completed; when the assembly error ΔP is greater than 0.5 mm, the touch dynamic compensation operation is repeatedly executed until the assembly is completed.

[0050] Preferably, in the S3, the assembly error compensation effect evaluation model calculates the position error, the attitude error, and the assembly gap error, and compares the calculation results with the set tolerance threshold, if all are less than the set tolerance threshold, it is determined that the assembly alignment is successful; wherein the position error is the deviation between the actual position of the assembly component and the target position; the attitude error is the deviation between the orientation of the assembly component and the target orientation; and the assembly gap error is the gap between the components, which is obtained by detecting the axial gap and the radial gap by the vision sensor.

[0051] Preferably, the S2 is specifically as follows:

[0052] The assembly area is divided into a voxel grid;

[0053] The actual positions of the target center points in the 3D images of the vision sensors are extracted by using a center point extraction image processing technology; the virtual positions of the target center points are calculated by combining the CAD model and the assembly task requirements through a space registration transformation; and the assembly error vectors are calculated based on the virtual positions and the actual positions of the target center points.

[0054] Each sampling point error value is attributed to the corresponding voxel grid in the division, and the unsampled areas are completed by Gaussian process regression;

[0055] The module value of the assembly error vector is calculated and the modulus value is mapped as a color gradient;

[0056] a guide vector arrow is drawn at the target actual position, wherein the arrow direction is the assembly error vector direction, and the left, right, up, down, rotation or propulsion icons are explicitly indicated by the arrow; the arrow length is set to wherein k is a regulation scale factor; the arrow color is the color mapped from the modulus value; a fixed refresh frequency is set to update the guide vector arrow;

[0057] the drawn guide vector arrow is mapped into the AR field of view, and a spatial error distribution heat map is displayed in real time in the AR field of view of the operator.

[0058] Preferably, the modulus value of the calculated assembly error vector is presented in a color gradient: when the value is greater than 3 mm, a red color is displayed, indicating a high error; when the value is greater than 1 mm and less than or equal to 3 mm, an orange color is displayed, indicating a low error; and when the value is less than or equal to 1 mm, a green color is displayed, indicating a safe range.

[0059] The application also provides a human-machine collaborative assembly system based on augmented reality and force feedback, which comprises a mechanical arm, a touch device, an AR device, a vision sensor and a force sensor, and adopts the above method for human-machine collaborative assembly.

[0060] Beneficial effects:

[0061] Compared with the existing assembly error perception and compensation method, the application has the following advantages: on the one hand, the AR device is used to superimpose the assembly path and the key operation prompt to realize more intuitive error perception; on the other hand, the assembly scene is improved to simulate the learning strategy, and the error compensation mapping relationship is established in combination with the touch feedback to achieve accurate compensation by imitating the assembly action of a human expert, and a set of quantitative evaluation mechanism is established according to the assembly scene to comprehensively evaluate the effectiveness of the compensation result, thereby supporting the closed-loop optimization effect of the constructed system. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 which is an assembly error perception and error compensation operation flowchart for the narrow space of the application. DETAILED DESCRIPTION

[0063] The application will be described in detail below with reference to the drawings and examples.

[0064] The application provides a human-machine collaborative assembly method and system based on augmented reality and force feedback, which comprises:

[0065] Step 1: build an integrated assembly error management system

[0066] Firstly, the collaborative robot, touch device, vision camera, force sensor, visualization large screen, operator and host computer are used to build an integrated assembly error management system for narrow space; secondly, based on the established assembly error perception, the system realizes accurate analysis of error sources and calculation of target position missing values; finally, the touch device is used to complete the assembly error compensation in narrow space, and the compensation effect is evaluated to confirm whether the target position is reached. Among them:

[0067] (1) Collaborative robot: as a six-degree-of-freedom collaborative robot for touch interaction from the end interactive device, responsible for performing assembly operation;

[0068] (2) Touch device: as the main end device for touch interaction, mainly used for realizing touch remote operation robot, and having the function condition of realizing force tactile information perception and feedback in narrow space assembly process;

[0069] (3) Vision camera: real-time capture of assembly environment image information, mainly obtaining depth image of target assembly area, and assisting robot to avoid obstacles;

[0070] (4) Force sensor: the force sensor is installed at the end of the collaborative robot, which is used to measure force and torque, and transmit the measured data to the host computer system, so as to realize accurate positioning and operation of the robot grabbing target object;

[0071] (5) Visualization large screen: the visualization large screen is mainly used to transmit the data collected by the vision camera, force sensor and robot pose in real time to the visualization large screen, realize real-time acquisition of assembly process error data, and help the operator to quickly read various parameters;

[0072] (6) AR glasses: the spatial error results output by the algorithm model are presented in the form of error flow field in AR glasses, and the operator completes the touch mapping error compensation operation by the assembly guide direction presented in the AR glasses;

[0073] (7) Operator: plays the roles of parameter reading, operation decision, interactive control, etc. in this process, wherein the parameter reading refers to reading the robot pose transformation data, real-time change data of force sensor, image data collected by vision camera, etc. integrated in the host computer, the operation decision refers to deciding the direction of touch interactive motion instruction;

[0074] (8) Host computer: the host computer is an important part of the integrated assembly error management system, which is integrated by robot control software, force sensor control software, vision camera control software and other hardware control software, so as to facilitate the operator to read the assembly error related parameter information in real time, and provide important judgment basis for error compensation.

[0075] The eight system components above jointly constitute an integrated assembly error management system, which provides software and hardware support for realizing assembly error perception and compensation.

[0076] Step 2: Construct an assembly error perception model based on AR visual guidance combined with sensors

[0077] On the one hand, an AR augmented reality based CAD assembly reference datum is established to provide assembly visual guidance. The CAD assembly reference datum is projected into the AR field of view to form a virtual contour of the assembly target and extract virtual assembly points. On the other hand, through the fusion of an RGBD visual sensor, a force sensor, and the position information of the mechanical arm, data of the assembly target after entering the cabin to the target position are collected. The RGBD visual sensor obtains the 3D image of the target assembly area for extracting the actual position and contour of the target. The force sensor obtains the force and torque of the mechanical arm during assembly to capture the physical error of the target position. In this process, the actual coordinate systems of the AR glasses, the mechanical arm, the visual sensor, and the force sensor are unified with the AR virtual coordinate system by using the calibration point method. All data are mapped to the global coordinate system for subsequent error analysis.

[0078] Step 3: Establish an intelligent analysis model for missing values based on spatial geometry

[0079] The established assembly error missing value intelligent analysis model is based on spatial geometric coordinates to construct an assembly error vector. The assembly error vector is the difference between the position of the target in the 3D image of the visual sensor and the virtual assembly point position of the assembly target in the AR augmented reality. At the target position of the visual sensor, the assembly error vector direction is used to generate a directional arrow, and the calculated assembly error vector value is presented in color distribution: when the error is greater than 3 mm, it is displayed in red as high error, when the error is greater than 1 mm and less than or equal to 3 mm, it is displayed in orange as low error, and when the error is less than or equal to 1 mm, it is displayed in green as within the safe range. The guiding arrow and real-time error distribution heat map are presented in the AR field of view of the operator to complete the analysis and judgment of the missing values of the error.

[0080] The specific process is as follows:

[0081] (1) Decompose the assembly error vector into linear offset distance (Δx, Δy, Δz) and angular error (Δα, Δβ, Δγ), as shown in the following formula:

[0082] E = [Δx, Δy, Δz, Δα, Δβ, Δγ] (1)

[0083] (2) Use the center point extraction image processing technology to obtain the position P of the target in the 3D image of the visual sensor actual ; combined with the CAD model and the assembly task requirements, the virtual position P of the target center point is calculated through spatial registration transformationtarget ; calculate P actual and P target The difference between the offset distance and the angle error is obtained. Wherein, the physical quantity information obtained is calculated in millimeters.

[0084] The offset distance is expressed as follows:

[0085] ΔP = P actual -P target = [Δx, Δy, Δz] (2)

[0086] Wherein, P actual represents the actual point position measured by the visual sensor, P target represents the target point position extracted from the CAD model and the assembly reference, and ΔP represents the three-dimensional offset vector formed.

[0087] The offset distance is calculated as follows:

[0088]

[0089] The rotation angle attitude error around the X, Y, and Z axes is expressed as follows:

[0090] Δθ = [Δα, Δβ, Δδ] (4)

[0091] In order to make the rotation angle error more accurate and stable, the quaternion difference is used as shown in equation (5), and the angle error is converted as shown in equation (6):

[0092]

[0093] θ error = 2 · cos -1 (|q0|) (6)

[0094] Where q0 is the real part of the quaternion q = [q0, q1, q2, q3], representing the angle component of rotation.

[0095] (3) Based on the collected assembly error vector, a spatial error field map structure is constructed, and a real-time error distribution heat map layer is generated. The spatial error field map constructed is to model and visually represent each key point or area error information in the assembly space in the form of a spatial field. Through the form of error distribution heat map, the purpose is to assist the system and the operator to intuitively analyze the error distribution and its compensation direction. The specific process of constructing the assembly error vector into the error distribution heat map is as follows:

[0096] ① Establish a spatial structure: divide the assembly area into a voxel grid.

[0097] ②Error intensity encoding: the assembly error vector of formula (1) is processed by modulus, as shown in the following formula:

[0098]

[0099] The error modulus value is mapped as a color gradient for visual representation. Each sampling point error value is attributed to the corresponding voxel grid of the partition, and the unsampled area is completed by Gaussian process regression.

[0100] ③Error direction encoding: the direction of the assembly error vector is used to generate an arrow direction, and an arrow model with a direction of actual is drawn at P , where the arrow represents the direction and amplitude of the point that should be compensated, and the arrow clearly indicates icons such as left, right, up, down, rotation, and advance, where the arrow length can be set to (where k is the adjustment scale factor), and the arrow color is the error intensity encoding, and a fixed refresh rate is set to update the error direction.

[0101] ④AR view superposition processing: the constructed guide arrow is mapped to the AR field of view of the operator, and a real-time error distribution heat map is displayed.

[0102] (4) In the dynamic assembly process, since the error state will change with the evolution of the action, Kalman filtering is needed at the key assembly points to predict the assembly error evolution trend, which is used to predict the next state of the error at each step and update it combined with the sensor observation results, to improve the real-time performance and noise resistance, and to plan the action path in advance for flexible compensation. The specific error evolution prediction process is as follows:

[0103] ① Determine the input parameters: the current error state of the system is represented by the state vector , which represents the error displacement and the error displacement rate, reflecting the current system calculation obtained assembly error position and its trend, as the prediction input of dynamic compensation control; the observation vector z t = [Δx', Δy', Δz'] T represents the current actual error value obtained by sensor observation, where the sensor observation data includes visual sensor, force sensor and mechanical arm end data; the observation noise term v t is used to reflect the accuracy of the sensor, which is defined as a Gaussian white noise v t ~ N(0, R) according to the standard framework of Kalman filtering, with a mean of 0 and an observation noise covariance of R; in addition, the state transition matrix F, the observation matrix H, the system noise covariance Q and the observation noise covariance R are also needed, and the specific representation of each matrix is as follows:

[0104] For the state transition matrix F, a constant velocity motion model is used to represent the coupling between position and velocity, as shown in the following equation:

[0105]

[0106] where Δt is the system sampling period, and I3is the third-order identity matrix.

[0107] For the observation matrix H, the observation is derived from the data measured by the sensor directly, and only the error displacement is measured. The matrix contains the position error and the error rate of change, as shown in the following equation:

[0108]

[0109] where I3is the third-order identity matrix.

[0110] For the system noise covariance Q, used to represent the prediction error accumulation of the system state over time, a block diagonal structure is usually used, and Δt 4 , Δt 3 , and Δt 2 are used to represent the high-order power of the time discrete step Δt in the system model, which is derived from the integral modeling derivation when the continuous-time system state is transferred to the discrete-time filter, as shown in the following equation:

[0111]

[0112] where Δt 4 represents the influence of noise on the variance of position error, which is obtained by twice integration and variance accumulation; Δt 3 represents the coupling between position and velocity error, which is obtained by cross-covariance accumulation; Δt 2 represents the influence of noise on the variance of velocity error, which is obtained by once integration; and I3is the third-order identity matrix.

[0113] For the observation noise covariance R matrix, used to describe the measurement error characteristics of the vision sensor and the force sensor, to control the observation noise influence weight, the value is derived from the modeling of the measurement ability of the error observation sensor. The observation is mainly derived from the force sensor, and the vision sensor is auxiliary, as shown in the following equation:

[0114]

[0115] where represents the variance estimate value of the sensor displacement error.

[0116] In the above Kalman filter framework, x t is the predicted state inside the system, and z tis the actual observation of the external sensor, the relationship between them is established by the observation matrix H as shown in the following formula:

[0117] z t = H x t + v t (12)

[0118] ② Dynamic update prediction: the selected Kalman filter does not require long period of training iteration process, but uses dynamic update recursive formula for real-time prediction and correction, the specific calculation can be divided into prediction stage and update stage.

[0119] The prediction stage is shown in the following formula:

[0120]

[0121] Where P t|t-1 represents the state covariance prediction matrix of the last time.

[0122] The update stage is shown in the following formula:

[0123] K t = P t-1|t-1 ·H T ·(H·P t|t-1 ·H T +R) -1 (15)

[0124] Where K t| is the Kalman gain matrix, used to weigh the predicted value and the observed value in the state update.

[0125]

[0126] P t|t = (I-K t ·H)·P t|t-1 (17)

[0127] ③ Output trend results: according to the calculation results of the above stages, the predicted error state vector is output to represent the displacement deviation and velocity deviation of the next step, and the dynamic trajectory of the error field graph is displayed in a visual form.

[0128] (5) According to the error prediction results, the spatial error field map is constructed to analyze and judge the missing values of the error, and the missing values are presented in the field of view of the AR glasses. When the missing value calculation result is a gap deviation, it means that the current direction deviation vector of the assembly part is not inserted to the bottom, and in the AR visual guidance picture, the target assembly hole edge is displayed in red, and a push-in direction arrow appears, guiding the operator to continue the push-in assembly operation; when the missing value calculation result is a pose angle deviation, the angle deviation means that there is an included angle between the target axis and the current axis, and in the AR visual guidance picture, two virtual axes representing the target pose and the current pose are displayed, guiding the operator to complete the angle adjustment operation. Provide AR visual guidance and compensation target for the next operator to achieve accurate compensation.

[0129] Step 4: Constructing an assembly error touch mapping compensation model based on improved imitation learning

[0130] The constructed assembly error touch mapping compensation model based on improved imitation learning is based on human-machine collaborative assembly to enable the operator to control the robot more flexibly, accurately and smoothly in a small space to complete error compensation. The model can support millimeter-level trajectory reproduction, reduce human shaking, hysteresis error, cumulative error and other problems. Because the traditional method of controlling the robot has unstable control accuracy, and there is a phenomenon of blind operation in a small space, on the one hand, the behavior cloning used in traditional imitation learning directly imitates the expert's action, which ignores the changes in action caused by space limitations, and has the problem of conflict in a small space and inability to completely reproduce the expert's action. On the other hand, the traditional inverse reinforcement learning often trains in a large environment space with high operation freedom, and the reward function cannot consider the influence of space limitations. Therefore, the inverse reinforcement learning method of imitation learning is improved for the assembly scene in a small space, and an action feasibility constraint factor and an improved strategy objective function are added to optimize the adaptability of the method to the assembly environment. In this process, while ensuring the safety of assembly, the human-machine collaborative mechanism is fully utilized to complete the precise assembly operation in a small space.

[0131] The specific construction and operation process of the model is as follows:

[0132] (1) The improved imitation learning framework constructed for the assembly scene is as follows:

[0133] ① Define the execution space parameters: define the state space S t used to describe the current state of the robot and the assembly environment, and the cabin space limitation factor S is introduced according to the assembly environment limit , which can be expressed as follows:

[0134]

[0135] wherein V free is the operational space volume, V total is the total volume of the cabin.

[0136] State space S t contains multi-dimensional features as represented by the following equation:

[0137] S t = [P arm , V arm , R arm , F contact , ΔP, S limit ] (19)

[0138] wherein P arm is the three-dimensional space position (x, y, z) of the end effector of the robot arm, V arm is the end effector velocity vector of the robot arm, R arm is the robot arm pose (quaternion), F contact is the contact force vector of the force sensor, ΔP is the assembly offset vector, and S limit is the space restriction factor.

[0139] Action space A is defined to represent the actions that the robot arm can perform, including adjustment of joint angles, adjustment of force feedback intensity, etc., as represented by the following equation:

[0140] A t = [Δθ'1, …, Δθ'6, I v ] (20)

[0141] wherein Δθ' is the adjustment of the i-th joint angle, and I v is the adjustment parameter of the touch feedback intensity, used to adjust the intensity of the haptic feedback.

[0142] 2) Collecting operation data: Collect the control actions (such as joint angles, motion trajectories) of the operator and the environmental data (such as force sensors, vision data) of the operator through the touch device to remotely operate the robot arm to perform the assembly task, and record the data in real time. Align the collected position, force, touch feedback instructions and AR guidance data state in time sequence, process these processed data as a training set for imitation learning, and mark the abnormal assembly situation in the data set to facilitate model identification of risk scenarios and optimization strategies.

[0143] ③imitation learning algorithm selection: the standard inverse reinforcement learning only based on the shortest path reward of completing the task, ignoring the operation comfort, space limited and other factors, so the improved inverse reinforcement learning is used for training. Based on the assembly scene information, multi-dimensional reward factors are introduced as shown in equation (21), for the reward weight coefficients a, b, g, d, l, adaptive adjustment is realized based on the meta-learning strategy, specifically, MAML (Model-Agnostic Meta-Learning) algorithm is used to find a set of optimal initial weight adjustment strategy through multiple task training, and the weight adjustment process is as an additional optimization task, which directly achieves the effect of optimizing the weight coefficient through gradient back propagation.

[0144] R(s,a)=-α||ΔP||-βT adj -γ(1-S limit )+δS user -λE effort (21)

[0145] Wherein, ||AP|| represents the spatial position residual, which is used to encourage to reduce the error, and the spatial position data used in this process is the spatial position state collected before Kalman filter prediction; T adj represents the action adjustment time, which is used to encourage to improve efficiency; S limit represents the space limited factor, which is used to encourage to preferentially use free space to complete adjustment; S user represents the human-computer interaction score, which reflects the quality and adaptability of human-computer interaction combined with the scene; E effort represents the motion intensity of the robot arm, which is used to represent the energy consumption of the assembly process. In addition, the weight coefficient a is the error penalty coefficient, which represents the sensitivity to the assembly position deviation, and the greater the value, the more the system pays attention to the accuracy; The weight coefficient b is the action adjustment penalty, which represents the posture adjustment amplitude in the action completion process, and the greater the value, the more inclined to assemble in place at one time; The weight coefficient g is the limit constraint coefficient, which reflects whether it is within the error tolerance range, and the greater the value, the more serious the over-limit penalty; The weight coefficient d is the user score weight, which represents the satisfaction of the human-computer collaborative user through touch feedback, and the greater the value, the more the system learns to human preference; The weight coefficient l is the energy consumption penalty, which represents the energy consumption corresponding to the assembly action, and the greater the value, the more inclined to save resources.

[0146] In addition, according to the assembly scene, negative reward punishment needs to be directly imposed on the unactionable action in the strategy optimization process, so the action feasibility constraint factor is introduced as shown in the following formula:

[0147]

[0148] Wherein, if the action a does not violate the space constraint, it is 1, otherwise it is 0.

[0149] ④ Training the imitation learning model: The improved imitation learning model is trained with the above-collected data set. The advantage of this model is not only that it can adaptively balance assembly accuracy, speed, human-machine comfort and space constraints, but also that it can obtain weight tuning from data instead of manually. The specific training process is: First, through the reward function network discriminator D Φ (s,a) training recovery reward function R(s,a), input state s, feature vector of action a, output D Φ (s,a)∈[0,1], determines whether the state-action pair comes from expert demonstration, in the discriminator D Φ The feasibility judgment result of the explicit fusion action in the output of (s,a) is Feasibility(a), and the improved form is D Φ (s,a)=σ(f Φ (s,a))·Feasibility(a); Secondly, the policy network learns the policy π that can obtain the maximum reward θ (a|s), input the current state s, output the distribution of action a, including joint angle adjustment and touch feedback intensity; finally, the value function network V is used Φ (s) Optimize the strategy.

[0150] ⑤Model verification and adjustment: Verify in a test environment that the residual accuracy ||ΔP|| of the imitation learning model is less than 0.5 mm, ensuring that the robotic arm can accurately imitate the operator's movements and adapt to the errors of the assembly environment.

[0151] (2) According to the trained improved imitation learning model, the touch dynamic feedback operation is performed using the established touch feedback adjustment mechanism, and the touch feedback adjustment factor I is introduced. v , which is used to control the real-time feedback strength of the touch device, as expressed in the following formula:

[0152]

[0153] in, Indicates the error change rate and is used to enhance dynamic feedback.

[0154] During the touch dynamic compensation operation, the touch feedback intensity needs to be adjusted according to the magnitude of the assembly error. The touch feedback adjustment mechanism is used to improve the operator's perception of changes in assembly trends, ensuring that the operator can perceive subtle differences in the assembly process and judge whether the compensation is correct based on the feedback intensity, while avoiding excessive or insufficient feedback intensity. v Defines a graded touch feedback adjustment mode. When the feedback intensity I v ∈[0, 1.0], when the assembly error ΔP is less than 0.5 mm, the error change trend is stable and the spatial limitation S limitFor high, feedback mode is slight vibration; when feedback intensity I v ∈[1.0, 2.0], assembly error ΔP is greater than 0.5mm and less than 1mm, error change trend increases, space limitation degree S limit is medium, feedback mode is medium intensity and makes directional prompt; when feedback intensity I v is greater than 2.0, assembly error ΔP is greater than 1mm, error change trend increases rapidly, space limitation degree S limit is low, feedback mode is strong vibration.

[0155] (3) when assembly error ΔP is less than 0.5mm, it indicates that assembly is completed; when assembly error ΔP is still greater than 0.5mm, the above operation process (2) is continuously repeated until assembly is completed.

[0156] Step 5: establishing assembly error compensation effect evaluation model

[0157] The established assembly error compensation effect evaluation model is to ensure that assembly error has been eliminated or reduced to an acceptable range after error compensation, and it has ensured that precise assembly is achieved. Assembly error compensation effect evaluation indexes mainly include position error, attitude error and assembly gap error, and after each compensation, the three index errors are automatically calculated and it is judged whether the assembly precision meets the standard, and error change before and after compensation is displayed. The specific process of assembly error compensation effect evaluation is as follows:

[0158] (1) after each operator performs fine adjustment error compensation through touch control, the system reacquires error data through sensors and updates error analysis model data.

[0159] (2) the calculation results of the three evaluation indexes of position error, attitude error and assembly gap error are compared with the set tolerance threshold, and when the evaluation result value is less than the set tolerance threshold, it is determined that assembly alignment is successful. The related definitions and calculations of evaluation indexes are as follows:

[0160] Position error refers to the deviation between actual position of assembly parts and target position. This error is usually measured by Euclidean distance in three-dimensional space, and the calculation formula is as shown in the following formula:

[0161]

[0162] Wherein, x actual , y actual , z actual represent actual position of assembly parts, x target , y target , z target represent target position.

[0163] The pose error refers to the difference between the orientation of the assembled component and the target orientation, which is usually described using a quaternion, and the calculation formula is as follows:

[0164]

[0165] Wherein, R actual represents the rotation matrix of the actual assembly, R target represents the rotation matrix of the target assembly position.

[0166] The assembly gap error refers to whether there is a non-required gap between the components, and the axial gap and radial gap are detected by using a visual sensor to calculate the distance.

[0167] To sum up, the above is only a preferred embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A human-machine collaborative assembly method based on augmented reality and force feedback, characterized in that: include: S1, a vision sensor and a force sensor are set at the end of the robotic arm. The vision sensor is used to capture a 3D image of the target assembly area, and the force sensor is used to measure the force and torque of the robotic arm; The operator uses a touch device to operate the robotic arm and wears an AR device. Using AR augmented reality technology, the CAD assembly reference datum is projected into the AR field of view to form a virtual outline of the assembly target and extract virtual assembly points. S2, extracting the position and outline of the target in the 3D image of the visual sensor, and comparing them with the virtual outline and virtual assembly point of the assembly target position in AR augmented reality to obtain an assembly error vector; generating a guidance vector arrow at the target position extracted by the visual sensor in the AR field of view, the direction of the guidance vector arrow is the direction of the assembly error vector, and the color of the vector arrow indicates the magnitude of the assembly error vector; In S3, the robotic arm imitates the operator's actions based on the imitation learning model, compensates for the perceived assembly error under the guidance of the guidance vector arrow, and uses the touch feedback adjustment mechanism to determine whether the compensation is correct; among them, Kalman filtering is used to predict and update the direction and color of the guidance vector arrow at key assembly points, and based on the assembly error compensation effect evaluation model, it is determined whether the assembly is aligned successfully.

2. The method according to claim 1, wherein In S3, the system observation equation of the Kalman filter is: z t =H·x t +v t Among them, the state vector Among them, Δx, Δy, Δz are the assembly error values ​​calculated by the current system. is the error value change rate; observation vector z t =[Δx',Δy',Δz'] T , Δx', Δy', Δz' are the current actual assembly error values ​​obtained by sensor observation, where the sensor observation data includes visual sensor, force sensor and robot end data; observation noise v t is a Gaussian white noise with a mean of 0 and an observation noise covariance of R; The observation matrix H is: The observation noise covariance R is: in, is the variance estimate of the sensor observation data; Prediction stage: P t|t-1 =F·P t-1|t-1 ·F T +Q Update phase: K t =P t-1|t-1 ·H T ·(H·P t|t-1 ·H T +R) -1 P t|t =(I-K t ·H)·P t|t-1 Among them, P t|t-1 Represents the state covariance prediction matrix of the previous moment; K t| is the Kalman gain matrix; F is the state transfer matrix: Q is the system noise covariance: Δt is the system sampling period, and I3 represents the third-order unit matrix.

3. The method according to claim 1 or 2, wherein: In S3, the imitation learning model adopts improved inverse reinforcement learning, wherein, State space S t for: S t =[P arm ,V arm ,R arm ,F contact ,ΔP,S limit ] Among them, P arm is the three-dimensional spatial position (x, y, z) of the end effector of the robotic arm, V arm is the velocity vector of the end of the robot arm, R arm is the posture of the robotic arm, F contact is the contact force vector of the force sensor, ΔP is the offset vector of the assembly error vector, S limit is the spatial restriction factor, V free is the volume of the operable space, V total is the total space volume of the cabin; The action space A is: A t =[Δθ′1,…,Δθ′6,I v ] Where Δθ i ′ is the i-th joint angle adjustment, I v Adjust parameters for touch feedback intensity; The reward function is R(s,a)=-α||ΔP||-βT adj -γ(1-S limit )+δS user -λE effort Among them, ||ΔP|| represents the spatial position residual; T adj Indicates the time taken to adjust the action; S limit represents the spatial restriction factor; S user represents the human-computer interaction score; E effort represents the strength of the robot arm movement; α, β, γ, δ, λ are weight coefficients; The optimal weight coefficients α, β, γ, δ, and λ are used as the optimization strategy to complete the training of the imitation learning model. In the process of strategy optimization, the action feasibility constraint factor is introduced to impose negative reward penalties on inaction actions. The action feasibility constraint factor is: Among them, if action a does not violate the spatial constraint, it is 1, otherwise it is 0.

4. The method according to claim 1, wherein Use the touch feedback adjustment mechanism to determine whether the compensation is correct, specifically: Calculate touch feedback intensity I v : in, represents the rate of change of assembly error; ω represents the absolute error; the larger its value, the more serious the error and the stronger the feedback; ζ represents the degree of approaching the limit, which is used to reflect the degree of danger; τ represents the error growth trend; the faster the trend, the earlier the warning, and the need to increase the feedback intensity; According to the touch feedback intensity I v The size of the compensation can be used to determine whether the compensation is correct.

5. The method according to claim 4, wherein Adopt graded touch feedback adjustment mode: when touch feedback intensity I v ∈[0, 1.0], the assembly error ΔP is less than or equal to 0.5 mm, the error change trend is stable, and the space limitation S limit When the feedback intensity is high, the feedback mode is slight vibration; when the feedback intensity is v ∈(1.0, 2.0], the assembly error ΔP is greater than 0.5 mm and less than or equal to 1 mm, the error change trend increases, and the space limitation S limit When the feedback intensity is medium, the feedback mode is medium intensity and direction prompts are given; when the feedback intensity is I v Greater than 2.0, assembly error ΔP greater than 1 mm, error change trend increases rapidly, space limitation S limit When it is low, the feedback mode is strong vibration.

6. The method according to claim 4 or 5, characterized in that When the assembly error ΔP is less than or equal to 0.5 mm, it indicates that the assembly is completed; when the assembly error ΔP is greater than 0.5 mm, the touch motion compensation operation is repeated until the assembly is completed.

7. The method according to claim 1, wherein In S3, the assembly error compensation effect evaluation model calculates the position error, posture error and assembly gap error, and compares the calculation results with the set tolerance threshold. If all are less than the set tolerance threshold, the assembly alignment is determined to be successful; wherein, the position error is the deviation between the actual position of the assembly component and the target position; the posture error is the deviation between the orientation of the assembly component and the target orientation; the assembly gap error is the gap between the components, which is obtained by detecting the axial gap and radial gap by the visual sensor.

8. The method according to claim 1, wherein The S2 is specifically as follows: Divide the assembly area into a voxel grid; Using center point extraction image processing technology, the actual position of the target center point in the visual sensor 3D image is extracted respectively; the virtual position of the target center point is calculated through spatial registration transformation based on the CAD model and assembly task requirements; Calculate the assembly error vector based on the virtual position and actual position of the target center point The error value of each sampling point is attributed to the corresponding divided voxel grid, and the unsampled area is supplemented by Gaussian process regression; Calculate the magnitude of the assembly error vector And map the modulus value to a color gradient; Draw a guidance vector arrow at the actual target position, where the arrow direction is the direction of the assembly error vector, and the arrow clearly indicates left, right, up, down, rotation or push icon; the arrow length is set to Where k is the adjustment scale factor; the arrow color is the color of the modulus mapping; set a fixed refresh frequency to update the guidance vector arrow; The drawn guidance vector arrows are mapped to the AR field of view, and the spatial error distribution heat map is displayed in real time in the operator's AR field of view.

9. The method according to claim 8, wherein The calculated assembly error vector magnitude Presented as a color gradient: When it is larger than 3 mm, it will be displayed in red, indicating a high error. When it is greater than 1 mm and less than or equal to 3 mm, it will be displayed in orange, indicating a low error. When it is less than or equal to 1 mm, it will be displayed in green, which is within the safe range.

10. A human-machine collaborative assembly system based on augmented reality and force feedback, characterized in that: The device comprises a robotic arm, a touch device, an AR device, a visual sensor and a force sensor, and adopts the method according to any one of claims 1 to 9 to perform human-machine collaborative assembly.

Images