A man-machine collaborative assembly method and system based on augmented reality and force feedback

Abstract

The application discloses a kind of based on augmented reality and force feedback man-machine collaborative assembly method and system.The application is directly perceived more intuitively by AR equipment superposition assembly path and key operation prompt in one aspect;On the other hand, through the assembly scene, the imitation learning strategy is improved, the error compensation mapping relationship is established by combining touch feedback, the precise compensation is achieved by imitating the assembly action of human expert, and a set of quantitative evaluation mechanism is established according to the assembly scene, the effectiveness of the compensation result is comprehensively evaluated, and the system constructed supports closed-loop optimization effect.

Description

Technical Field

[0001] This invention relates to the field of human-machine collaborative assembly technology, specifically to a human-machine collaborative assembly method and system based on augmented reality and force feedback. Background Technology

[0002] Current methods for error perception and compensation in confined spaces mainly involve technologies such as automated assembly with industrial robots, human-machine collaborative assembly, geometric modeling, and linear deviation correction.

[0003] Existing industrial robot automated assembly technologies primarily rely on robotic arms to complete assembly tasks within relatively open workspaces, using static fixtures and path planning to position, move, and connect parts. However, when faced with assembly errors in confined spaces, issues arise such as limited robotic arm movement, inability to accurately reach target positions, and the inability to adjust automated assembly strategies in real time.

[0004] Human-robot collaborative assembly involves introducing operators into the assembly process, allowing humans to lead and control robots or share control with robots, thereby improving task flexibility and adaptability to complex environments. However, it suffers from problems such as a lack of intuitive error perception, information fragmentation, and strong individual dependence. Specifically, operators find it difficult to perceive "whether the assembly is currently accurate," "where the error is," and "how to adjust the next action." The perception results (vision system) and human hand actions (force control) are not integrated. Successful assembly highly depends on the operator's experience, lacks a unified strategy model, and cannot be promoted or reused.

[0005] Currently, error compensation is generally used to predict and adjust assembly errors. However, most existing error compensation methods use geometric modeling and linear deviation correction, such as using pose correction models and static error offset correction. These methods have problems such as excessive model rigidity, lack of feedback mechanisms, and low accuracy requirements. Specifically, most static models cannot cope with dynamic errors caused by changes in people or environment during operation. The compensation results cannot be verified, and deviation correction is only completed in theory, without real-time closed-loop confirmation. Summary of the Invention

[0006] In view of this, the present invention provides a human-machine collaborative assembly method and system based on augmented reality and force feedback. Through an integrated assembly error management system of "error perception - error analysis - touch compensation - result evaluation", it realizes the perception and compensation of assembly errors in a cylindrical cabin with a diameter of 500 mm.

[0007] The human-computer collaborative assembly method based on augmented reality and force feedback of the present invention includes:

[0008] S1, A vision sensor and a force sensor are set at the end of the robotic arm. The vision sensor is used to capture a 3D image of the target assembly area, and the force sensor is used to measure the force and torque of the robotic arm. The operator operates the robotic arm through a touch device and wears an AR device to project the CAD assembly reference datum into the AR field of view to form a virtual outline of the assembly target and extract the virtual assembly points.

[0009] S2, extract the position and outline of the target in the 3D image of the visual sensor, and compare it with the virtual outline and virtual assembly point of the assembly target position in AR augmented reality to obtain the assembly error vector; generate a guide vector arrow at the target position extracted by the visual sensor in the AR field of view, the direction of the guide vector arrow is the direction of the assembly error vector, and the color of the vector arrow indicates the magnitude of the assembly error vector.

[0010] S3: The robotic arm imitates the operator's actions based on an imitation learning model. Guided by the guide vector arrow, it compensates for perceived assembly errors and uses a touch feedback adjustment mechanism to determine whether the compensation is correct. Specifically, at key assembly points, Kalman filtering is used to predict and update the direction and color of the guide vector arrow, and the assembly alignment is determined based on the assembly error compensation effect evaluation model.

[0011] Preferably, in S3, the system observation equation of the Kalman filter is:

[0012] z t =H·x t +v t

[0013] Wherein, the state vector Where Δx, Δy, and Δz are the assembly error values ​​calculated by the current system. The rate of change of the error value; the observation vector z t = [Δx′, Δy′, Δz′]T, where Δx′, Δy′, and Δz′ are the current actual assembly error values ​​obtained from sensor observations. The sensor observation data includes data from vision sensors, force sensors, and the robotic arm's end effector; the observation noise v t It is Gaussian white noise with a mean of 0 and an observation noise covariance of R;

[0014] The observation matrix H is:

[0015]

[0016] The observation noise covariance R is:

[0017]

[0018] in, This is an estimate of the variance of the sensor observation data;

[0019] Prediction phase:

[0020]

[0021] Update phase:

[0022] K t =P t-1|t-1 ·H T ·(H·P t|t-1 ·H T +R) -1

[0023]

[0024] P t|t =(IK t ·H)·P t|t-1

[0025] Among them, P t|t-1 K represents the state covariance prediction matrix at the previous time step; t| Here, F is the Kalman gain matrix; F is the state transition matrix.

[0026]

[0027] Q represents the system noise covariance.

[0028]

[0029] Δt is the system sampling period, and I3 represents the third-order identity matrix.

[0030] Preferably, in S3, the imitation learning model employs improved inverse reinforcement learning, wherein,

[0031] State space S t for:

[0032] S t =[P arm V arm ,R arm ,F contact ,ΔP,S limit ]

[0033] Among them, P arm V represents the three-dimensional spatial position (x, y, z) of the robotic arm's end effector. arm R is the velocity vector at the end effector of the robotic arm. arm For the robotic arm's posture, F contact Let S be the contact force vector of the force sensor, ΔP be the offset vector of the assembly error vector, and S be the contact force vector of the force sensor. limit As a spatially constrained factor, V free V is the operable space volume. total This refers to the total volume of the cabin space;

[0034] Action space A is:

[0035] A t =[Δθ′1,…,Δθ′6,I v ]

[0036] Where, Δθ i ′ represents the angle adjustment of the i-th joint, I v This is a parameter for adjusting the intensity of touch feedback;

[0037] The reward function is

[0038] R(s,a)=-α||ΔP||-βT adj -γ(1-S limit )+δS user -λE effort

[0039] Where ||ΔP|| represents the spatial location residual; T adj Indicates the time taken for action adjustment; S limit S represents the space constraint factor; user E represents the human-computer interaction score; effort The values ​​represent the motion intensity of the robotic arm; α, β, γ, δ, and λ are weighting coefficients.

[0040] The imitation learning model is trained using optimal weight coefficients α, β, γ, δ, and λ as the optimization strategy. An action feasibility constraint factor is introduced during the strategy optimization process to impose a negative reward penalty on infeasible actions. This action feasibility constraint factor is:

[0041]

[0042] The value is 1 if action a does not violate spatial constraints, and 0 otherwise.

[0043] A better approach is to use a haptic feedback adjustment mechanism to determine whether the compensation is correct, specifically:

[0044] Calculate the touch feedback intensity I v :

[0045]

[0046] in, ζ represents the rate of change of assembly error; ω represents the magnitude of absolute error, the larger the value, the more serious the error and the stronger the feedback; ζ represents the degree of approaching the limit, used to reflect the degree of danger; τ represents the error growth trend, the faster the trend, the earlier the warning, and the need to increase the feedback intensity.

[0047] Based on touch feedback intensity I v The size of the value determines whether the compensation is correct.

[0048] A preferred approach is to adopt a graded touch feedback adjustment mode: when the touch feedback intensity I... v ∈[0, 1.0], assembly error ΔP is less than or equal to 0.5 mm, error variation trend is stable, and space constraint S limit When the level is high, the feedback mode is slight vibration; when the feedback intensity I... v ∈(1.0, 2.0], assembly error ΔP is greater than 0.5 mm but less than or equal to 1 mm, the error trend is increasing, and the space constraint S limit When the feedback intensity is medium, the feedback mode is of medium strength, and directional prompts are provided; when the feedback intensity is I... v Greater than 2.0, assembly error ΔP greater than 1 mm, error change trend increases rapidly, space constraint S limit When the setting is low, the feedback mode is strong vibration.

[0049] Ideally, when the assembly error ΔP is less than or equal to 0.5 mm, the assembly is considered complete; when the assembly error ΔP is greater than 0.5 mm, the touch dynamic compensation operation is repeated until the assembly is complete.

[0050] Preferably, in step S3, the assembly error compensation effect evaluation model calculates the position error, attitude error, and assembly gap error, and compares the calculation results with a set tolerance threshold. If all are less than the set tolerance threshold, the assembly alignment is determined to be successful. Here, the position error is the deviation between the actual position and the target position of the assembled component; the attitude error is the deviation between the orientation of the assembled component and the target orientation; and the assembly gap error is the gap between the components, which is obtained by detecting the axial gap and the radial gap using a vision sensor.

[0051] Preferably, S2 is as follows:

[0052] The assembly area is divided into voxel grids;

[0053] Using center point extraction image processing technology, the actual position of the target center point in the 3D image from the vision sensor is extracted; combined with the CAD model and assembly task requirements, the virtual position of the target center point is calculated through spatial registration transformation; and the assembly error vector is calculated based on the virtual and actual positions of the target center point.

[0054] Each sampling point error value is assigned to the corresponding voxel grid, and the unsampled areas are completed by Gaussian regression.

[0055] Calculate the magnitude of the assembly error vector And map the magnitude to a color gradient;

[0056] Draw a guide vector arrow at the actual target location, where the arrow direction is the assembly error vector direction, clearly indicating left, right, up, down, rotation, or advance icons; the arrow length is set to... Where k is the adjustment scaling factor; the arrow color is the color mapped by the modulus; a fixed refresh rate is set to update the guide vector arrow;

[0057] The drawn guide vector arrows are mapped onto the AR field of view, and a heat map of spatial error distribution is displayed in real time in the operator's AR field of view.

[0058] Ideally, the calculated assembly error vector magnitude will be... Presented as a color gradient: when A value greater than 3 mm will be displayed in red, indicating a high error; when... An error greater than 1 mm but less than or equal to 3 mm is indicated by an orange color, signifying a low error. When the diameter is less than or equal to 1 mm, it will be displayed in green, indicating that it is within the safe range.

[0059] The present invention also provides a human-machine collaborative assembly system based on augmented reality and force feedback, including a robotic arm, a touch device, an AR device, a vision sensor and a force sensor, and performs human-machine collaborative assembly using the above method.

[0060] Beneficial effects:

[0061] Compared to existing assembly error perception and compensation methods, this invention achieves more intuitive error perception by overlaying assembly paths and key operation prompts onto AR devices. Furthermore, it improves the imitation learning strategy through assembly scenarios, establishes an error compensation mapping relationship by combining touch feedback, and achieves precise compensation by mimicking the assembly actions of human experts. A quantitative evaluation mechanism is also established based on the assembly scenarios to comprehensively assess the effectiveness of the compensation results, supporting the constructed system to achieve closed-loop optimization. Attached Figure Description

[0062] Figure 1 This is a flowchart of the assembly error sensing and error compensation operation in a confined space according to the present invention. Detailed Implementation

[0063] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0064] This invention provides a human-computer collaborative assembly method and system based on augmented reality and force feedback, comprising:

[0065] Step 1: Establish an integrated assembly error management system

[0066] First, a collaborative robotic arm, touch-screen devices, vision cameras, force sensors, a large visualization screen, operators, and a host computer are used to jointly build an integrated assembly error management system for confined spaces. Second, based on the established assembly error perception, the system accurately analyzes the sources of errors and calculates the missing values ​​at target positions. Finally, the touch-screen devices are used to compensate for assembly errors in confined spaces, and the compensation effect is evaluated to confirm whether the target position has been achieved. Specifically:

[0067] (1) Collaborative robotic arm: A six-degree-of-freedom collaborative robotic arm that serves as a touch-interactive slave device is responsible for performing assembly operations;

[0068] (2) Touch device: As the main touch interaction device, it is mainly used to realize touch remote operation of robotic arm, and has the functional conditions to achieve force tactile information perception and feedback during assembly in a narrow space.

[0069] (3) Vision camera: captures real-time image information of the assembly environment, mainly to obtain depth images of the target assembly area, and assists the robotic arm in avoiding obstacles;

[0070] (4) Force sensor: A force sensor is installed at the end of the collaborative robotic arm to measure force and torque, and transmit the measured data to the host computer system, thereby realizing the precise positioning and operation of the robotic arm to grasp the target object;

[0071] (5) Visualization screen: The visualization screen is mainly used to transmit the data collected in real time by the vision camera, force sensor and robot arm posture to the visualization screen, so as to realize the real-time acquisition of assembly process error data and help operators quickly read various parameters.

[0072] (6) AR glasses: The spatial error results output by the algorithm model are presented in the AR glasses as an error flow field. The operator completes the touch mapping error compensation operation by following the assembly guidance direction presented in the AR glasses.

[0073] (7) Operator: In this process, the operator plays the role of parameter reading, operation decision-making, and interactive control. Parameter reading refers to reading the posture transformation data of the robotic arm integrated in the host computer, the real-time change data of the force sensor, and the image data acquired by the vision camera. Operation decision-making refers to deciding the direction of the tactile interactive motion command.

[0074] (8) Host computer: The host computer is an important component of the integrated assembly error management system. It is a combination of hardware control software such as robotic arm control software, force sensor control software, and vision camera control software. It allows operators to read assembly error-related parameter information in real time and provides an important basis for error compensation.

[0075] The above eight system components together constitute an integrated assembly error management system, providing hardware and software support for realizing assembly error perception and compensation.

[0076] Step 2: Construct an assembly error perception model based on AR visual guidance and sensors

[0077] On the one hand, augmented reality (AR) is used to establish a CAD-based assembly reference datum to provide visual guidance for assembly. The CAD assembly reference datum is projected into the AR field of view to form a virtual outline of the assembly target and extract virtual assembly points. On the other hand, by fusing RGBD vision sensors with force sensors and the position information of the robotic arm's end effector, data on the assembly target's arrival at its target position after entering the cabin is collected. The RGBD vision sensor acquires a 3D image of the target assembly area to extract the target's actual position and outline, while the force sensor acquires the force and torque of the robotic arm during assembly to capture the physical errors in the target position. In this process, a calibration point method is used to unify the actual coordinate systems of the AR glasses, robotic arm, vision sensors, and force sensors with the AR virtual coordinate system, mapping all data to a global coordinate system for subsequent error analysis.

[0078] Step 3: Establish a missing value intelligent analysis model based on spatial geometry

[0079] The established intelligent analysis model for missing assembly error values ​​is based on constructing an assembly error vector using spatial geometric coordinates. This assembly error vector represents the difference between the target's position in the 3D image from the visual sensor and the virtual assembly point in the AR (Augmented Reality) assembly target location. At the target position on the visual sensor, the direction of the assembly error vector is used as a generative arrow guide, and the calculated assembly error vector value is presented using a color distribution: red indicates a high error when the error is greater than 3 mm, orange indicates a low error when the error is greater than 1 mm but less than or equal to 3 mm, and green indicates a safe range when the error is less than or equal to 1 mm. Guiding arrows and a real-time error distribution heatmap are presented in the operator's AR field of view to complete the analysis and judgment of missing error values.

[0080] The specific process is as follows:

[0081] (1) The assembly error vector is decomposed into linear offset distances (Δx, Δy, Δz) and angular errors (Δα, Δβ, Δγ), as shown in the following formula:

[0082] E=[Δx,Δy,Δz,Δα,Δβ,Δγ] (1)

[0083] (2) Using center point extraction image processing technology, the position P of the target in the 3D image of the visual sensor is obtained. actual The virtual position P of the target center point is calculated through spatial registration transformation by combining the CAD model and assembly task requirements.target ; Calculate P actual and P target The difference between the values ​​yields the offset distance and angular error. All physical quantities are calculated in millimeters.

[0084] The offset distance is expressed by the following formula:

[0085] ΔP=P actual -P target =[Δx,Δy,Δz] (2)

[0086] Among them, P actual P represents the actual point position measured by a vision sensor. target ΔP represents the target point position extracted from the CAD model and assembly datum, and ΔP represents the three-dimensional offset vector formed.

[0087] The offset distance is calculated as follows:

[0088]

[0089] The rotational angle attitude error around the X, Y, and Z axes is expressed by the following formula:

[0090] Δθ=[Δα,Δβ,Δδ] (4)

[0091] To make the rotation angle error more accurate and stable, a quaternion difference is used as shown in equation (5), which is then converted into an angle error as shown in equation (6):

[0092]

[0093] θ error =2·cos -1 (|q0|) (6)

[0094] Where q0 is the real part of the quaternion q = [q0, q1, q2, q3], representing the angular component of the rotation.

[0095] (3) Based on the collected assembly error vectors, a spatial error field map structure is constructed to generate a real-time error distribution heatmap. The constructed spatial error field map models and visualizes the error information of each key point or region in the assembly space in the form of a spatial field, and presents it in the form of an error distribution heatmap. The purpose is to help the system and operators intuitively analyze the error distribution and its compensation direction. The specific process of constructing the assembly error vector into an error distribution heatmap is as follows:

[0096] ① Establish spatial structure: Divide the assembly area into voxel meshes.

[0097] ② Error intensity encoding: The assembly error vector of formula (1) is modulo-valued, as shown in the following formula:

[0098]

[0099] Error magnitude The error value is mapped to a color gradient for visualization. Each sampling point error value is assigned to the corresponding voxel grid, and unsampled regions are filled in using Gaussian regression.

[0100] ③ Error direction encoding: The direction of the assembly error vector is used to generate arrow guidance, in P actual The drawing direction is The arrow model represents the direction and magnitude that should be compensated for at that point. The arrows clearly indicate left, right, up, down, rotation, and advance icons. The arrow length can be set to... (where k is the adjustment scaling factor), the arrow color is the error intensity code, and a fixed refresh frequency is set to update the error direction.

[0101] ④ AR view overlay processing: The constructed guide arrows are mapped onto the operator's AR field of view, and the error distribution heat map is displayed in real time.

[0102] (4) During dynamic assembly, since the error state changes with the evolution of actions, Kalman filtering is needed at key assembly points to predict the evolution trend of assembly errors. This is used to predict the next state of the error at each step and update it in conjunction with sensor observations, improving real-time performance and noise resistance, and allowing for advance planning of action paths for flexible compensation. The specific error evolution prediction process is as follows:

[0103] ① Determine the input parameters: through the state vector This represents the current error state of the system, namely the error displacement and the rate of change of the error displacement, reflecting the assembly error position and its changing trend obtained by the current system calculation, and serving as the predictive input for dynamic compensation control; through the observation vector z t =[Δx′,Δy′,Δz′] T This represents the current actual error value obtained from sensor observations, where the sensor observation data includes data from vision sensors, force sensors, and the robotic arm's end effector; the noise term v is observed. t To reflect the accuracy of the sensor, according to the Kalman filter standard framework, it is defined as Gaussian white noise v t ~N(0,R), with a mean of 0 and an observation noise covariance of R; in addition, the state transition matrix F, the observation matrix H, the system noise covariance Q, and the observation noise covariance R are also required. The specific representations of each matrix are as follows:

[0104] For the state transition matrix F, a constant velocity motion model is used to characterize the coupling between position and velocity, as shown in the following equation:

[0105]

[0106] Where Δt is the system sampling period, and I3 represents the third-order identity matrix.

[0107] For the observation matrix H, the observation source is data directly measured by the sensor, and only the error displacement is measured. The matrix contains the position error and the rate of change of error, as shown in the following formula:

[0108]

[0109] Where I3 represents the third-order identity matrix.

[0110] The system noise covariance Q, used to represent the cumulative prediction error of the system state over time, is often constructed using a block diagonal structure, and the constructed Δt... 4 , Δt 3 , Δt 2 The higher powers used to represent the time discrete step length Δt in the system model are derived from the integral modeling derivation when the continuous-time system state transitions to the discrete-time filter, as shown in the following equation:

[0111]

[0112] Where Δt 4 The effect of noise on the position error variance is represented by two integrals and variance accumulation; Δt 3 This indicates the coupling between position and velocity errors, obtained through cross-covariance accumulation; Δt 2 I3 represents the effect of noise on the velocity error variance, obtained through a first integration; I3 represents the third-order identity matrix.

[0113] The observation noise covariance matrix R is used to describe the measurement error characteristics of the vision sensor and the force sensor, and to control the influence weight of the observation noise. Its value comes from the modeling of the measurement capability of the error observation sensor. The observation mainly comes from the force sensor, and the auxiliary observation comes from the vision sensor, as shown in the following formula:

[0114]

[0115] in This represents the variance estimate of the sensor displacement error.

[0116] In the above framework for constructing a Kalman filter, x t It is the predicted state within the system, z tThe actual observations from external sensors are used to establish the relationship between the two through the observation matrix H, as shown in the following formula:

[0117] z t =H·x t +v t (12)

[0118] ② Dynamic update prediction: The selected Kalman filter does not require a long period of training iteration. Instead, it uses a dynamic update recursive formula for real-time prediction and correction. The specific calculation can be divided into a prediction stage and an update stage.

[0119] The prediction phase is represented by the following formula:

[0120]

[0121] Where P t|t-1 This represents the state covariance prediction matrix at the previous time step.

[0122] The update phase is represented by the following formula:

[0123] K t =P t-1|t-1 ·H T ·(H·P t|t-1 ·H T +R) -1 (15)

[0124] Where K t| This is the Kalman gain matrix, used to weigh the weights of predicted and observed values ​​in state updates.

[0125]

[0126] P t|t =(IK t ·H)·P t|t-1 (17)

[0127] ③ Output trend results: Based on the calculation results of the above stages, output the predicted error state vector. This is used to represent the next displacement and velocity deviations, and the dynamic trajectory of the error field map is displayed in a visual form.

[0128] (5) Based on the spatial error field map constructed from the above error prediction results, analyze and judge the missing error values, and present the missing values ​​in the AR glasses' field of view. When the missing value calculation result is that there is a deviation in the gap, it indicates that the directional deviation vector of the assembly part is not currently inserted to the bottom. In the AR visual guidance screen, the edge of the target assembly hole turns red and a push-in direction arrow appears, guiding the operator to continue performing the push-in assembly operation. When the missing value calculation result is that there is a deviation in the attitude angle, the angle deviation indicates that there is an angle between the target axis and the current axis. In the AR visual guidance screen, two virtual axes are displayed and rendered to represent the target attitude and the current attitude respectively, guiding the operator to complete the angle adjustment and alignment operation. This provides AR visual guidance and compensation targets for the operator to achieve accurate compensation in the next step.

[0129] Step 4: Construct an assembly error touch mapping compensation model based on improved imitation learning

[0130] The constructed assembly error touch mapping compensation model based on improved imitation learning is based on human-machine collaborative assembly. To enable operators to more flexibly, accurately, and smoothly complete error compensation in confined spaces by manipulating the robotic arm through touch, the model aims to learn and imitate the operator's handheld touch device's movement trajectory, thereby adaptively adjusting the slave robotic arm's state in new environments and learning to compensate for existing pose errors. This model can support millimeter-level trajectory reproduction, reducing problems such as human shaking, hysteresis errors, and cumulative errors. Traditional robotic arm manipulation methods suffer from unstable control precision and blind operation in confined spaces. On the one hand, traditional imitation learning methods, which directly imitate expert actions through behavior cloning, ignore the changes in action caused by spatial constraints, leading to conflicts and incomplete reproduction of expert actions in confined spaces. On the other hand, traditional inverse reinforcement learning often uses large training environments with high degrees of freedom, and the reward function cannot consider the impact of spatial constraints. Therefore, this paper improves the inverse reinforcement learning method of imitation learning for confined space assembly scenarios by adding action feasibility constraint factors and an improved strategy objective function to optimize the method's adaptability to the assembly environment. While ensuring assembly safety, this process fully leverages the collaborative mechanism between humans and machines to complete precise assembly operations within confined spaces.

[0131] The specific model construction and operation process is as follows:

[0132] (1) The improved imitation learning framework constructed for the assembly scenario is as follows:

[0133] ① Define execution space parameters: Define state space S t This describes the current state of the robotic arm and the assembly environment, and introduces a cabin space constraint factor S based on the assembly environment. limit It can be represented by the following formula:

[0134]

[0135] Among them, V free V is the operable space volume. total This refers to the total volume of the cabin space.

[0136] State space S t Multidimensional features are represented by the following formula:

[0137] S t =[P arm V arm ,R arm ,F contact ,ΔP,S limit (19)

[0138] Where P arm V represents the three-dimensional spatial position (x, y, z) of the robotic arm's end effector. arm R is the velocity vector at the end effector of the robotic arm. arm Robotic arm pose (quaternion), F contact Let S be the contact force vector of the force sensor, ΔP be the assembly offset vector, and S be the force vector. limit This is a space-constrained factor.

[0139] Define the motion space A to represent the actions that the robotic arm can perform, including adjustments to joint angles and force feedback intensity, as shown in the following formula:

[0140] A t =[Δθ′1,…,Δθ′6,I v (20)

[0141] Where Δθ' represents the angle adjustment of the i-th joint, I v This is a parameter for adjusting the intensity of tactile feedback, used to adjust the strength of the tactile feedback.

[0142] ② Collect execution operation data: Collect data on the assembly tasks performed by the operator remotely controlling the robotic arm through the touch device. The system records the operator's control actions (such as joint angles and motion trajectories) and environmental data (such as force sensor and visual data) in real time. The collected position, force, touch feedback commands and AR guidance data status are processed in chronological order. This processed data is used as the training set for imitation learning. Abnormal assembly situations are marked in the dataset to facilitate the model's identification of risk scenarios and optimization strategies.

[0143] ③ Selection of Imitation Learning Algorithm: Standard inverse reinforcement learning is based only on the shortest path reward for completing the task, ignoring factors such as operational comfort and space constraints. Therefore, improved inverse reinforcement learning is used for training. Based on the assembly scene information, multidimensional reward factors are introduced as shown in the following formula (21). For the reward weight coefficients α, β, γ, δ, and λ involved, an adaptive adjustment is achieved based on a meta-learning strategy. Specifically, the MAML (Model-Agnostic Meta-Learning) algorithm is used to find a set of optimal initial weight adjustment strategies by training on multiple tasks. The weight adjustment process is an additional optimization task, which directly achieves the effect of optimizing the weight coefficients through gradient backpropagation.

[0144] R(s,a)=-α||ΔP||-βT adj -γ(1-S limit )+δS user -λE effort (twenty one)

[0145] Where ||ΔP|| represents the spatial location residual, used to encourage error reduction; the spatial location data used in this process is the spatial location state collected before Kalman filtering prediction; T adj Indicates the time required for action adjustment, used to encourage increased efficiency; S limit The space constraint factor is used to encourage prioritizing the use of free space for adjustments; S user This represents the human-computer interaction score, reflecting the quality and adaptability of the human-computer interaction in conjunction with the scenario; E effort The system's motion intensity is used to characterize the energy consumption during assembly. Additionally, the weighting coefficient α is an error penalty coefficient, indicating the system's sensitivity to assembly position deviations; a larger value indicates a greater focus on precision. The weighting coefficient β is a motion adjustment penalty, representing the magnitude of posture adjustment during the motion; a larger value indicates a greater tendency to assemble correctly in one go. The weighting coefficient γ is a constraint coefficient, reflecting whether the system is within the error tolerance range; a larger value indicates a more severe penalty for exceeding the limit. The weighting coefficient δ is the user rating weight, representing the user's satisfaction with touch feedback in human-machine collaboration; a larger value indicates that the system's learning is biased towards human preferences. The weighting coefficient λ is the energy consumption penalty, representing the energy consumption corresponding to the assembly motion; a larger value indicates a greater tendency to conserve resources.

[0146] Furthermore, based on the assembly scenario, it is necessary to directly impose negative reward penalties on infeasible actions during the strategy optimization process. Therefore, an action feasibility constraint factor is introduced, as shown in the following formula:

[0147]

[0148] The value is 1 if action a does not violate spatial constraints, and 0 otherwise.

[0149] ④ Training the Imitation Learning Model: Train the improved imitation learning model using the collected dataset. This model's advantages lie not only in its adaptive dynamic balancing of assembly accuracy, speed, human-machine comfort, and space constraints, but also in its ability to obtain weights from the data for optimization rather than manual tuning. The specific training process is as follows: First, through the reward function network discriminator D... Φ The training and recovery reward function R(s,a) is given by inputting the feature vectors of state s and action a, and outputting D. Φ (s,a)∈[0,1], determine whether the state-action pair comes from an expert demonstration, in the discriminator D Φ The output of (s,a) contains the explicit fusion action feasibility judgment result Feasibility(a), which is improved to D. Φ (s,a)=σ(f Φ (s,a))·Feasibility(a); Secondly, the policy network learns the policy π that yields the maximum reward. θ (a|s), input the current state s, output the distribution of action a, including joint angle adjustment and touch feedback intensity; finally, a value function network V is used. Φ (s) Optimize the strategy.

[0150] ⑤ Model Validation and Adjustment: Verify the residual accuracy ||ΔP|| of the imitation learning model in the test environment to be less than 0.5 mm, ensuring that the robotic arm can accurately imitate the operator's actions and adapt to the errors of the assembly environment.

[0151] (2) Based on the trained improved imitation learning model, the established touch feedback adjustment mechanism is used to perform touch dynamic feedback operation, and a touch feedback adjustment factor I is introduced. v The real-time feedback intensity of a touch device is controlled by the following formula:

[0152]

[0153] in, This represents the rate of change of error and is used to enhance dynamic feedback.

[0154] During dynamic touch compensation operations, the touch feedback intensity needs to be adjusted according to the magnitude of assembly errors. This adjustment mechanism enhances the operator's perception of changes in assembly trends, ensuring they can detect subtle differences during assembly. The feedback intensity is used to determine the accuracy of compensation, while avoiding excessively strong or weak feedback. (Regarding feedback intensity I...) v A graded touch feedback adjustment mode is defined, when the feedback intensity I... v When the assembly error ΔP is less than 0.5 mm, the error trend is stable, and the space constraint S ∈ [0, 1.0]. limitFor advanced systems, the feedback mode is slight vibration; when the feedback intensity I... v When the assembly error ΔP is greater than 0.5 mm but less than 1 mm, ∈ [1.0, 2.0], the error trend increases, and the space constraint S limit The feedback mode is medium intensity, providing directional cues; when the feedback intensity is I... v When the assembly error ΔP is greater than 1 mm, the error changes rapidly, and the space constraint S limit It is at a low level, and the feedback mode is strong vibration.

[0155] (3) When the assembly error ΔP is less than 0.5 mm, the assembly is complete; when the assembly error ΔP is still greater than 0.5 mm, continue to repeat the above operation process (2) until the assembly is completed.

[0156] Step 5: Establish an evaluation model for assembly error compensation effect

[0157] The established assembly error compensation effect evaluation model aims to ensure that assembly errors have been eliminated or reduced to an acceptable range after error compensation, thus ensuring precise assembly. The assembly error compensation effect evaluation indicators mainly include positional error, posture error, and assembly clearance error. After each compensation, the model automatically calculates these three error indicators, determines whether the assembly accuracy meets the standards, and displays the error changes before and after compensation. The specific process for evaluating the assembly error compensation effect is as follows:

[0158] (1) Each time the operator performs fine-tuning error compensation by controlling the robotic arm via touch, the system re-collects error data through sensors and updates the error analysis model data.

[0159] (2) The calculated results of three evaluation indicators—position error, attitude error, and assembly clearance error—are compared with the set tolerance thresholds. When the evaluation result is less than the set tolerance threshold, the assembly alignment is considered successful. The relevant definitions and calculations of the evaluation indicators are as follows:

[0160] Positional error refers to the deviation between the actual position and the target position of an assembled component. This error is usually measured by Euclidean distance in three-dimensional space, and the calculation formula is shown below:

[0161]

[0162] Where, x actual y actual , z actual Indicates the actual location of the assembly, x target y target , z target Indicates the target location.

[0163] Attitude error refers to the difference between the orientation of an assembled part and the target orientation. It is usually described using quaternions, and the calculation formula is shown below:

[0164]

[0165] Among them, R actual R represents the rotation matrix of the actual assembly. target The rotation matrix represents the target assembly position.

[0166] Assembly clearance error inspection refers to whether there are gaps between components that do not meet the requirements. It is obtained by calculating the distance in two directions, axial clearance and radial clearance, using a vision sensor.

[0167] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A human-computer collaborative assembly method based on augmented reality and force feedback, characterized in that, include: S1, A vision sensor and a force sensor are installed at the end of the robotic arm. The vision sensor is used to capture a 3D image of the target assembly area, and the force sensor is used to measure the force and torque of the robotic arm. Operators operate the robotic arm through touch devices and wear AR devices to project the CAD assembly reference datum into the AR field of view to form a virtual outline of the assembly target and extract virtual assembly points using AR augmented reality technology. S2, extract the position and outline of the target in the 3D image of the visual sensor, and compare it with the virtual outline and virtual assembly point of the assembly target position in AR augmented reality to obtain the assembly error vector; generate a guide vector arrow at the target position extracted by the visual sensor in the AR field of view, the direction of the guide vector arrow is the direction of the assembly error vector, and the color of the vector arrow indicates the magnitude of the assembly error vector. S3, the robotic arm imitates the operator's actions based on the imitation learning model, compensates for the perceived assembly error under the guidance of the guide vector arrow, and uses the touch feedback adjustment mechanism to judge whether the compensation is correct; among them, Kalman filtering is used to predict and update the direction and color of the guide vector arrow at key assembly points, and the assembly alignment is determined based on the assembly error compensation effect evaluation model. The imitation learning model employs an improved inverse reinforcement learning approach, wherein the state space... S t for: in, P arm The three-dimensional spatial position of the end effector of the robotic arm ( x , y , z ), V arm The velocity vector at the end of the robotic arm. R arm For the robotic arm posture, F contact The force vector is the contact force vector of the force sensor. ΔP This is the offset vector of the assembly error vector. S limit As a spatially constrained factor, , V free For the operable space volume, V total This refers to the total volume of the cabin space; Action space A for: in, Δθ i ′ for the first i Adjustment of joint angles I v This is a parameter for adjusting the intensity of touch feedback; The reward function is Among them, ||Δ P || represents the spatial location residual; T adj Indicates the time taken for action adjustment; S limit Indicates the space constraint factor; S user Indicates the human-computer interaction rating; E effort Indicates the intensity of the robotic arm's motion; α , β , γ , δ , λ These are the weighting coefficients; With optimal weighting coefficients α , β , γ , δ , λ To optimize the strategy and complete the training of the imitation learning model, an action feasibility constraint factor is introduced during the strategy optimization process. Negative reward penalties are imposed on infeasible actions. The action feasibility constraint factor is as follows: Among them, if the action a The value is 1 if the spatial constraints are not violated, and 0 otherwise. The correctness of the compensation is determined using a touch feedback adjustment mechanism, specifically as follows: Calculate touch feedback intensity I v : in, Indicates the rate of change of assembly error; ω This indicates the magnitude of the absolute error; the larger the value, the more severe the error and the stronger the feedback. ζ Indicates the degree of approaching the limit, used to reflect the level of danger; τ This indicates the trend of error growth; the faster the trend, the earlier the warning should be issued, and the stronger the feedback needs to be. Based on the intensity of touch feedback I v The size of the value determines whether the compensation is correct.

2. The method as described in claim 1, characterized in that, In S3, the system observation equation for the Kalman filter is: Wherein, the state vector ,in, This is the assembly error value calculated by the current system. The rate of change of the error value; the observation vector. , This represents the current actual assembly error value obtained from sensor observations, including data from vision sensors, force sensors, and the robotic arm's end effector; observation noise. v t With a mean of 0 and an observation noise covariance of , R Gaussian white noise; Observation matrix H for: Observation noise covariance R for: in, , , This is an estimate of the variance of the sensor observation data; Prediction phase: Update phase: in, This represents the state covariance prediction matrix at the previous time step; The Kalman gain matrix; F Here is the state transition matrix: Q For the system noise covariance: Δ t The system sampling period is I 3 represents a third-order identity matrix.

3. The method as described in claim 1, characterized in that, Employs a graded touch feedback adjustment mode: when the touch feedback intensity... I v ∈[0, 1.0], assembly error Δ P Less than or equal to 0.5 mm, with a stable error trend and limited space. S limit When the level is high, the feedback mode is slight vibration; when the feedback intensity is high... I v ∈(1.0, 2.0], assembly error Δ P For diameters greater than 0.5 mm but less than or equal to 1 mm, the error trend increases, leading to greater space constraints. S limit When the value is medium, the feedback mode is of medium intensity and directional prompts are provided; when the feedback intensity is... I v Greater than 2.0, assembly error Δ P When the error exceeds 1 mm, the error rate increases rapidly, and the space becomes more limited. S limit When the setting is low, the feedback mode is strong vibration.

4. The method as described in claim 1 or 3, characterized in that, When assembly error Δ P Assembly is complete when the error is less than or equal to 0.5 mm; when the assembly error Δ... P If the difference is greater than 0.5 mm, continue to repeat the touch dynamic compensation operation until the assembly is completed.

5. The method as described in claim 1, characterized in that, In step S3, the assembly error compensation effect evaluation model calculates the position error, attitude error, and assembly gap error, and compares the calculation results with a set tolerance threshold. If all are less than the set tolerance threshold, the assembly alignment is determined to be successful. Among them, the position error is the deviation between the actual position of the assembled part and the target position; the attitude error is the deviation between the orientation of the assembled part and the target orientation; and the assembly gap error is the gap between the parts, which is obtained by detecting the axial gap and radial gap by a vision sensor.

6. The method as described in claim 1, characterized in that, S2 is specifically as follows: The assembly area is divided into voxel grids; Using center point extraction image processing technology, the actual position of the target center point in the 3D image of the vision sensor is extracted; combined with the CAD model and assembly task requirements, the virtual position of the target center point is calculated through spatial registration transformation. Calculate the assembly error vector based on the virtual and actual positions of the target center point. ; Each sampling point error value is assigned to the corresponding voxel grid, and the unsampled areas are completed by Gaussian regression. Calculate the magnitude of the assembly error vector And map the magnitude to a color gradient; Draw a guide vector arrow at the actual target location, where the arrow direction is the assembly error vector direction, clearly indicating left, right, up, down, rotation, or advance icons; the arrow length is set to... ,in k To adjust the scaling factor; the arrow color is the color mapped to the modulus; a fixed refresh rate is set to update the guide vector arrow; The drawn guide vector arrows are mapped onto the AR field of view, and a heat map of spatial error distribution is displayed in real time in the operator's AR field of view.

7. The method as described in claim 6, characterized in that, The calculated assembly error vector magnitude Presented as a color gradient: when A value greater than 3 mm will be displayed in red, indicating a high error; when... An error greater than 1 mm but less than or equal to 3 mm is indicated by an orange color, signifying a low error. When the diameter is less than or equal to 1 mm, it will be displayed in green, indicating that it is within the safe range.

8. A human-machine collaborative assembly system based on augmented reality and force feedback, characterized in that, It includes robotic arms, touch devices, AR devices, vision sensors, and force sensors, and performs human-machine collaborative assembly using the method described in any one of claims 1 to 7.

Images