A bolt tightening method for construction robots based on vision and force feedback

Abstract

The present invention discloses a bolt tightening method for a construction robot based on vision and force feedback, and relates to the field of intelligent construction technology. The present invention comprises at least the following steps: S1: using Mamba's conditional variational autoencoder, by feature embedding the posture sequence, action sequence and end force feedback of the manipulator, encoding the mean and variance of the conditional distribution, and generating the action sequence for the next step; S2: setting the condition of the trained conditional autoencoder decoder to 0, using the current observation image and posture of the manipulator as input, predicting the action at the next moment, the end force feedback and the end distance from the ground. The present invention proposes a bolt tightening method for a construction robot with vision-force feedback, which can sense subtle changes in force during the tightening process and adjust in real time, thereby improving the robot's operating accuracy and adaptability to complex tasks; at the same time, after the present invention is promoted and applied, it can reduce the labor intensity of workers and reduce the occurrence of accidents.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent construction, and in particular to a bolt tightening method for a construction robot based on vision and force feedback. Background Art

[0002] High-strength bolt connections can be used for web connections, flange connections of steel sections, and flange connections of steel pipes. They are widely used in traditional industries and construction.

[0003] Generally, the tightening of high-strength bolt connections is divided into initial tightening, re-tightening, and final tightening, which must be completed within a day. Currently, tightening is usually done manually or with construction tools.

[0004] However, due to human factors, construction tool matching and on-site construction conditions, some high-strength bolts are under-tightened or over-tightened, the overall reliability deviation of the nodes is large, and the construction efficiency is low, making it difficult to ensure the connection quality.

[0005] Existing robots based on reinforcement learning and imitation learning often rely solely on vision or robot pose data for learning, making it difficult to learn the force and gestures required for bolt tightening tasks. Furthermore, the lack of force feedback prevents the robot from accurately sensing the applied force during operation, making it difficult to adjust its force and position. This can easily lead to structural damage or insufficient bolt tightening, compromising construction quality and operational safety.

[0006] Therefore, a new solution to the above problems needs to be proposed. Summary of the Invention

[0007] The purpose of the present invention is to provide a bolt tightening method for a construction robot based on vision and force feedback to solve the technical problems raised in the background technology.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a bolt tightening method for a construction robot based on vision and force feedback, comprising at least the following steps:

[0009] S1: Using Mamba’s conditional variational autoencoder, we embed the robot’s posture sequence, action sequence, and end-point force feedback into features, encode the mean and variance of the conditional distribution, and generate the next action sequence.

[0010] S2: Set the condition of the trained conditional autoencoder decoder to 0, use the current observation image and posture of the robot arm as input, and predict the next moment's action, end force feedback, and end distance from the ground;

[0011] S3: When the end-to-ground distance is valid, the predicted end-to-ground force feedback is used as the target control input, combined with the admittance controller of the manipulator to control the manipulator to complete the fastening operation.

[0012] 5. The method for tightening bolts by a construction robot based on vision and force feedback according to claim 1, wherein said S1 comprises at least the following steps:

[0013] Encoder of Conditional Variational Autoencoder Fit the distribution of current actions and states;

[0014] Assuming the conditional distribution is normal, the posture sequence of the robot arm With action sequence A t =(a t ,…,a t+k ) are respectively embedded with features, where J t is the robot posture from time t to time t+k, F t is the end force feedback (force and torque) of the robot from time t to time t+k, G t The distance from the end to the ground from time t to time t+k, used to mark the start and end;

[0015] The feature embedding is further encoded through the Mamba module to obtain the mean μ and variance δ of the conditional distribution, denoted as z = (μ, δ), see the following formula:

[0016]

[0017] Decoder of Conditional Variational Autoencoder Generate the next robotic arm action and feedback force, the current observation image N is the number of cameras, which is encoded into a feature vector through the feature extraction network ResNet18 and the current posture The embedded features and conditional distributions are spliced ​​and passed through the encoding layer composed of Mamba modules. enc and decoding layer Mamba dec Get the generated action sequence, see the following formula:

[0018]

[0019] The loss function of the conditional variational autoencoder is as follows:

[0020]

[0021] Where MSE represents the root mean square loss; W is the Wasserstein metric.

[0022] 6. The method for tightening bolts by a construction robot based on vision and force feedback according to claim 2, wherein said S2 comprises at least the following steps:

[0023] Set the condition of the decoder of the trained conditional autoencoder to 0, and use the current observation image and posture of the robot arm as the input of the decoder. The robot arm action a at the next moment t , end force feedback f t , distance from the end to the ground g t It can be determined by the following formula:

[0024]

[0025] in, is the predicted action at time t+i, is the predicted terminal geodesic distance at time t+i, is the predicted end force feedback at time t+i, m is the predefined factor of the control weight term exp(-m×i), and K is the predicted sequence length.

[0026] 7. The bolt tightening method of a construction robot based on vision and force feedback according to claim 3, wherein said S3 at least comprises the following steps:

[0027] When the distance from the end to the ground is g t When valid, the end force feedback f predicted by the conditional autoencoder is t As the target control, input the following into the robot arm admittance controller to control the robot arm to complete the fastening operation, refer to the following formula:

[0028]

[0029] Where x is the position signal, and are the first and second derivatives of the position signal, M, B, K i are the mass, stiffness and drag coefficient of the robot, f ext is the external force, and t′ is the integral variable.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] This invention proposes a construction robot bolt tightening method using vision and force feedback. This method can sense subtle changes in force during the tightening process and adjust in real time, improving the robot's operational precision and adaptability to complex tasks. Furthermore, widespread application of this method could reduce worker labor intensity and accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 This is a network block diagram of the present invention;

[0035] Figure 3 This is a schematic diagram of the Mamba layer of the present invention;

[0036] Figure 4 Schematic diagram of the bolt tightening process of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] See also Figure 1 - Figure 4 ,A bolt tightening method for a construction robot based on vision and force feedback;

[0039] In the first step, the proposed method uses Mamba's conditional variational encoder as follows: Figure 2 shown.

[0040] Encoder of Conditional Variational Autoencoder Fit the distribution of the current action and state; assuming the conditional distribution is normal distribution, the posture sequence of the robot arm With action sequence A t =(a t ,…,a t+k ) are respectively embedded with features, where J t is the robot posture from time t to time t+k, F t is the end force feedback (force and torque) of the robot from time t to time t+k, G t is the distance from the end to the ground from time t to time t+k, which is used to mark the beginning and end; the feature embedding is further encoded through the Mamba module to obtain the mean μ and variance δ of the conditional distribution, denoted as z = (μ, δ).

[0041]

[0042] Decoder of Conditional Variational Autoencoder Generate the next robotic arm action and feedback force, the current observation image N is the number of cameras, which is encoded into a feature vector through the feature extraction network ResNet18 and the current posture The embedded features and conditional distributions are spliced ​​and passed through the encoding layer composed of Mamba modules. enc and decoding layer Mamba dec Get the generated action sequence.

[0043]

[0044] The loss function of the conditional variational autoencoder is as follows:

[0045]

[0046] Where MSE represents the root mean square loss and W is the Wasserstein metric.

[0047] In the second step, the condition of the decoder of the trained conditional autoencoder is set to 0, and the observation image and posture of the current robot arm are used as the input of the decoder. The robot arm action a at the next moment is t , end force feedback f t , distance from the end to the ground g t It can be determined by the following formula:

[0048]

[0049] in, is the predicted action at time t+i, is the predicted terminal geodesic distance at time t+i, is the predicted end force feedback at time t+i, m is the predefined factor of the control weight term exp(-m×i), and K is the predicted sequence length.

[0050] Step 3: When the end is at a distance g from the ground t When valid, the end force feedback f predicted by the conditional autoencoder is t As the target control, input the following into the robot arm admittance controller to control the robot arm to complete the fastening operation, refer to the following formula:

[0051]

[0052] Where x is the position signal, and are the first and second derivatives of the position signal, M, B, K i are the mass, stiffness and drag coefficient of the robot, f ext is the external force, and t′ is the integral variable.

[0053] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A bolt tightening method for a construction robot based on vision and force feedback, characterized by: At least the following steps are included: S1: Using Mamba’s conditional variational autoencoder, we embed the robot’s posture sequence, action sequence, and end-point force feedback into features, encode the mean and variance of the conditional distribution, and generate the next action sequence. Said S1 at least comprises the following steps: Encoder of Conditional Variational Autoencoder Fit the distribution of current actions and states; Assuming the conditional distribution is normal, the posture sequence of the robot arm and action sequences After feature embedding, is the robot posture from time t to time t+k, is the end force feedback of the robot from time t to t+k, The distance from the end to the ground from time t to time t+k, used to mark the start and end; The feature embedding is further encoded through the Mamba module to obtain the mean of the conditional distribution and variance , denoted as z , see the following formula: ； Decoder of Conditional Variational Autoencoder Generate the next robotic arm action and feedback force, the current observation image , N is the number of cameras, which is encoded into a feature vector through the feature extraction network ResNet18 and the current posture The embedded features and conditional distribution are spliced ​​and passed through the encoding layer composed of Mamba modules and decoding layer Get the generated action sequence, see the following formula: ； The loss function of the conditional variational autoencoder is as follows: ； in represents the root mean square loss; W is the Wasserstein metric S2: Set the condition of the trained conditional variational autoencoder decoder to 0, use the current observation image and posture of the robot arm as input, and predict the next moment's action, end-point force feedback, and end-point distance from the ground; S3: When the end-to-ground distance is valid, the predicted end-to-ground force feedback is used as the target control input, combined with the admittance controller of the manipulator to control the manipulator to complete the fastening operation.

2. The bolt tightening method of a construction robot based on vision and force feedback according to claim 1, characterized in that: Said S2 at least comprises the following steps: The condition of the decoder of the trained conditional variational autoencoder is set to 0, and the observation image and posture of the current manipulator are used as the input of the decoder. The manipulator action at the next moment is , end force feedback , distance from the end to the ground It can be determined by the following formula: ； ； ； in: is the predicted action at time t+i, is the predicted terminal geodesic distance at time t+i, is the predicted end force feedback at time t+i, and m is the control weight term is a predefined factor of , and K is the predicted sequence length.

3. The bolt tightening method of a construction robot based on vision and force feedback according to claim 2, characterized in that: The S3 at least includes the following steps: When the end is at a distance from the ground When valid, the end force feedback predicted by the conditional variational autoencoder As the target control, input the following into the robot arm admittance controller to control the robot arm to complete the fastening operation, refer to the following formula: ； in is the position signal, and are the first-order derivative and second-order derivative of the position signal, , B, are the mass, stiffness and drag coefficient of the robot respectively, For external force, is the integration variable.

Images