Building robot bolt fastening method based on vision and force feedback

Abstract

The invention discloses a building robot bolt fastening method based on vision and force feedback, and relates to the technical field of intelligent construction. The method at least comprises the following steps that S1, a conditional variation auto-encoder of Mamba is utilized, a posture sequence, an action sequence and tail end force feedback of the mechanical arm are embedded through features, the mean value and the variance of conditional distribution are obtained through encoding, and an action sequence of the next step is generated; s2, the condition of a trained conditional auto-encoder decoder is set to be 0, and the action, the tail end force feedback and the tail end ground distance at the next moment are predicted by using the observation image and the posture of the current mechanical arm as input. According to the building robot bolt fastening method based on vision-force feedback, fine stress changes in the fastening process can be sensed and adjusted in real time, and the operation precision of a robot and the adaptation degree of the robot to complex tasks are improved; meanwhile, after the device is popularized and applied, the labor intensity of workers can be reduced, and accidents can be reduced.

Description

Technical Field

[0001] The invention relates to the technical field of intelligent construction, and in particular to a bolt tightening method of a construction robot based on vision and force feedback. Background Art

[0002] High-strength bolt connections can be used for web connections, flange connections of steel sections, and flange connections of steel pipes. They are widely used in traditional industries and construction.

[0003] Generally, the tightening of high-strength bolt connections is divided into initial tightening, re-tightening and final tightening, which are required to be completed within one day. Currently, manual tightening or construction tools are often used for tightening.

[0004] However, due to human factors, construction tool matching and on-site construction conditions, some high-strength bolts are under-tightened or over-tightened, the overall reliability deviation of the nodes is large, and the construction efficiency is low, making it difficult to ensure the connection quality.

[0005] Existing robots based on reinforcement learning and imitation learning often only use vision or robot posture data for learning, which makes it difficult to learn the force and gestures required for bolt tightening tasks. At the same time, the lack of force feedback will cause the robot to be unable to accurately perceive the applied force during operation, making it difficult to adjust its own strength and position. This situation can easily lead to structural damage or insufficient bolt tightening, affecting construction quality and operational safety.

[0006] Therefore, a new solution to the above problems needs to be proposed. Summary of the invention

[0007] The purpose of the present invention is to provide a bolt tightening method for a construction robot based on vision and force feedback to solve the technical problems raised in the background technology.

[0008] To achieve the above object, the present invention provides the following technical solution: a bolt tightening method of a construction robot based on vision and force feedback, comprising at least the following steps:

[0009] S1: Using Mamba’s conditional variational autoencoder, we embed the robot’s posture sequence, action sequence, and end force feedback into features, encode the mean and variance of the conditional distribution, and generate the next action sequence.

[0010] S2: Set the condition of the trained conditional autoencoder decoder to 0, use the current observation image and posture of the robot arm as input, and predict the action, end force feedback, and end distance from the ground at the next moment;

[0011] S3: When the distance between the end and the ground is valid, the predicted end force feedback is used as the target control input, combined with the admittance controller of the robot arm, to control the robot arm to complete the fastening operation.

[0012] 5. The bolt tightening method of a construction robot based on vision and force feedback according to claim 1, characterized in that: said S1 at least comprises the following steps:

[0013] Encoder of Conditional Variational Autoencoder Fit the distribution of current actions and states;

[0014] Assuming the conditional distribution is normal, the posture sequence of the robot arm With action sequence A t =(a t ,…,a t+k ) are respectively embedded with features, where J t is the robot posture from time t to time t+k, F t is the end force feedback (force and torque) of the robot from time t to time t+k, G t It is the distance from the end to the ground from time t to time t+k, used to mark the beginning and the end;

[0015] The feature embedding is further encoded through the Mamba module to obtain the mean μ and variance δ of the conditional distribution, denoted as z = (μ, δ), see the following formula:

[0016]

[0017] Decoder of Conditional Variational Autoencoder Generate the next robot arm action and feedback force, the current observation image N is the number of cameras, which is encoded into a feature vector through the feature extraction network ResNet18 and the current posture The embedded features and conditional distribution are concatenated and passed through the encoding layer composed of Mamba modules. enc and decoding layer Mamba dec Get the generated action sequence, see the following formula:

[0018]

[0019] The loss function of the conditional variational autoencoder is as follows:

[0020]

[0021] Where MSE represents the root mean square loss; W is the Wasserstein metric.

[0022] 6. The bolt tightening method of a construction robot based on vision and force feedback according to claim 2, characterized in that: said S2 at least comprises the following steps:

[0023] Set the condition of the decoder of the trained conditional autoencoder to 0, and use the current observation image and posture of the robot as the input of the decoder. The robot action a at the next moment t , end force feedback f t , distance from the end to the ground g t It can be determined by the following formula:

[0024]

[0025] in, is the predicted action at time t+i, is the predicted terminal geodesic distance at time t+i, is the predicted end force feedback at the t+ith moment, m is the predefined factor of the control weight term exp(-m×i), and K is the predicted sequence length.

[0026] 7. The bolt tightening method of a construction robot based on vision and force feedback according to claim 3, characterized in that: said S3 at least comprises the following steps:

[0027] When the end is at a distance g from the ground t When valid, the end force feedback f predicted by the conditional autoencoder is t As the target control, input the following into the robot admittance controller to control the robot to complete the fastening operation, refer to the following formula:

[0028]

[0029] Where x is the position signal, and are the first and second derivatives of the position signal, M, B, K i are the mass, stiffness and drag coefficient of the robot, respectively, ext is the external force, and t′ is the integral variable.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] The present invention proposes a visual-force feedback bolt tightening method for a construction robot, which can sense subtle changes in force during the tightening process and make real-time adjustments, thereby improving the robot's operating accuracy and adaptability to complex tasks. At the same time, after the present invention is promoted and applied, it can reduce the labor intensity of workers and reduce the occurrence of accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0033] Figure 1 It is a schematic diagram of the process of the present invention;

[0034] Figure 2 This is a network block diagram of the present invention;

[0035] Figure 3 This is a schematic diagram of the Mamba layer of the present invention;

[0036] Figure 4 It is a schematic diagram of the bolt tightening process of the present invention. DETAILED DESCRIPTION

[0037] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0038] See also Figure 1 - Figure 4 ,A bolt tightening method for construction robots based on vision and force feedback;

[0039] In the first step, the proposed method uses Mamba's conditional variational encoder as follows: Figure 2 shown.

[0040] Encoder of Conditional Variational Autoencoder Fit the distribution of the current action and state; assume that the conditional distribution is a normal distribution, and the posture sequence of the robot arm With action sequence A t =(a t ,…,a t+k ) are respectively embedded with features, where J t is the robot posture from time t to time t+k, F t is the end force feedback (force and torque) of the robot from time t to time t+k, G t is the distance from the end to the ground from time t to time t+k, which is used to mark the beginning and the end. The feature embedding is further encoded through the Mamba module to obtain the mean μ and variance δ of the conditional distribution, denoted as z=(μ,δ).

[0041]

[0042] Decoder of Conditional Variational Autoencoder Generate the next robot arm action and feedback force, the current observation image N is the number of cameras, which is encoded into a feature vector through the feature extraction network ResNet18 and the current posture The embedded features and conditional distribution are concatenated and passed through the encoding layer composed of Mamba modules. enc and decoding layer Mamba dec Get the generated action sequence.

[0043]

[0044] The loss function of the conditional variational autoencoder is as follows:

[0045]

[0046] Where MSE represents the root mean square loss and W is the Wasserstein metric.

[0047] In the second step, the condition of the decoder of the trained conditional autoencoder is set to 0, and the observation image and posture of the current robot arm are used as the input of the decoder. t , end force feedback f t , distance from the end to the ground g t It can be determined by the following formula:

[0048]

[0049] in, is the predicted action at time t+i, is the predicted terminal geodesic distance at time t+i, is the predicted end force feedback at the t+ith moment, m is the predefined factor of the control weight term exp(-m×i), and K is the predicted sequence length.

[0050] Step 3: When the end is at a distance g from the ground t When valid, the end force feedback f predicted by the conditional autoencoder is t As the target control, input the following into the robot admittance controller to control the robot to complete the fastening operation, refer to the following formula:

[0051]

[0052] Where x is the position signal, and are the first and second derivatives of the position signal, M, B, K i are the mass, stiffness and drag coefficient of the robot, respectively, ext is the external force, and t′ is the integral variable.

[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A bolt tightening method for a construction robot based on vision and force feedback, characterized in that: At least the following steps are included: S1: Using Mamba’s conditional variational autoencoder, we embed the robot’s posture sequence, action sequence, and end force feedback into features, encode the mean and variance of the conditional distribution, and generate the next action sequence. S2: Set the condition of the trained conditional autoencoder decoder to 0, use the current observation image and posture of the robot arm as input, and predict the action, end force feedback and end distance to the ground at the next moment; S3: When the distance between the end and the ground is valid, the predicted end force feedback is used as the target control input, combined with the admittance controller of the robot arm, to control the robot arm to complete the fastening operation.

2. The bolt tightening method of a construction robot based on vision and force feedback according to claim 1, characterized in that: The S1 at least comprises the following steps: Encoder of Conditional Variational Autoencoder Fit the distribution of current actions and states; Assuming the conditional distribution is normal, the posture sequence of the robot arm With action sequence A t =(a t ,…,a t+k ) are respectively embedded with features, where J t is the robot posture from time t to time t+k, F t is the end force feedback (force and torque) of the robot from time t to time t+k, G t It is the distance from the end to the ground from time t to time t+k, used to mark the beginning and the end; The feature embedding is further encoded through the Mamba module to obtain the mean μ and variance δ of the conditional distribution, denoted as z = (μ, δ), see the following formula: Decoder of Conditional Variational Autoencoder Generate the next robot arm action and feedback force, the current observation image N is the number of cameras, which is encoded into a feature vector through the feature extraction network ResNet18 and the current posture The embedded features and conditional distribution are concatenated and passed through the encoding layer composed of Mamba modules. enc and decoding layer Mamba dec Get the generated action sequence, see the following formula: The loss function of the conditional variational autoencoder is as follows: Where MSE represents the root mean square loss; W is the Wasserstein metric.

3. The bolt tightening method of a construction robot based on vision and force feedback according to claim 2 is characterized in that: The S2 at least comprises the following steps: Set the condition of the decoder of the trained conditional autoencoder to 0, and use the current observation image and posture of the robot as the input of the decoder. The robot action a at the next moment t , end force feedback f t , distance from the end to the ground g t It can be determined by the following formula: in: is the predicted action at time t+i, is the predicted terminal geodesic distance at time t+i, is the predicted end force feedback at the t+ith moment, m is the predefined factor of the control weight term exp(-m×i), and K is the predicted sequence length.

4. The bolt tightening method of a construction robot based on vision and force feedback according to claim 3 is characterized in that: The S3 at least includes the following steps: When the end is at a distance g from the ground t When valid, the end force feedback f predicted by the conditional autoencoder is t As the target control, input the following into the robot admittance controller to control the robot to complete the fastening operation, refer to the following formula: Where x is the position signal, and are the first and second derivatives of the position signal, M, B, K i are the mass, stiffness and drag coefficient of the robot, respectively, ext is the external force, and t′ is the integral variable.

Images