A bolt tightening method based on a vision-language-action model

Abstract

The application discloses a bolt tightening method based on a vision-language-action model, and belongs to the technical field of intelligent manufacturing. The method comprises the following steps: collecting scene image data of a target bolt and a workpiece region around the target bolt, extracting multi-scale visual features from the collected scene image data, and obtaining a multi-scale visual feature vector Fv; performing semantic analysis on a natural language instruction to obtain a language feature vector Fl; performing cross-modal feature alignment on the Fv and the Fl to obtain an aligned feature vector Fvl; performing action decoding and parameterized mapping processing on the Fvl to generate an action parameter vector Fd; inputting the Fd into a control execution module to perform kinematics solving and control instruction generation, so as to drive an automated execution device to complete a bolt tightening operation. The method can improve the accuracy, adaptability and stability of bolt tightening operation under complex working conditions, and effectively reduce the labor cost of bolt tightening work.

Description

Technical Field

[0001] This invention relates to the field of intelligent manufacturing technology, specifically to a bolt tightening method based on a vision-language-action model. Background Technology

[0002] Bolt tightening is one of the most common and critical operations in machinery manufacturing, assembly, and equipment maintenance. The quality of tightening directly affects the safety, reliability, and service life of products. In industrial settings, bolts are typically located in different places, and there are various factors such as workpiece obstruction, changes in lighting, assembly deviations, and interference from multiple targets around the bolts. Therefore, bolt tightening operations not only require accurate target positioning but also demand high adaptability and stability in the execution process.

[0003] Existing tightening methods mainly include two categories: manual tightening and automated tightening controlled by preset programs via robot teaching. Manual tightening relies on operator experience and suffers from high labor intensity, low efficiency, poor quality consistency, and large torque control dispersion. Automated tightening controlled by preset programs via robot teaching typically involves pre-setting the target position, tightening sequence, and torque parameters, which are then executed by automated execution equipment such as industrial robotic arms, linear modules, gantry mechanisms, or electric tightening devices. While robot teaching can achieve high efficiency under conditions of a single workstation, single bolt size, and fixed bolt posture, it often requires re-teaching, recalibration, and manual parameter modification when bolt position shifts, bolt orientation changes, or multiple target bolts exist. This makes it difficult to adapt to the dynamic changes required in flexible manufacturing scenarios. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a bolt tightening method based on a vision-language-action model, aiming to improve the accuracy, adaptability, and stability of bolt tightening operations under complex working conditions, and effectively reduce the labor costs of bolt tightening work.

[0005] The technical solution of this invention is:

[0006] A bolt tightening method based on a vision-language-action model, the method comprising:

[0007] Collect scene image data of the target bolt and its surrounding workpiece area;

[0008] Multi-scale visual features are extracted from the collected scene image data to obtain the multi-scale visual feature vector Fv;

[0009] Semantic parsing is performed on the natural language commands input by the user, converting the natural language commands from text descriptions into computable semantic representations, and obtaining a language feature vector Fl; the language feature vector Fl is used to characterize the semantic description information, spatial orientation information, task execution order information, and task constraint information of the bolt;

[0010] Cross-modal feature alignment is performed on Fv and Fl using the visual-language feature alignment module to obtain the aligned feature vector Fvl;

[0011] The action generation module performs action decoding and parameter mapping on the aligned feature vector Fvl to generate the action parameter vector Fd.

[0012] The motion parameter vector Fd is input into the control execution module for kinematic solution and control command generation, thereby driving the automated execution equipment to complete the bolt tightening operation.

[0013] Furthermore, according to the bolt tightening method, multi-scale visual features are extracted from the acquired scene image data by a visual encoder to obtain a multi-scale visual feature vector Fv. The multi-scale visual feature vector Fv includes low-level visual features, mid-level structural features, and high-level semantic features. The low-level visual features are used to characterize the edge, texture, and contour information of the target bolt, the mid-level structural features are used to characterize the local spatial relationship between the target bolt and its surrounding workpieces, and the high-level semantic features are used to characterize the semantic association between the area where the target bolt is located and the global scene.

[0014] Furthermore, according to the bolt tightening method, the visual-language feature alignment module includes:

[0015] The query feature generation unit is used to generate query features based on the language feature vector Fl;

[0016] The key-value feature generation unit is used to generate key features and value features based on the multi-scale visual feature vector Fv.

[0017] The cross-attention calculation unit is used to receive the query features, key features, and value features. Through feature similarity calculation, it calculates the correlation score between the visual content of each image region in the key features and the semantic information of the natural language instruction in the query features. The score is normalized to generate a weight distribution. Then, the value features are weighted according to the weight coefficients in the weight distribution to enhance the visual region features related to the instruction semantics and suppress irrelevant features. Finally, the semantically and visually aligned feature vector Fvl is output.

[0018] Furthermore, according to the bolt tightening method, the query feature generation unit includes a fully connected layer, a normalization layer, and an activation function layer connected in sequence; the fully connected layer is used to receive the language feature vector Fl, perform a linear transformation on Fl, and generate an intermediate representation of the query features; the normalization layer is used to normalize the intermediate representation of the query features to enhance the stability of the distribution of the intermediate representation of the query features; the activation function layer is used to perform a nonlinear transformation on the normalized intermediate representation of the query features to enhance the expressive power of the normalized query features and generate the final query features.

[0019] Furthermore, according to the bolt tightening method, the key-value feature generation unit includes a feature mapping layer, a normalization layer, and an activation function layer connected in sequence; the feature mapping layer is used to receive a multi-scale visual feature vector Fv, perform a linear transformation on Fv, generate an intermediate representation of key-value features, and realize the mapping transformation of Fv to key features and value features; the normalization layer is used to normalize the intermediate representation of key-value features to improve the stability of the distribution of the intermediate representation of key-value features; the activation function layer is used to perform a nonlinear transformation on the normalized intermediate representation of key-value features to enhance the expressive power of the normalized key-value features and generate the final key features and value features.

[0020] Furthermore, according to the bolt tightening method, the feature similarity calculation is used to perform position-by-position matching between the query feature and the key feature. By calculating the similarity between the query feature and the key feature in the feature space, the corresponding relevance score is obtained, specifically through the following formula:

[0021]

[0022] in, This represents the i-th query feature; Represents the j-th key feature; This represents the relevance score between the i-th query feature and the j-th key feature.

[0023] Furthermore, according to the bolt tightening method, the action generation module includes:

[0024] The action decoding unit is used to perform feature parsing on the received aligned feature vector Fvl, output the action category step by step according to the time step, and then generate an action sequence Fa to represent the operation to be performed and its order.

[0025] The parameter mapping unit is used to perform layer-by-layer feature mapping on the action sequence Fa through a multi-layer fully connected network to generate an action parameter vector Fd. The action parameter vector Fd is used to represent the control parameter information when executing each action step, including the pose parameters of the end effector of the industrial robot arm, the switching parameters of the end effector of the industrial robot arm, and the target tightening torque. The pose parameters are used to represent the position and attitude information of the end effector of the industrial robot arm. The switching parameters are used to represent the switching state of the end effector of the industrial robot arm during operation.

[0026] Furthermore, according to the bolt tightening method, the action decoding unit adopts a multilayer perceptron network structure, which includes at least a first fully connected layer, a normalization layer, an activation function layer, a second fully connected layer, and an action classification output layer connected in sequence.

[0027] The first fully connected layer receives the aligned feature vector Fvl and performs a linear mapping on Fvl to obtain intermediate action decoding features. The normalization layer normalizes the intermediate action decoding features. The activation function layer performs a nonlinear transformation on the normalized intermediate action decoding features to enhance feature representation. The second fully connected layer further maps the nonlinearly transformed intermediate action decoding features to output an action decoding feature representation suitable for action classification. The action classification output layer performs a linear mapping on the action decoding feature representation through the fully connected layer to obtain an action category score. Then, it normalizes the action category score using a softmax function to obtain the probability distribution of the action category. Finally, it determines the action category at the current time based on the probability distribution of the action category.

[0028] Furthermore, according to the bolt tightening method, the control execution module includes:

[0029] The inverse kinematics unit is used to perform inverse kinematics calculations based on the pose parameters of the end effector of the industrial robot arm in the motion parameter vector Fd, and to obtain the target joint angles of each joint of the robot arm using an iterative solution method. The specific calculation process includes: first, setting the current joint angle of the industrial robot arm as the initial joint angle vector; then, calculating the current pose of the end effector of the industrial robot arm based on the initial joint angle vector using the forward kinematics model; then, calculating the pose error between the current pose and the target pose of the end effector of the industrial robot arm; then, mapping the pose error to the joint space according to the Jacobian matrix to obtain the joint angle correction amount; finally, updating the current joint angle vector using the joint angle correction amount, stopping the iteration when the pose error is less than a preset threshold, and outputting the target joint angle.

[0030] The path planning unit is used to determine the key points of the joint trajectory based on the initial joint angle vector and the target joint angle, and then perform interpolation calculation on the key points of the joint trajectory to generate a smooth and continuous joint trajectory that satisfies the joint motion constraints.

[0031] The control command generation unit is used to generate control signals based on the continuous joint trajectory and the switching parameters of the end effector of the industrial robot arm and the target tightening torque in the motion parameter vector Fd, using the PD controller algorithm, and finally control the industrial robot arm to complete the bolt tightening operation.

[0032] Furthermore, according to the bolt tightening method, the generation of control signals using the PD controller algorithm includes:

[0033] The current angle and angular velocity of each joint of the industrial robotic arm are collected and compared with the expected angle and angular velocity in the continuous joint trajectory to calculate the joint angle error and angular velocity error.

[0034] Real-time torque data during bolt tightening is collected and compared with the target tightening torque of the end effector of the industrial robotic arm in the motion parameter vector Fd to obtain the torque error.

[0035] By integrating joint angle error, angular velocity error, torque error, and target tightening torque, control torque commands for each joint are generated.

[0036] At different stages of bolt tightening, the weight ratio of each parameter is dynamically adjusted: during the stage when the industrial robotic arm approaches the bolt, the weight of angle error and angular velocity error is increased, with joint trajectory tracking as the core; during the bolt pre-tightening stage, the weight of torque error is gradually increased; and during the final tightening stage, torque control is the core, ensuring that the output torque is stable within the target tightening torque range.

[0037] The control torque commands of each joint are converted into control signals that conform to the industrial robotic arm drive interface specifications and sent to the joint drive unit of the industrial robotic arm. This drives each joint to track the continuous joint trajectory and adjusts the output torque in real time during the bolt tightening process to complete the bolt tightening operation.

[0038] The beneficial effects of this invention are as follows:

[0039] (1) This invention constructs a visual-language feature alignment module, uses the language feature vector Fl to generate query features, and performs cross-attention calculation with the multi-scale visual feature vector Fv, so that the visual features of each region in the image are weighted and reconstructed according to the degree of matching with the language semantics. Visual information is filtered at the feature level, so that the proportion of region features consistent with the semantics of natural language instructions is increased in the aligned feature vector Fvl, while the region features unrelated to natural language instructions are weakened, so that subsequent action generation is based on features with clear semantic orientation. Therefore, in the presence of multiple candidate bolts or interference regions, the consistency of target selection can be improved and the erroneous action decisions caused by visual information ambiguity can be reduced.

[0040] (2) This invention inputs the alignment feature vector Fvl into the action generation module, generates the action sequence Fa using the action decoding unit, and directly outputs the action parameter vector Fd through the parameterized mapping unit. Since the action parameter vector Fd is automatically generated by the action generation module based on the current alignment feature vector Fvl, rather than relying on a preset trajectory or manually taught path, the system can dynamically generate and execute actions according to the current scene state. Therefore, when the bolt position or task sequence changes, there is no need to re-teach or adjust the path, reducing the dependence on manual intervention and lowering maintenance costs.

[0041] (3) The present invention sets up a path planning unit in the control execution module, which generates a continuous joint trajectory by interpolating the joint space according to the target joint angle. Since the continuous joint trajectory performs a smooth transition processing on the discrete target joint angle, it avoids abrupt or discontinuous movements of the robotic arm during the movement. Therefore, compared with the control method of directly executing the discrete target position, it can improve the smoothness of the robotic arm movement process and reduce mechanical impact.

[0042] (4) In the control command generation unit, this invention adopts a multi-parameter fusion control strategy based on joint angle error, angular velocity error, and torque error, and dynamically adjusts the weight of each error term according to different stages of the tightening process. Since position and speed control are the main focus in the initial stage of tightening, and the torque control weight is gradually increased in the later stage of tightening, the control strategy can match the needs of different stages of the tightening process. Therefore, it can avoid the control instability problem that occurs in a single control mode, improve the torque control stability in the tightening process, and help ensure the consistency of the final tightening effect.

[0043] (6) This invention adopts a modular structure design, decoupling the visual encoding, language encoding, visual-language feature alignment, action generation, and control execution modules. Since the modules are connected through standard feature interfaces, the visual encoder, language encoder, or automated execution device can be replaced or expanded. Therefore, this invention is not only applicable to bolt tightening tasks, but can also be extended to various industrial tasks such as assembly, handling, and inspection, improving the system's versatility and scalability. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating the bolt tightening method based on the vision-language-action model in this embodiment.

[0045] Figure 2 This is a schematic diagram of the visual-language feature alignment module in this embodiment. Detailed Implementation

[0046] To facilitate understanding of this application, a more comprehensive description of this application will be provided below with reference to the accompanying drawings.

[0047] Figure 1 This is a flowchart illustrating the bolt tightening method based on a vision-language-action model in this embodiment. Figure 1 As shown, the bolt tightening method based on the vision-language-action model includes the following steps:

[0048] Step 1: Collect scene image data of the target bolt and its surrounding workpiece area at the bolt tightening station;

[0049] In this embodiment, an industrial camera, such as a RealSense D435i camera, is used to acquire image data of the bolts and surrounding workpieces. The image format can be RGB or RGB-D. The industrial camera is installed in two ways: a third-party view and an end-effector view. In the third-party view, the camera is installed in front of or to the side of the working area of ​​the automated execution device, such as an industrial robotic arm. The installation position of the third-party view camera can completely cover the distribution area of ​​multiple bolts to be tightened in the bolt tightening station, thereby ensuring that the automated execution device can obtain a complete global image of the target candidate area before performing the task. In the end-effector view, the camera is installed at the end of the automated execution device, such as an industrial robotic arm, and is parallel to the end of the automated execution device, such as an industrial robotic arm, to acquire local real-time image data of the area where the bolts to be tightened.

[0050] Step 2: Extract multi-scale visual features from the scene image data of the collected bolt and its surrounding workpiece area to obtain the multi-scale visual feature vector Fv;

[0051] In this embodiment, the acquired image data of the bolt and its surrounding workpiece scene is input into the visual encoder DINOv2 (Depth-Induced Noise for Online Video Self-Supervision, Version 2). The visual encoder DINOv2 extracts features from the image data to obtain a multi-scale visual feature vector Fv representing the target bolt and its surrounding workpiece scene. The multi-scale visual feature vector Fv includes low-level visual features, mid-level structural features, and high-level semantic features. The low-level visual features are used to represent the bolt's edge, texture, and contour information; the mid-level structural features are used to represent the local spatial relationship between the target bolt and its surrounding workpieces; and the high-level semantic features are used to represent the semantic association between the bolt area and the global scene. This can improve the ability to distinguish between the bolt area, the surrounding structural area, and the interference area in complex industrial scenes.

[0052] In a preferred embodiment, when the DINOv2 model contains 12 Transformer coding blocks, the output features of the 3rd layer are selected as low-level visual features, the output features of the 6th layer are selected as mid-level structural features, and the output features of the 12th layer are selected as high-level semantic features. The output features of the 3rd, 6th, and 12th layers are then concatenated and fused along the feature dimension to obtain a multi-scale visual feature vector Fv. The 3rd layer output features are selected as low-level visual features because features closer to the input still retain more image edge, texture, and contour information; the 6th layer output features are selected as mid-level structural features because the intermediate layer outputs have formed a local structural relationship representation, reflecting the spatial layout relationship between the target bolt and its surrounding workpieces; and the 12th layer output features are selected as high-level semantic features because the final layer output contains stronger global contextual semantic information, representing the semantic association between the area where the target bolt is located and the global scene.

[0053] Step 3: Perform semantic parsing on the natural language instructions to obtain the language feature vector Fl;

[0054] In this embodiment, the input natural language command is fed into the language encoder LLaMA (Large Language Model Meta AI). The language encoder LLaMA performs semantic parsing on the natural language command to obtain the language feature vector Fl. The language feature vector Fl is used to represent the semantic description information, spatial orientation information, task execution order information, and task constraint information of the bolt.

[0055] The role of the language feature vector Fl is to uniformly encode the task semantics, target selection conditions, and execution constraints in natural language instructions, so as to transform the natural language instructions from text descriptions into computable semantic representations, thereby improving the accuracy of target parsing and task constraint expression under complex natural language instructions.

[0056] Step 4: Align the multi-scale visual feature vector Fv and the language feature vector Fl across modalities using the visual-language feature alignment module to obtain the aligned feature vector Fvl;

[0057] In this embodiment, the multi-scale visual feature vector Fv and the language feature vector Fl are input into the visual-language feature alignment module to achieve the alignment and fusion of visual and language features, and obtain the aligned feature vector Fvl.

[0058] like Figure 2 As shown, the visual-language feature alignment module includes: a query feature generation unit, a key-value feature generation unit, and a cross-attention calculation unit.

[0059] The query feature generation unit is used to receive the language feature vector Fl and generate query features through a fully connected layer and normalization processing. These features are used to represent the direction of the semantic information of natural language instructions on visual features. For example, the text semantics provided by natural language instructions such as "which bolt to tighten, in which position" are converted into query signals for filtering visual regions.

[0060] The query feature generation unit includes a fully connected layer, a normalization layer, and an activation function layer connected in sequence.

[0061] The fully connected layer is used to receive the language feature vector Fl, perform a linear transformation on the language feature vector Fl, and generate an intermediate representation of the query features.

[0062] The normalization layer is used to normalize the intermediate representation of query features to enhance the stability of the distribution of the intermediate representation of query features.

[0063] The activation function layer is used to perform nonlinear transformation on the intermediate representation of the normalized query features to enhance the expressive power of the normalized query features and generate the final query features.

[0064] The key-value feature generation unit is used to receive a multi-scale visual feature vector Fv and generate key features and value features to characterize the visual content information of each region in the image.

[0065] The key-value feature generation unit includes a feature mapping layer, a normalization layer, and an activation function layer connected in sequence.

[0066] The feature mapping layer is used to receive the multi-scale visual feature vector Fv. The feature mapping layer can be implemented using a convolutional layer to perform a linear transformation on the multi-scale visual feature vector Fv to generate an intermediate representation of key-value features, which is used to complete the mapping from the multi-scale visual feature vector Fv to key features and value features.

[0067] The normalization layer is used to normalize the intermediate representation of the key-value features in order to improve the stability of the distribution of the intermediate representation of the key-value features.

[0068] The activation function layer is used to perform nonlinear transformation on the intermediate representation of the normalized key-value features to enhance the expressive power of the normalized key-value features and generate the final key features and value features.

[0069] The cross-attention calculation unit is used to receive the query features, key features, and value features, calculate the correlation score between the visual content information of each region in the image in the key features and the semantic information of the natural language instructions in the query features through feature similarity calculation, and perform weighted processing on the value features according to the weight distribution obtained after normalization of the correlation score to obtain the aligned feature vector Fvl.

[0070] The feature similarity calculation is used to perform position-by-position matching between the query feature and the key feature. By calculating the similarity between the query feature and the key feature in the feature space, the corresponding relevance score is obtained.

[0071] The cross-attention calculation unit is also used to perform normalization processing based on the relevance score to generate a weight distribution, which consists of multiple weight coefficients to represent the degree of matching between the visual content information of each region in the image in the key features and the semantic information of the natural language instructions in the query features.

[0072] The cross-attention calculation unit performs weighted processing on the value features according to the weight coefficients in the weight distribution to enhance visual region features related to natural language instructions and suppress visual region features unrelated to natural language instructions, thereby obtaining the aligned feature vector Fvl.

[0073] In a preferred embodiment, the feature similarity calculation is implemented using a dot product similarity method, which involves performing a dot product operation between the query feature and the key feature to obtain the corresponding relevance score. The calculation method is as follows:

[0074]

[0075] in, This represents the i-th query feature; Represents the j-th key feature; This represents the relevance score between the i-th query feature and the j-th key feature.

[0076] Furthermore, the relevance score is normalized to obtain a weight distribution, which consists of multiple weight coefficients to represent the degree of matching between the visual content information of each region in the image in the key features and the semantic information of the natural language instructions in the query features.

[0077] The weight coefficients in the weight distribution are weighted and summed to obtain the aligned visual feature vector Fvl, which is calculated as follows:

[0078]

[0079] in, This represents the feature of the j-th value; This represents the corresponding weighting coefficient; This represents the visual feature after alignment (i).

[0080] Step 5: Input the aligned feature vector Fvl into the action generation module to generate the action sequence Fa, and further parameterize and map it to obtain the action parameter vector Fd.

[0081] In this embodiment, the aligned feature vector Fvl is input into the action generation module, and the action sequence Fa and the action parameter vector Fd are obtained through action decoding and parameterized mapping.

[0082] The action generation module includes an action decoding unit and a parameterized mapping unit.

[0083] The action decoding unit is used to receive the alignment feature vector Fvl and perform feature parsing on the alignment feature vector Fvl to generate the action sequence Fa.

[0084] The action sequence Fa is a discrete sequence of action steps arranged in chronological order, used to represent the operation flow during the execution of natural language instructions. The aligned feature vector Fvl is converted into high-level action semantics to represent which operations are performed and their order.

[0085] In this embodiment, the action decoding unit adopts a multilayer perceptron network structure, which includes at least a first fully connected layer, a normalization layer, an activation function layer, a second fully connected layer, and an action classification output layer connected in sequence.

[0086] The first fully connected layer is used to receive the aligned feature vector Fvl and perform a linear mapping on the aligned feature vector Fvl to obtain intermediate features for action decoding;

[0087] The normalization layer is used to normalize the intermediate features of the action decoding;

[0088] The activation function layer is used to perform nonlinear transformation on the normalized intermediate features of action decoding to enhance the feature representation capability;

[0089] The second fully connected layer is used to further map the intermediate features of the nonlinearly transformed action decoding to obtain the action decoding feature representation for action classification;

[0090] The action classification output layer receives the action decoding feature representation, performs a linear mapping on the action decoding feature representation through a fully connected layer to obtain the action category score, further normalizes the score through a softmax function to obtain the probability distribution of the action category, and determines the action category at the current time based on the probability distribution.

[0091] Furthermore, the action decoding unit generates an action sequence Fa by cyclically outputting action categories according to time steps.

[0092] The parameterization mapping unit is used to receive the action sequence Fa and map the action sequence into a continuous action parameter vector Fd.

[0093] The parameterization mapping unit is implemented using a multilayer perceptron network structure. It performs layer-by-layer feature mapping on the action sequence Fa through a multilayer fully connected network, transforming the discrete action sequence into a continuous action parameter vector Fd with clear control meaning.

[0094] The motion parameter vector Fd is used to characterize the control parameter information when performing each motion step. The motion parameter vector Fd includes the end effector pose parameters of the automated execution device, such as an industrial robotic arm, used to characterize the position and attitude information of the end effector of the automated execution device, such as an industrial robotic arm; the switching parameters of the end effector of the automated execution device, such as an industrial robotic arm, used to characterize the switching state of the end effector of the automated execution device, such as an industrial robotic arm, during operation; and the target tightening torque of the end effector of the automated execution device, such as an industrial robotic arm, used to set the final torque value that the end effector should achieve.

[0095] In a preferred embodiment, the end effector of an automated execution device, such as an industrial robotic arm, is preferably a tightening gun.

[0096] Step 6: Input the motion parameter vector Fd into the control execution module to control the automated execution equipment to complete the bolt tightening operation.

[0097] This embodiment uses the motion parameter vector Fd to perform kinematic solving and generate control commands, thereby driving the automated execution equipment to complete the bolt tightening operation.

[0098] The control execution module includes: an inverse kinematics solving unit, a path planning unit, and a control command generation unit.

[0099] The inverse kinematics solving unit is used to receive the end pose parameters of the automated execution device, such as an industrial robotic arm, in the motion parameter vector Fd, and calculate the target joint angles of each joint of the robotic arm.

[0100] The inverse kinematics solving unit performs inverse kinematics calculations using an iterative method. First, the current joint angle of the automated execution device, such as an industrial robotic arm, is set as the initial joint angle vector. Based on the initial joint angle vector, the current pose of the end effector of the automated execution device, such as the industrial robotic arm, is calculated using the forward kinematics model. Then, the pose error between the current pose of the end effector of the automated execution device, such as the industrial robotic arm, and the target pose is calculated. Next, the pose error is mapped to the joint space using the Jacobian matrix to obtain the joint angle correction amount. Finally, the current joint angle vector is updated using the joint angle correction amount until the pose error is less than a preset threshold, at which point the iteration stops and the target joint angle is output.

[0101] The path planning unit is used to generate continuous joint trajectories for the movement of automated execution devices, such as industrial robotic arms, based on the target joint angle.

[0102] In this embodiment, the path planning unit determines the key points of the joint trajectory based on the initial joint angle vector and the target joint angle, and then performs interpolation calculation on the key points of the joint trajectory to generate a smooth and continuous joint trajectory that satisfies the joint motion constraints.

[0103] In a preferred embodiment, the path planning unit uses joint space interpolation to interpolate key points of the joint trajectory to obtain a continuous joint trajectory.

[0104] The control command generation unit is used to generate control signals based on the continuous joint trajectory and the switching parameters of the end effector of the industrial robot arm and the target tightening torque in the motion parameter vector Fd, using existing PD controller algorithms, and finally control the automated execution equipment, such as the industrial robot arm, to complete the bolt tightening operation.

[0105] The PD controller algorithm includes real-time error calculation, multi-parameter fusion control, and control signal output.

[0106] The real-time error calculation involves collecting real-time status data such as the current angle, angular velocity, and output torque of each joint using automated execution devices, such as joint position sensors and torque sensors at the end effector of an industrial robotic arm. This data is then compared with the desired angle and angular velocity in the continuous joint trajectory to calculate the joint angle error and angular velocity error. Simultaneously, real-time torque data during bolt tightening is collected and compared with the target tightening torque of the end effector of the industrial robotic arm in the motion parameter vector Fd to obtain the torque error.

[0107] The multi-parameter fusion control integrates joint angle error, angular velocity error, torque error, and the target tightening torque in the motion parameter vector Fd to generate control torque commands for each joint. The weighting of each parameter is dynamically adjusted at different stages of bolt tightening: during the approach phase of automated execution equipment such as an industrial robotic arm, the weighting of angle error and angular velocity error is increased, with joint trajectory tracking as the core; during the bolt pre-tightening phase, the weighting of torque error is gradually increased; and in the final tightening phase, torque control is the core, ensuring that the output torque remains stable within the target tightening torque range.

[0108] The control signal output converts the generated joint control torque command into a control signal that conforms to the drive interface specification of automated execution equipment, and sends it to the joint drive unit of automated execution equipment such as an industrial robotic arm to drive each joint to accurately track the continuous joint trajectory, and adjust the output torque in real time during the bolt tightening process to complete the bolt tightening operation.

[0109] The control signals include: a joint drive torque control signal, used to control the motors of each joint of the automated execution device, such as an industrial robotic arm, to output corresponding drive torques, so that each joint of the automated execution device, such as an industrial robotic arm, moves according to a continuous joint trajectory; a joint position reference signal, used to provide the target angle of each joint of the automated execution device, such as an industrial robotic arm, within the current control cycle, to assist in joint motion control; a joint speed reference signal, used to provide the target movement speed of each joint of the automated execution device, such as an industrial robotic arm, within the current control cycle, to ensure the continuity and smoothness of the robotic arm's movement process; an end effector control signal, used to control the on / off state of the end effector of the automated execution device, such as an industrial robotic arm; a tightening torque control signal, used to control the end effector of the automated execution device, such as an industrial robotic arm, to output a target tightening torque; and a tightening angle control signal, used to control the rotation angle of the end effector of the automated execution device, such as an industrial robotic arm.

[0110] It should be understood that, inspired by the technical concept of this invention, those skilled in the art can make various improvements or modifications based on the above content without departing from the scope of this invention, and these modifications still fall within the protection scope of this invention.

Claims

1. A bolt tightening method based on a vision-language-action model, characterized in that, The method includes: Collect scene image data of the target bolt and its surrounding workpiece area; Multi-scale visual features are extracted from the collected scene image data to obtain the multi-scale visual feature vector Fv; Semantic parsing is performed on the natural language commands input by the user, converting the natural language commands from text descriptions into computable semantic representations, and obtaining a language feature vector Fl; the language feature vector Fl is used to characterize the semantic description information, spatial orientation information, task execution order information, and task constraint information of the bolt; Cross-modal feature alignment is performed on Fv and Fl using the visual-language feature alignment module to obtain the aligned feature vector Fvl; The action generation module performs action decoding and parameter mapping on the aligned feature vector Fvl to generate the action parameter vector Fd. The motion parameter vector Fd is input into the control execution module for kinematic solution and control command generation, thereby driving the automated execution equipment to complete the bolt tightening operation.

2. The bolt tightening method according to claim 1, characterized in that, Multi-scale visual features are extracted from the acquired scene image data by a visual encoder to obtain a multi-scale visual feature vector Fv. The multi-scale visual feature vector Fv includes low-level visual features, mid-level structural features and high-level semantic features. The low-level visual features are used to characterize the edge, texture and contour information of the target bolt, the mid-level structural features are used to characterize the local spatial relationship between the target bolt and its surrounding workpieces, and the high-level semantic features are used to characterize the semantic association between the area where the target bolt is located and the global scene.

3. The bolt tightening method according to claim 1, characterized in that, The visual-language feature alignment module includes: The query feature generation unit is used to generate query features based on the language feature vector Fl; The key-value feature generation unit is used to generate key features and value features based on the multi-scale visual feature vector Fv. The cross-attention calculation unit is used to receive the query features, key features, and value features. Through feature similarity calculation, it calculates the correlation score between the visual content of each image region in the key features and the semantic information of the natural language instruction in the query features. The score is normalized to generate a weight distribution. Then, the value features are weighted according to the weight coefficients in the weight distribution to enhance the visual region features related to the instruction semantics and suppress irrelevant features. Finally, the semantically and visually aligned feature vector Fvl is output.

4. The bolt tightening method according to claim 3, characterized in that, The query feature generation unit includes a fully connected layer, a normalization layer, and an activation function layer connected in sequence; the fully connected layer is used to receive the language feature vector Fl, perform a linear transformation on Fl, and generate an intermediate representation of the query features. The normalization layer is used to normalize the intermediate representation of query features to enhance the stability of the distribution of the intermediate representation of query features; the activation function layer is used to perform nonlinear transformation on the normalized intermediate representation of query features to enhance the expressive power of the normalized query features and generate the final query features.

5. The bolt tightening method according to claim 3, characterized in that, The key-value feature generation unit includes a feature mapping layer, a normalization layer, and an activation function layer connected in sequence. The feature mapping layer receives a multi-scale visual feature vector Fv, performs a linear transformation on Fv, and generates an intermediate representation of the key-value features, realizing the mapping transformation of Fv to key features and value features. The normalization layer normalizes the intermediate representation of the key-value features to improve the stability of the distribution of the intermediate representation of the key-value features. The activation function layer performs a nonlinear transformation on the normalized intermediate representation of the key-value features to enhance the expressive power of the normalized key-value features and generate the final key features and value features.

6. The bolt tightening method according to claim 5, characterized in that, The feature similarity calculation is used to perform positional matching between the query feature and the key feature. By calculating the similarity between the query feature and the key feature in the feature space, the corresponding relevance score is obtained, specifically through the following formula: ； in, This represents the i-th query feature; Represents the j-th key feature; This represents the relevance score between the i-th query feature and the j-th key feature.

7. The bolt tightening method according to claim 1, characterized in that, The action generation module includes: The action decoding unit is used to perform feature parsing on the received aligned feature vector Fvl, output the action category step by step according to the time step, and then generate an action sequence Fa to represent the operation to be performed and its order. The parameter mapping unit is used to perform layer-by-layer feature mapping on the action sequence Fa through a multi-layer fully connected network to generate an action parameter vector Fd. The action parameter vector Fd is used to represent the control parameter information when executing each action step, including the pose parameters of the end effector of the industrial robot arm, the switching parameters of the end effector of the industrial robot arm, and the target tightening torque. The pose parameters are used to represent the position and attitude information of the end effector of the industrial robot arm. The switching parameters are used to represent the switching state of the end effector of the industrial robot arm during operation.

8. The bolt tightening method according to claim 7, characterized in that, The action decoding unit adopts a multilayer perceptron network structure, which includes at least a first fully connected layer, a normalization layer, an activation function layer, a second fully connected layer, and an action classification output layer connected in sequence. The first fully connected layer receives the aligned feature vector Fvl and performs a linear mapping on Fvl to obtain intermediate action decoding features; the normalization layer normalizes the intermediate action decoding features; the activation function layer performs a nonlinear transformation on the normalized intermediate action decoding features to enhance feature representation capability; the second fully connected layer further maps the nonlinearly transformed intermediate action decoding features to output an action decoding feature representation suitable for action classification. The action classification output layer is used to linearly map the action decoding feature representation through a fully connected layer to obtain an action category score. Then, the action category score is normalized by a softmax function to obtain the probability distribution of the action category. Finally, the action category at the current moment is determined based on the probability distribution of the action category.

9. The bolt tightening method according to claim 1, characterized in that, The control execution module includes: The inverse kinematics unit is used to perform inverse kinematics calculations based on the pose parameters of the end effector of the industrial robot arm in the motion parameter vector Fd, and to obtain the target joint angles of each joint of the robot arm using an iterative solution method. The specific calculation process includes: first, setting the current joint angle of the industrial robot arm as the initial joint angle vector; then, calculating the current pose of the end effector of the industrial robot arm based on the initial joint angle vector using the forward kinematics model; then, calculating the pose error between the current pose and the target pose of the end effector of the industrial robot arm; then, mapping the pose error to the joint space according to the Jacobian matrix to obtain the joint angle correction amount; finally, updating the current joint angle vector using the joint angle correction amount, stopping the iteration when the pose error is less than a preset threshold, and outputting the target joint angle. The path planning unit is used to determine the key points of the joint trajectory based on the initial joint angle vector and the target joint angle, and then perform interpolation calculation on the key points of the joint trajectory to generate a smooth and continuous joint trajectory that satisfies the joint motion constraints. The control command generation unit is used to generate control signals based on the continuous joint trajectory and the switching parameters of the end effector of the industrial robot arm and the target tightening torque in the motion parameter vector Fd, using the PD controller algorithm, and finally control the industrial robot arm to complete the bolt tightening operation.

10. The bolt tightening method according to claim 9, characterized in that, The generation of control signals using the PD controller algorithm includes: The current angle and angular velocity of each joint of the industrial robotic arm are collected and compared with the expected angle and angular velocity in the continuous joint trajectory to calculate the joint angle error and angular velocity error. Real-time torque data during bolt tightening is collected and compared with the target tightening torque of the end effector of the industrial robotic arm in the motion parameter vector Fd to obtain the torque error. By integrating joint angle error, angular velocity error, torque error, and target tightening torque, control torque commands for each joint are generated. At different stages of bolt tightening, the weight ratio of each parameter is dynamically adjusted: during the stage when the industrial robotic arm approaches the bolt, the weight of angle error and angular velocity error is increased, with joint trajectory tracking as the core; during the bolt pre-tightening stage, the weight of torque error is gradually increased; and during the final tightening stage, torque control is the core, ensuring that the output torque is stable within the target tightening torque range. The control torque commands of each joint are converted into control signals that conform to the industrial robotic arm drive interface specifications and sent to the joint drive unit of the industrial robotic arm. This drives each joint to track the continuous joint trajectory and adjusts the output torque in real time during the bolt tightening process to complete the bolt tightening operation.

Images