A bolt loosening detection and automatic re-tightening method based on model drive

Abstract

The present invention discloses a method for detecting and automatically re-tightening bolt loosening based on model driving, the method comprising: real-time image acquisition of the device to be tested; identifying the bolt position in the image based on a pre-trained deep learning bolt recognition model, and calculating the loosening angle of the bolt; wherein, when the loosening angle reaches a preset threshold, a re-tightening task is established for a robot arm based on the loosening angle; the robot arm plans a motion path based on the re-tightening task; the robot arm completes the bolt re-tightening task according to the motion path. Through the processing scheme disclosed in the present invention, automatic detection and re-tightening of bolt loosening can be achieved, significantly improving the efficiency and safety in industrial production.

Description

Technical Field

[0001] The invention relates to the field of automatic control technology, and in particular to a bolt loosening detection and automatic re-tightening method based on model driving. Background Art

[0002] In modern industrial production, the stability of bolt connections is crucial to the safe operation of equipment. However, the problem of bolt loosening and failure remains a common challenge in industrial applications. At present, the solution commonly adopted in the industry for loosening of connection systems is a method that combines "automatic detection and manual re-tightening". Specifically, the current automatic detection method mainly relies on analyzing the signal differences before and after the bolts are loosened. For example, image processing technology is used to extract feature points in the image, and the changes of these feature points during the loosening process are analyzed to detect loosening failures. This method can realize loosening detection and calculate the loosening angle with high accuracy, which has important application value for industrial production.

[0003] Although bolt loosening detection technology is relatively mature in theory, the problem of loosening failure in threaded connection systems still exists. The main constraint is that existing methods can only detect loosening failures, but fail to provide effective response strategies and systems. For example, once a loose bolt is detected, manual intervention is usually required to manually re-tighten it according to the detection results. This practice not only causes delays in the re-tightening operation, but also often causes the re-tightening angle to be inconsistent with the loosening angle due to the delay, which seriously hinders the automation and intelligent development of the production process. Especially in experimental environments or dangerous production environments, it is more inclined that the system can automatically repair rather than rely on manual intervention. In addition, there is currently a lack of necessary control algorithms to guide how to complete the re-tightening operation in a safe and efficient manner while satisfying position and mechanical constraints, which limits the application potential of automatic re-tightening technology in complex industrial scenarios with precise torque and angle.

[0004] In modern industry, the reliability of bolt connections is crucial to the stability and safety of mechanical equipment, and bolt loosening is one of the main causes of connection failure. Therefore, developing an effective model-driven bolt loosening detection and automatic re-tightening method is of great significance for improving production efficiency and ensuring equipment operation safety. Summary of the invention

[0005] In view of this, an embodiment of the present disclosure provides a model-driven method for detecting and automatically tightening bolt loosening, which at least partially solves the problems existing in the prior art.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for detecting and automatically tightening bolt looseness based on a model drive, the method comprising the following steps:

[0007] Real-time image acquisition of the device under test;

[0008] Based on a pre-trained deep learning bolt recognition model, the bolt position in the image is identified, and the loosening angle of the bolt is calculated; wherein, when the loosening angle reaches a preset threshold, a re-tightening task is established for the robot arm based on the loosening angle;

[0009] The robot arm plans a motion path based on the re-twisting task;

[0010] The robot arm completes the task of re-tightening the bolts according to the motion path.

[0011] According to a specific implementation of the embodiment of the present disclosure, the pre-trained deep learning bolt recognition model includes:

[0012] The target detection network of the deep learning bolt recognition model is set to YOLOv8, the backbone network is set to CSPNet, and the feature fusion network is set to FPN;

[0013] Collect bolt images and build basic data sets;

[0014] Performing data enhancement on the basic data set to obtain an extended data set;

[0015] Annotating the position of the bolt in each image in the extended dataset based on Labelbox;

[0016] Dividing the extended data set into a training set, a test set, and a validation set;

[0017] A deep learning bolt recognition model is trained based on the extended data set, and the model parameters are optimized using stochastic gradient descent.

[0018] According to a specific implementation of the embodiment of the present disclosure, the step of performing data enhancement on the basic data set includes:

[0019] Improve image contrast through histogram equalization, grayscale adjustment, denoising and filtering.

[0020] According to a specific implementation of the embodiment of the present disclosure, the calculating the loosening angle of the bolt includes the following steps:

[0021] The image is partially cropped based on the identified bolt position to obtain a bolt image;

[0022] enhancing visibility of feature points of the bolt image;

[0023] Feature point extraction and matching of the enhanced bolt image are performed based on the ORB feature point matching algorithm;

[0024] Compare the changes of feature points in the initial image and the real-time image to calculate the loosening angle of the bolt.

[0025] According to a specific implementation of the embodiment of the present disclosure, the robot arm plans a motion path based on the re-tightening task, including:

[0026] Build a robot model based on the urdf file and Moveit planner, and define the physical properties and kinematic parameters of the robot arm in the robot model;

[0027] Plan the motion path through the path planning algorithm based on the robot's working range, speed requirements, and spatial obstacles;

[0028] The motion path is optimized based on an optimization algorithm.

[0029] According to a specific implementation of the embodiment of the present disclosure, the motion path is optimized based on an optimization algorithm, including:

[0030] Based on the calculation, the process of the robot arm performing the complex twisting task is divided into k+1 stages, and the starting and ending load intervals of each stage are P1:[0, Te shift1 ],P2:[Te shift1 , Te shift2 ]…P k :[Te shiftk-1 , Te shiftk ], P k+1 :[Te shift2 ,T (x,y,z) ]; For each stage, in the target posture group q goal In , select all postures that meet the mechanical requirements;

[0031] Calculate the optimal posture q of the first stage P1 p1 and the final stage P k+1 The optimal posture q p(k+1) .

[0032] According to a specific implementation of the embodiment of the present disclosure, the optimal posture q of the first stage P1 is calculated. p1 and the final stage P k+1 The optimal posture q p(k+1) ,include:

[0033] Calculate q based on the following formula p1 and q p(k+1) :

[0034]

[0035] Among them, q p1 is the optimal posture in stage P1; q initis the initial posture of the robot, ||·|| is the Euclidean distance of the posture; n is the number of joints of the robot; j is the number of inverse solutions of the target posture; q j is the jth inverse solution posture; q p（k+1） P k+1 The optimal posture of the stage; m is the number of posture inverse solutions;

[0036] Taking the minimization of adjustment amplitude as the constraint function and combining the Dijkstra shortest path algorithm, the optimal posture switching path of the re-twisting process is calculated, and the optimal posture switching path of the re-twisting process is obtained from q p1 to q p（k+1） The optimal posture sequence q={q p1 , q p2 ,…, q pk , q p（k+1）}.

[0037] According to a specific implementation of the embodiment of the present disclosure, the method further includes:

[0038] Get the initial posture q of the robot arm init , get the spatial position p of the target bolt (x,y,z) and target re-tightening torque T (x,y,z) , and define the direction of the end grab bolt dir goal ;

[0039] According to the target position p (x,y,z) and end direction dir goal , based on the inverse kinematics operation, the target posture group q of the manipulator is obtained goal ={q g1 ,q g2 ,…,q gm}, where m is the number of inverse solutions of attitude;

[0040] Calculate the Jacobian matrix J of the robot arm for all postures in the target posture group goal ={J g1 ,J g2 ,…,J gm}, and its inverse matrix ;

[0041] The joint load Tj in each posture is calculated based on the following inverse dynamics formula goal ={ }:

[0042] = ·Te

[0043] in, is the joint load of the mth posture; is the Jacobian inverse matrix of the mth posture; Te is the load torque at the end of the robot arm;

[0044] Based on joint load Tj and joint maximum load Tj lim By comparing the load limit, the switching point Te is determined. shift ;

[0045] Get all joints of all postures curve, and compare Tj and Tj along the Te coordinate lim , define to satisfy Tj(Te)>Tj lim The point is the switching point Te shift ∈{Te shift1 ,Te shift2 ,…, Te shiftk};

[0046] Among them, Tj lim ={Tj 1 ,Tj 2 ,…,Tj n} is the maximum load torque of the robot arm 1~n joints, Te shiftk is the kth switching point.

[0047] According to a specific implementation of the embodiment of the present disclosure, the method further includes:

[0048] When the loosening angle of the bolt does not reach a preset threshold, image acquisition is performed at a first frequency;

[0049] When the loosening angle of the bolt reaches a preset threshold, image acquisition is performed at a second frequency.

[0050] According to a specific implementation of the embodiment of the present disclosure, the method further includes:

[0051] When the robot arm completes the task of re-tightening the bolt according to the motion path, the bolt loosening angle is updated in real time:

[0052] The bolt loosening angle during the re-tightening task is calculated based on the following formula:

[0053]

[0054] in, is the re-tightening angle; is the initial tightening angle; The amount of adjustment caused by the change in loose angle.

[0055] The model-driven bolt loosening detection and automatic re-tightening method in the disclosed embodiment has specifically designed a re-tightening information update strategy and a re-tightening posture optimization algorithm, which can detect the loosening failure state of the bolt in a high-precision manner and perform safe and efficient automatic re-tightening operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0057] Figure 1 A schematic diagram of a process flow of a bolt loosening detection and automatic re-tightening method based on model driving provided in an embodiment of the present disclosure;

[0058] Figure 2 A flowchart of a method for detecting and automatically tightening bolt looseness based on a model drive provided in an embodiment of the present disclosure;

[0059] Figure 3 A technical roadmap for a model-driven bolt loosening detection and automatic re-tightening method provided in an embodiment of the present disclosure;

[0060] Figure 4 A schematic diagram of a looseness detection and angle calculation method based on image processing provided in an embodiment of the present disclosure;

[0061] Figure 5 A schematic diagram of a kinematic model of a robotic arm and a path planning process based on ROS provided in an embodiment of the present disclosure;

[0062] Figure 6 A schematic diagram of a method for optimizing a re-twisting posture based on terminal torque provided in an embodiment of the present disclosure;

[0063] Figure 7 A schematic diagram of a model-driven bolt loosening detection and automatic re-tightening system provided in an embodiment of the present disclosure;

[0064] Figure 8 A schematic diagram of a re-twisting information update strategy combined with robot arm motion control provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0065] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0066] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0067] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement a device and / or practice a method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0068] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0069] In view of the limitations of existing methods in automated and intelligent production and manufacturing, the present invention proposes a model-based bolt loosening detection and automatic re-tightening method. This method combines computer vision and automatic control technology, and can accurately locate the bolt position in the industrial image and accurately calculate its loosening angle. Furthermore, a re-tightening information update strategy and a robot arm re-tightening posture optimization algorithm were developed, which can drive the robot arm to perform precise re-tightening operations on loose bolts in a safe and efficient manner. This technology has high automation characteristics, which not only significantly improves the efficiency and accuracy of re-tightening operations, but also provides effective technical support for realizing the automation and intelligence of industrial production processes.

[0070] First, a deep learning-based bolt recognition model is built, which can accurately identify the location of bolts in complex industrial images. Then, image processing technology is used to enhance the bolt features and extract feature points. By comparing the changes in feature points before and after the bolts are loosened, the loosening failure is detected and the loosening angle is calculated.

[0071] Next, the present invention designs a robot arm motion control method for the re-tightening task, ensuring that the robot arm can reach the specified target position in a spatially planned manner and perform the re-tightening operation in a safe and accurate manner.

[0072] Finally, an integrated automatic re-tightening system and information update strategy were developed. The system integrates the image acquisition system, looseness detection system, control system, re-tightening system and connection system, and can realize the re-tightening operation of bolts under precise control. Through closed-loop control and real-time adjustment strategy, the re-tightening parameters are detected and transmitted in real time to ensure the high consistency of the re-tightening angle and the loosening angle, so as to achieve the best re-tightening effect.

[0073] Figure 1 A schematic diagram of the model-driven bolt loosening detection and automatic re-tightening method process provided in an embodiment of the present disclosure.

[0074] Figure 2 For Figure 1 Corresponding model-driven flow chart of bolt loosening detection and automatic re-tightening method.

[0075] Figure 3 For Figure 1 Schematic diagram of the corresponding technical route of model-driven bolt loosening detection and automatic re-tightening method.

[0076] like Figure 1 As shown, in step S110, real-time image acquisition is performed on the device to be tested.

[0077] In an embodiment of the present invention, the method further includes: when the loosening angle of the bolt does not reach a preset threshold, performing image acquisition at a first frequency; when the loosening angle of the bolt reaches a preset threshold, performing image acquisition at a second frequency.

[0078] More specifically, the process proceeds to step S120.

[0079] In step S120, the bolt position in the image is identified based on the pre-trained deep learning bolt recognition model, and the loosening angle of the bolt is calculated; wherein, when the loosening angle reaches a preset threshold, a re-tightening task is established for the robot arm based on the loosening angle.

[0080] In an embodiment of the present invention, a pre-trained deep learning bolt recognition model includes: a target detection network of the deep learning bolt recognition model is set to YOLOv8, a backbone network is set to CSPNet, and a feature fusion network is set to FPN; bolt images are collected to construct a basic data set; data enhancement is performed on the basic data set to obtain an extended data set; the position of the bolt in each image in the extended data set is annotated based on Labelbox; the extended data set is divided into a training set, a test set, and a validation set; a deep learning bolt recognition model is trained based on the extended data set, and model parameters are optimized using stochastic gradient descent.

[0081] In the embodiment of the present invention, the data enhancement of the basic data set includes: improving image contrast through histogram equalization, grayscale adjustment, denoising and filtering.

[0082] More specifically, a deep learning-based bolt recognition model is constructed to accurately identify and locate the bolts by learning their image features.

[0083] The steps for model building and training are:

[0084] 1) In the initial stage, the basic dataset was constructed by manually collecting bolt images and numbering them. To increase sample diversity, data augmentation techniques, including image cropping, rotation, scaling, noise addition, and HSV adjustment, were used to expand the dataset. Subsequently, annotation tools such as Labelbox were used to accurately mark the bolt positions, and the dataset was divided into training, test, and validation sets to provide high-quality data for model training and evaluation.

[0085] 2) In terms of model selection, this method uses efficient target detection networks such as YOLOv8. This network uses advanced backbone networks and feature fusion networks such as CSPNet and FPN, which enhances the capabilities of feature extraction and multi-scale target detection, and introduces anchor-free detection heads to simplify the model while improving detection performance. In addition, YOLOv8 uses new activation functions such as Mish and attention mechanisms such as SE modules to further improve the expressiveness and accuracy of the model. This network achieves high detection speed while maintaining high accuracy, and is suitable for real-time monitoring application scenarios of this method.

[0086] 3) During the model training phase, stochastic gradient descent (SGD) and its variants, such as momentum or adaptive learning rate algorithms (such as Adam), are used to optimize model parameters. These algorithms update model parameters by randomly selecting samples to calculate gradients. The momentum method uses historical gradient information to enhance the stability of the model in dimensions where gradients change dramatically. The Adam algorithm combines the advantages of the momentum method and RMSprop to accelerate the convergence of the model by adaptively adjusting the learning rate. By reasonably setting the batch size and convergence conditions, the model can be continuously optimized through iterative training until the required recognition accuracy is achieved. After training, the model is deployed to the bolt loosening detection and automatic re-tightening process to achieve automated quality control.

[0087] For example, we first collected 1,000 images of bolts of different types and in different states, and numbered the bolts in each image. In order to increase the diversity of samples, we used data augmentation techniques, including image cropping, rotation, scaling, noise addition, and HSV adjustment, to expand the original 1,000 images to 5,000.

[0088] Then, Labelbox is used to annotate the bolt positions in the image, and the data set is divided into training set (70%), test set (15%) and validation set (15%). Then, the YOLOv8 target detection network is used, and the backbone network is configured as CSPNet and the feature fusion network is configured as FPN to enhance the feature extraction capability of the model. In the model training stage, stochastic gradient descent (SGD) and its variants, including momentum method and adaptive learning rate algorithm Adam, are used to finally achieve high-precision bolt recognition.

[0089] In an embodiment of the present invention, the calculation of the loosening angle of the bolt includes the following steps: locally cropping the image based on the identified bolt position to obtain a bolt image; enhancing the visibility of feature points of the bolt image; extracting and matching feature points of the enhanced bolt image based on an ORB feature point matching algorithm; and comparing the changes in feature points in the initial image and the real-time image to calculate the loosening angle of the bolt.

[0090] More specifically, in this step, the present invention adopts a method for bolt loosening detection and angle calculation based on image feature points. The image of the connection structure is captured in real time by an industrial camera, and a series of image enhancement techniques are used to enhance the discernibility of the bolt features in the image. Subsequently, a feature point matching algorithm is used to compare the feature point differences between the initial image and the detection image, thereby realizing loosening detection and calculating the bolt loosening angle. Key information such as bolt position and loosening angle will provide input for subsequent re-tightening operations. The principle of this method is as follows Figure 4 shown.

[0091] The steps for looseness detection and angle calculation are as follows:

[0092] 1) Use industrial cameras to collect images of the connection structure in real time. The acquisition frequency is set according to application requirements and engineering experience. Then the bolt recognition algorithm is used to identify the bolts in the image and output the bolt positions.

[0093] 2) After the bolt position is successfully identified, the image is partially cropped to retain only the area containing the key bolt feature information to simplify the subsequent image processing process and improve computational efficiency. The cropped image will be processed through a series of image enhancement techniques to optimize the identification of bolt feature points. For example, histogram equalization technology is applied to adjust the image contrast to make the bolt feature points more obvious in the image, and the accuracy of feature point detection is further improved by refining the image grayscale level and filtering denoising. These image processing steps provide high-quality image data for subsequent operations and improve the accuracy of the bolt loosening detection process.

[0094] 3) Subsequently, the ORB (Oriented FAST and Rotated BRIEF) method is used to extract, describe and match feature points. This algorithm uses a local fast feature point binary descriptor. By combining the FAST key point detector and the BRIEF descriptor, it achieves consistency under rotation and scale transformation, providing an efficient and robust method for feature point matching of bolt images. Among them, the FAST corner point detection algorithm achieves the purpose of fast corner point extraction by adjusting the floating size of the grayscale difference between pixel points. The BRIEF feature point description algorithm randomly selects several point pairs near the extracted feature points and combines the grayscale values ​​corresponding to these point pairs into feature descriptors. The advantage of the ORB algorithm is that it provides stable feature point matching results while maintaining high speed, which is particularly important for real-time detection applications.

[0095] 4) Finally, by comparing the difference between the feature points in the initial image and the detected image, the loosening angle of the bolt is calculated. This process involves accurate measurement and analysis of the position of the feature points to determine whether the bolt is loose. Through precise angle calculation, the present invention can accurately determine the loose state of the bolt and provide data for subsequent automatic re-tightening.

[0096] For example, the looseness detection and angle calculation method based on image feature points is as follows:

[0097] An industrial camera is used to collect real-time images of the mechanical connection system, and the collection frequency is set according to the actual industrial application requirements and experience to ensure the real-time and accuracy of the data. Through the bolt recognition algorithm, the system automatically detects and identifies the bolts in the image, realizes the precise positioning and focusing of the target bolts, and provides high-quality image data for subsequent loosening detection. After the bolt position is successfully identified, the system performs local cropping on the image, retaining only the part containing the key bolt feature information. Subsequently, by implementing image processing techniques such as histogram equalization, grayscale adjustment, denoising and filtering, the image contrast is significantly improved, and the visibility of the bolt feature points is enhanced, thereby providing a guarantee for the accurate detection and reliability of the feature points. Furthermore, the present invention uses the ORB feature point matching algorithm to extract and match the bolt image. The ORB algorithm combines the FAST key point detector and the BRIEF descriptor, and can maintain the consistency of the feature points under rotation and scale transformation. Finally, by comparing the changes in the feature points in the initial image and the monitoring image, the loosening angle of the bolt is calculated.

[0098] Next, go to step S130.

[0099] In step S130, the robot arm plans a motion path based on the re-tightening task.

[0100] In an embodiment of the present invention, the robotic arm plans a motion path based on the complex twisting task, including: establishing a robot model based on a urdf file and a Moveit planner, defining the physical properties and kinematic parameters of the robotic arm in the robot model; planning the motion path through a path planning algorithm based on the working range, speed requirements and spatial obstacles of the robotic arm; and optimizing the motion path based on an optimization algorithm.

[0101] In an embodiment of the present invention, the motion path is optimized based on an optimization algorithm, including: dividing the process of the robot arm performing the complex twisting task into k+1 stages based on calculation, and the start and end load intervals of each stage are P1:[0, Te shift1 ],P2:[Te shift1 , Te shift2 ]…P k :[Te shiftk-1 , Te shiftk ], P k+1 :[Te shift2 , T (x,y,z) ]; For each stage, in the target posture group q goal In the above example, all postures that meet the mechanical requirements are selected; the optimal posture q of the first stage P1 is calculated. p1 and the final stage P k+1 The optimal posture q p(k+1) .

[0102] In the embodiment of the present invention, the optimal posture q of the first stage P1 is calculated p1 and the final stage P k+1 The optimal posture q p(k+1) , including: Calculate q based on the following formula p1 and q p(k+1) :

[0103]

[0104] Among them, q p1 is the optimal posture in stage P1; q init is the initial posture of the robot, ||·|| is the Euclidean distance of the posture; n is the number of joints of the robot; j is the number of postures of the target posture; q j is the jth inverse solution posture; q p（k+1） P k+1 The optimal posture of the stage; m is the number of posture inverse solutions;

[0105] Taking the minimization of adjustment amplitude as the constraint function and combining the Dijkstra shortest path algorithm, the optimal posture switching path of the re-twisting process is calculated, and the optimal posture switching path of the re-twisting process is obtained from q p1 to q p（k+1） The optimal posture sequence q={q p1 , q p2 ,…, q pk , q p（k+1）}.

[0106] In an embodiment of the present invention, the method further includes:

[0107] Get the initial posture q of the robot arm init , get the spatial position p of the target bolt (x,y,z) and target re-tightening torque T (x,y,z) , and define the direction of the end grab bolt dir goal ;

[0108] According to the target position p (x,y,z) and end direction dir goal , based on the inverse kinematics operation, the target posture group q of the manipulator is obtained goal ={q g1 ,q g2 ,…,q gm}, where m is the number of inverse solutions of attitude;

[0109] Calculate the Jacobian matrix J of the robot arm for all postures in the target posture group goal ={J g1 ,J g2 ,…,J gm}, and its inverse matrix ;

[0110] The joint load Tj in each posture is calculated based on the following inverse dynamics formula goal ={ }:

[0111] = ·Te

[0112] in, is the joint load of the mth posture; is the Jacobian inverse matrix of the mth posture; Te is the load torque at the end of the robot arm;

[0113] Based on joint load Tj and joint maximum load Tj lim By comparing the load limit, the switching point Te is determined. shift ;

[0114] Get all joints of all postures curve, and compare Tj and Tj along the Te coordinate lim , define to satisfy Tj(Te)>Tj lim The point is the switching point Te shift ∈{Te shift1 ,Te shift2 ,…, Te shiftk};

[0115] Among them, Tj lim ={Tj 1 ,Tj 2 ,…,Tj n} is the maximum load torque of the robot arm 1~n joints, Te shiftk is the kth switching point.

[0116] More specifically, the present invention develops a safe and accurate robot arm motion control method for complex twisting tasks. First, an accurate kinematic model is built based on the Robot Operating System (ROS), and the physical characteristics and motion limits of the robot arm are defined in detail to ensure that the position and posture of the robot arm in the physical space can be accurately mapped. Then, the working range, load limit and spatial obstacles of the robot arm are comprehensively considered, and the path tracking of the robot arm in space is realized through a search-based path planning algorithm.

[0117] In addition, the present invention develops a robot arm re-twisting posture optimization algorithm, which calculates the joint torque in the re-twisting task based on inverse dynamics, and dynamically adjusts the posture of the robot arm to optimize its joint load, ensuring that the posture of the robot arm in the entire re-twisting process always has high precision and high safety. Finally, remote communication technology, such as TCP socket, is used to achieve real-time control of the robot arm.

[0118] The specific steps are as follows:

[0119] 1) In the initial stage, the kinematic model of the robot arm is built based on the Robot Operating System (ROS). The robot model is established through the urdf file and the Moveit planner, and the physical properties and kinematic parameters of the robot arm are defined in detail to ensure that the model can accurately map the position and posture of the robot arm in space, and lay the foundation for the subsequent control strategy development.

[0120] 2) Then formulate a robot arm control strategy including path planning algorithm and execution instructions, aiming to control the robot arm to move and perform operations according to the preset path. When formulating the control strategy, the robot arm's working range, speed requirements and spatial obstacles are fully considered to ensure collision-free and high-precision path tracking in a complex space environment. The robot arm kinematic model and path planning process are as follows Figure 5 shown.

[0121] 3) In addition, the present invention develops an optimization algorithm for the re-twisting posture of the robot arm based on inverse dynamics, aiming to improve the accuracy and safety of the re-twisting task by optimizing the mechanical properties of the robot arm, such as Figure 6 As shown. In the re-tightening task, the influence of the force on the end of the robot arm will be transmitted to each joint, which may cause joint overload, thereby affecting the safety and working accuracy of the robot arm. The present invention calculates the load torque of all joints based on inverse dynamics, and dynamically adjusts the posture of the robot arm to ensure that it is always safe during the re-tightening process.

[0122] This method divides the re-twisting process into multiple stages. At the beginning of each stage, the system evaluates the safety of the initial posture. If the posture is safe, the re-twisting of this stage is performed; if it is unsafe, all possible postures and their force conditions are calculated, and the safest posture is selected for adjustment. The specific operation is to first calculate the torque Te exerted on the end of the robot arm due to the re-twisting, and then calculate the torque Tj of each joint through inverse dynamics. Compare the relationship between the rated torque Tlim and the actual torque Tj of each joint, and define the point where the actual torque exceeds the rated torque as the switching point. According to the switching point, the posture adjustment of the robot arm is divided into multiple stages to ensure that the posture of the robot arm in each stage is within the torque safety range. This method effectively ensures the joint safety and overall accuracy of the robot arm during the re-twisting process.

[0123] The above steps 2) and 3) include the following steps:

[0124] S1: Get the initial posture q of the robot arm init , get the spatial position p of the target bolt (x,y,z) and target re-tightening torque T (x,y,z) , and define the direction of the end grab bolt dir goal ; Then according to the target position p(x,y,z) and end direction dir goal , based on the inverse kinematics operation, the target posture group q of the manipulator is obtained goal ={q g1 ,q g2 ,…,q gm}, where m is the number of inverse solutions of attitude;

[0125] S2: Calculate the Jacobian matrix J of the robot arm for all postures in the target posture group goal ={J g1 ,J g2 ,…,J gm}, and its inverse matrix ; and calculate the joint load Tj in each posture through inverse dynamics goal ={ }; The inverse dynamics calculation method is:

[0126] = ·Te

[0127] in, is the joint load of the mth posture, is the Jacobian inverse matrix of the mth posture, Te is the load torque at the end of the robot arm;

[0128] S3: Then, based on the joint load Tj and the joint maximum load Tj lim By comparing the load limit, the switching point Te is determined. shift For all poses, get the curve, and compare Tj and Tj along the Te coordinate lim , define to satisfy Tj(Te)>Tj lim The point is the switching point Te shift ∈{Te shift1 ,Te shift2 ,…, Te shiftk}.

[0129] Among them, Tj lim ={Tj 1 ,Tj 2 ,…,Tj n}, is the maximum load torque of the robot arm 1~n joints, Te shiftk is the kth switching point;

[0130] S4: The re-tightening process is divided into several stages, and the starting and ending load intervals of each stage are P1: [0, Te shift1 ],P2:[Te shift1 , Te shift2 ]…P k :[Teshiftk-1 , Te shiftk ], P k+1 :[Te shift2 , T (x,y,z) ]. For each stage, in the target pose group q goal , select all postures that meet the mechanical requirements.

[0131] S5: Calculate the optimal posture of the first and last stages. For the first stage P1, select the posture q that meets the mechanical requirements p1 ∈{q g1 ,q g2 ,…,q gn}, and calculate the optimal posture q of the first stage P1 with the minimization of the adjustment amplitude as the optimization goal p1 , similarly select the final stage P k+1 The optimal posture q p(k+1) The optimization function is as follows:

[0132]

[0133] Among them, q init is the initial posture, and ||·|| is the Euclidean distance of the posture. Through this objective function, the stage 1 posture closest to the initial posture is selected to achieve a smooth transition.

[0134] S6: Calculate the optimal twisting posture in all stages. p1 and q p(k+1) As the boundary condition, and taking the minimization of adjustment amplitude as the constraint function, combined with Dijkstra and other shortest path algorithms, the optimal posture switching path of the re-twisting process is calculated; ensuring that the posture change from the first stage to the last stage is smooth and the load is always within the safe range, and finally obtaining the optimal posture switching path from q p1 to q p(k+1) The optimal posture sequence q={q p1 , q p2 ,…, q pk , q p(k+1)};

[0135] 4) Finally, remote communication technology, such as TCP socket, is used to achieve real-time control of the robotic arm. This step ensures that the control system can send instructions to the robotic arm in a timely manner and monitor its execution, achieving rapid response and precise control. Through the connection between the computer and the robotic arm, precise control of the robotic arm's movement to the specified target point is completed.

[0136] For example, in this example, the physical characteristics and motion limits of the robot arm are described in detail through the URDF (Unified Robot Description Format) file, and the kinematic model of the robot arm is established through the Robot Operating System (ROS). This model provides a basis for the development of control strategies, ensuring that the model can accurately map the position and posture of the robot arm in space. On this basis, the robot arm's working range, load limit and other information are comprehensively considered, and the sampling-based RRT* path planning algorithm is adopted. The real-time communication between the model and the real robot arm is established to accurately guide the robot arm to the target position along the preset path.

[0137] In addition, the present invention also includes an optimization algorithm for the safe re-tightening posture of the robotic arm. The algorithm dynamically adjusts the posture of the robotic arm to ensure that the safety performance and working accuracy of the robotic arm are always maintained during the re-tightening operation, effectively improving the joint safety and overall accuracy of the robotic arm during the re-tightening process. Finally, through remote communication technology, such as TCP socket, the connection between the robotic arm and the computer terminal is realized, and real-time control of the robotic arm is achieved. This step ensures that the operator or automatic control system can send instructions to the robotic arm in a timely manner and monitor its execution, achieving rapid response and precise control, thereby improving the efficiency and safety of industrial applications.

[0138] Next, go to step S140.

[0139] In step S140, the robot arm completes the task of re-tightening the bolt according to the motion path.

[0140] In an embodiment of the present invention, the method further includes:

[0141] When the robot arm completes the task of re-tightening the bolt according to the motion path, the bolt loosening angle is updated in real time:

[0142] The bolt loosening angle during the re-tightening task is calculated based on the following formula:

[0143]

[0144] in, is the re-tightening angle; is the initial tightening angle; The amount of adjustment caused by the change in loose angle.

[0145] More specifically, the present invention designs a re-tightening system and develops a re-tightening information update mechanism to ensure the safety and reliability of the re-tightening operation. The re-tightening system integrates five core modules: a connection system, an image acquisition system, a looseness detection system, a control system, and a re-tightening system to achieve automatic detection and re-tightening of loose bolts. In addition, the present invention develops a re-tightening information update strategy to improve detection efficiency and re-tightening accuracy by dynamically synchronizing the robot arm operation information and re-tightening information.

[0146] The detailed steps are as follows:

[0147] 1) The designed system consists of five core components, such as Figure 7 As shown in the figure. The connection system involves various bolt fastening structures, providing input signals and control strategies. The image acquisition system uses industrial cameras to collect bolt image data and monitor images. The looseness detection system is responsible for identifying bolt features and detecting their looseness status. The control system performs precise signal control on the movement and re-tightening of the robot arm. The automatic re-tightening system drives the robot arm to the specified position and performs re-tightening to the predetermined angle. The synergy of these five systems realizes the automatic detection and re-tightening of bolt looseness, improving efficiency and safety in industrial applications.

[0148] 2) In view of the dynamic and rapid characteristics of the bolt loosening process, this paper proposes a re-tightening information update strategy to solve the real-time and accuracy problems of bolt loosening detection and re-tightening operations, such as Figure 8 As shown in the figure. When the bolt is loose, its loose angle will increase dynamically in a short period of time, which requires that the image acquisition frequency must be able to update the image information of the bolt in time to capture the latest loose state. In addition, during the movement of the robot arm from the initial position to the bolt position, the dynamic change of the loose angle requires that the re-tightening information must be updated in real time to ensure that the re-tightening operation is synchronized with the actual loose state of the bolt. The re-tightening information update strategy realizes the dynamic interaction between the loose detection module and the operation control module by establishing real-time communication between the two modules.

[0149] For example, the strategy adapts to the immediate changes in bolt loosening by increasing the image acquisition frequency and adjusting the loosening angle in real time. In the case where no loosening is detected, the loosening detection module periodically detects the bolt status at a preset initial frequency. Once the bolt loosening is detected, the system will automatically increase the image acquisition frequency to capture the loosening status of the bolt in real time. At the same time, the operation control module is responsible for precise motion control of the robot arm and real-time monitoring of the position of the robot arm. When the robot arm reaches the predetermined target point, the dynamic communication mechanism is used to ensure that the robot arm can receive the latest loosening data, calculate the accurate loosening angle, and perform precise re-tightening operations to ensure the tightening state of the bolt. Through this real-time monitoring and dynamic adjustment method, this strategy optimizes the utilization of information acquisition resources and significantly improves the accuracy of re-tightening operations, thereby ensuring the reliability and safety of bolt connections.

[0150] The above algorithm includes the following steps:

[0151] S1: Image acquisition and frequency adjustment.

[0152] Initially, the looseness detection module uses a preset initial frequency Capture the image of the bolt state. For example, assuming f init Once per second. When a loose bolt is detected, the image acquisition frequency is automatically increased to , for example, collecting images multiple times per second to more accurately capture rapid changes in looseness. The specific frequency switching is achieved through the following rules:

[0153]

[0154] in > to ensure that the dynamically changing loose angle can be monitored in time.

[0155] S2: Instant update of the loosening angle is synchronized with the re-tightening operation.

[0156] The looseness detection module and the operation control module are synchronized through a real-time communication mechanism. When the looseness detection module detects looseness and updates the looseness angle, the control module immediately receives this information and adjusts the re-tightening strategy. Based on the latest looseness angle data, the control module recalculates the re-tightening angle. , ensuring that the re-tightening operation is synchronized with the actual loosening state of the bolt. The process can be described by the following equation:

[0157]

[0158] in, is the re-tightening angle, is the initial tightening angle, The amount of adjustment caused by the change in loose angle.

[0159] 3) Finally, the bolt tightening strategy is combined to realize automatic re-tightening of the bolts. The tightening strategy includes a series of steps to ensure that the bolts are re-tightened according to the predetermined torque and angle requirements. This strategy refers to the existing bolt tightening methods, including torque control, torque and angle monitoring combined with other tightening control methods to meet the tightening requirements of bolts of different levels. Finally, by precisely controlling the movement of the robot arm and the rotation of the re-tightening mechanism, the re-tightening mechanism is driven to achieve re-tightening of the bolts at a specific angle.

[0160] For example, designing a re-tightening system and developing a re-tightening information update mechanism is as follows:

[0161] In this example, an integrated system including five core parts, connection, image acquisition, looseness detection, control and automatic re-tightening, was built to realize the automatic detection and re-tightening of bolt loosening. Figure 7 As shown. Then, a re-tightening information update strategy was developed according to the application requirements of the re-tightening task. After detecting the loose bolt, the detection frequency was automatically increased to quickly identify the loose situation, and the re-tightening information was updated in real time to ensure the timeliness and accuracy of the re-tightening operation. Finally, by precisely controlling the movement of the robot arm and the rotation of the re-tightening mechanism, the re-tightening mechanism was driven to achieve re-tightening of the bolt at a specific angle.

[0162] The beneficial effects of this technical solution are as follows:

[0163] 1. Innovative integration of looseness detection and automatic re-tightening method: This invention proposes a method for detecting and automatically re-tightening bolts by combining computer vision and automatic control technology. This method can realize rapid detection and accurate re-tightening of loose bolts, replacing the traditional "signal detection + manual re-tightening" method, significantly improving the efficiency and safety of industrial production, and has important value in promoting technological progress in the field of industrial automation.

[0164] 2. Robot arm re-twisting posture optimization algorithm: In view of the influence of the force on the end of the robot arm on the accuracy and safety in the re-twisting task, the present invention provides a dynamic optimization algorithm. The algorithm realizes a safer and more efficient robot arm re-twisting process than the existing technology by introducing multi-solution optimization, dynamic joint load management, staged processing of the re-twisting process and the shortest path algorithm. Compared with the traditional method, the algorithm not only considers the mechanical constraints and minimizes the adjustment amplitude in the inverse kinematics solution, but also adjusts the load through real-time feedback to avoid the risk of overload. The re-twisting process is divided into multiple stages, which optimizes the smoothness of posture switching, reduces unnecessary vibration and energy consumption, improves the adaptability and stability of the algorithm, and significantly enhances the execution effect of the robot arm in complex tasks.

[0165] 3. Real-time update strategy for re-tightening information: The present invention aims at the real-time and accuracy issues of bolt loosening detection and re-tightening operations, and proposes a re-tightening strategy based on dynamic information update. Compared with the prior art, traditional methods usually rely on static image acquisition frequency and fixed re-tightening strategies, and fail to fully consider the rapid changes and dynamic characteristics of the bolt loosening process. By increasing the image acquisition frequency and updating the loosening angle in real time, the present invention adjusts the re-tightening strategy in time when loosening occurs, ensuring that the re-tightening operation is synchronized with the actual loosening state of the bolt. At the same time, a dynamic communication mechanism is used to realize real-time interaction between the loosening detection module and the operation control module, which optimizes the resource utilization of information acquisition, improves the accuracy of the re-tightening operation, and ultimately ensures the reliability and safety of the bolt connection. This innovative solution effectively solves the problems of response delay and insufficient operation accuracy in the prior art, and significantly improves the intelligence and automation level of the re-tightening process.

[0166] The present invention demonstrates the workflow and application scenarios of a model-based bolt loosening detection and automatic re-tightening method through specific embodiments. The method first uses deep learning technology to build a bolt recognition model, which can accurately identify and locate bolts in complex industrial environments. Then, through advanced image processing technology, the loose state of the bolt is detected, and the loose angle is calculated to provide accurate data support for the re-tightening operation. On this basis, the present invention combines automatic control technology to realize the automatic re-tightening operation of the bolt. By accurately controlling the motion path and re-tightening force of the robotic arm, the safety and reliability of the re-tightening operation are ensured, thereby optimizing the assembly performance of the bolt connection system. The application of this method significantly improves the reliability of bolt connections and reduces the safety risks caused by loose bolts, which is of great practical significance for improving the level of industrial automation.

[0167] In summary, the bolt loosening failure detection and automatic re-tightening method of the present invention effectively solves the low efficiency problem of relying on manual detection and re-tightening in the prior art. Through automated detection and re-tightening, not only the production efficiency is improved, but also the safety risks caused by loose bolts are reduced. At the same time, through the safety posture optimization algorithm and the real-time update technology of re-tightening information, the present invention significantly improves the safety, accuracy and reliability of industrial bolt re-tightening operations, which has important practical significance for promoting the development of industrial automation and intelligentization.

[0168] In summary, the present invention provides a model-based method for detecting and automatically re-tightening bolt loosening. Therefore, it has wide application potential in the fields of mechanical engineering, aerospace, automotive engineering, etc. Through the efficient recognition of the deep learning model and the precise motion control of the robot arm, the present invention can realize the automatic detection and re-tightening of bolt loosening, significantly improving the efficiency and safety in industrial production. In addition, the method can also be extended to steel structure monitoring in civil engineering, maintenance systems for intelligent networked vehicles, precision assembly in electronic product manufacturing, and automated production in the toy and hardware industries, thereby ensuring the reliability and safety of bolt connections in various industries. Therefore, the present invention has broad application prospects and can promote the further development of industrial automation and intelligence.

[0169] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for detecting and automatically tightening bolt looseness based on model drive, characterized in that: The method comprises the following steps: Real-time image acquisition of the device under test; Based on a pre-trained deep learning bolt recognition model, the bolt position in the image is identified, and the loosening angle of the bolt is calculated; wherein, when the loosening angle reaches a preset threshold, a re-tightening task is established for the robot arm based on the loosening angle; The robot arm plans a motion path based on the re-tightening task; the robot arm plans a motion path based on the re-tightening task, including: establishing a robot model based on a urdf file and a Moveit planner, defining physical properties and kinematic parameters of the robot arm in the robot model; planning the motion path through a path planning algorithm; optimizing the motion path based on an optimization algorithm; The robot arm completes the task of re-tightening the bolt according to the motion path; Optimizing the motion path based on an optimization algorithm includes: Based on the calculation, the process of the robot arm performing the complex twisting task is divided into k+1 stages, and the start and end load interval of each stage is P1:[0,Te shift1 ],P2:[Te shift1 ,Te shift2 ]…P k :[Te shiftk-1 ,Te shiftk ], P k+1 :[Te shiftk ,T (x,y,z) ]; For each stage, in the target posture group q goal In , select all postures that meet the mechanical requirements; The method further comprises: Get the initial posture q of the robot arm init , get the spatial position p of the target bolt (x,y,z) and target re-tightening torque T (x,y,z) , and define the direction of the end grab bolt dir goal ; According to the target position p (x,y,z) and end direction dir goal , based on the inverse kinematics operation, the target posture group q of the manipulator is obtained goal = {q g1 ,q g2 ,…,q gm }, where m is the number of inverse solutions of attitude; Calculate the Jacobian matrix J of the robot arm for all postures in the target posture group goal ={J g1 ,J g2 ,…,J gm }, and its inverse matrix The joint load Tj in each posture is calculated based on the following inverse dynamics formula goal ={Tj g1 ,Tj g2 ,…,Tj gm }: Among them, Tj gm is the joint load of the mth posture; is the Jacobian inverse matrix of the mth posture; Te is the load torque at the end of the robot arm; Based on joint load Tj and joint maximum load Tj lim By comparing the load limit, the switching point Te is determined. shift ; Get the Tj-Te curves of all joints in all postures, and compare Tj with Tj along the Te coordinate lim , define to satisfy Tj(Te)>Tj lim The point is the switching point Te shift ∈{Te shift1 ,Te shift2 ,…,Te shiftk }; Among them, Tj lim ={Tj 1 ,Tj 2 ,…,Tj n } is the maximum load torque of joints 1 to n of the robot arm, Te shiftk is the kth switching point.

2. The method for detecting and automatically tightening bolt looseness based on model drive according to claim 1 is characterized in that: Pre-trained deep learning bolt recognition model, including: The target detection network of the deep learning bolt recognition model is set to YOLOv8, the backbone network is set to CSPNet, and the feature fusion network is set to FPN; Collect bolt images and build basic data sets; Performing data enhancement on the basic data set to obtain an extended data set; Annotating the position of the bolt in each image in the extended dataset based on Labelbox; Dividing the extended data set into a training set, a test set, and a validation set; A deep learning bolt recognition model is trained based on the extended data set, and the model parameters are optimized using stochastic gradient descent.

3. The model-driven bolt loosening detection and automatic re-tightening method according to claim 2 is characterized in that: The step of performing data enhancement on the basic data set includes: Improve image contrast through histogram equalization, grayscale adjustment, denoising and filtering.

4. The model-driven bolt loosening detection and automatic re-tightening method according to claim 1 is characterized in that: The calculation of the loosening angle of the bolt comprises the following steps: The image is partially cropped based on the identified bolt position to obtain the bolt image; enhancing visibility of feature points of the bolt image; Feature point extraction and matching of the enhanced bolt image are performed based on the ORB feature point matching algorithm; Compare the changes of feature points in the initial image and the real-time image to calculate the loosening angle of the bolt; Calculate the optimal posture q of the first stage P1 p1 and the final stage P k+1 The optimal posture q p(k+1) .

5. The model-driven bolt loosening detection and automatic re-tightening method according to claim 1 is characterized in that: Calculate the optimal posture q of the first stage P1 p1 and the final stage P k+1 The optimal posture q p(k+1) ,include: Calculate q based on the following formula p1 and q p(k+1) : Among them, q p1 is the optimal posture in stage P1; q init is the initial posture of the robot, ||·|| is the Euclidean distance of the posture; n is the number of joints of the robot; j is the number of inverse solutions of the current posture; q j is the jth inverse solution posture; q p(k+1) P k+1 The optimal posture of the stage; m is the number of posture inverse solutions; Taking the minimization of adjustment amplitude as the constraint function and combining the Dijkstra shortest path algorithm, the optimal posture switching path of the re-twisting process is calculated, and the optimal posture switching path of the re-twisting process is obtained from q p1 to q p(k+1) The optimal posture sequence q = {q p1 ,q p2 ,…,q pk ,q p(k+1) }.

6. The model-driven bolt loosening detection and automatic re-tightening method according to claim 1 is characterized in that: The method further comprises: When the loosening angle of the bolt does not reach a preset threshold, image acquisition is performed at a first frequency; When the loosening angle of the bolt reaches a preset threshold, image acquisition is performed at a second frequency.

7. The model-driven bolt loosening detection and automatic re-tightening method according to claim 1 is characterized in that: The method further comprises: When the robot arm completes the task of re-tightening the bolt according to the motion path, the bolt loosening angle is updated in real time: The bolt loosening angle during the re-tightening task is calculated based on the following formula: i tight (t)=θ init +Δθ tight (t) Among them, θ tight (t) is the re-tightening angle; θ init is the initial tightening angle; Δθ tight (t) is the adjustment amount caused by the change in loose angle.

Images