A method and system for identifying loose bolts based on image feature point matching

Abstract

The present invention discloses a method and system for identifying loose bolts based on image feature point matching, comprising the steps of: obtaining two frames of initial bolt group images at different times; performing perspective transformation according to row-column ratio; performing image cutting and super-resolution reprocessing; performing semantic segmentation on bolt sub-images; matching and pairing the segmented fastening bolts at different times; using SIFT algorithm to generate the first feature descriptor and the second feature descriptor of each fastening bolt in the matching pair; matching the first feature descriptor and the second feature descriptor through a matcher, calculating the homography matrix, and determining the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image by transforming the homography matrix. The method for identifying loose bolts based on image feature point matching provided by the present invention solves the technical problem that the existing method for identifying loose bolts based on computer vision can only identify loose rotation angles within a range of 60°.

Description

Technical Field

[0001] The present invention relates to the technical field of bolt loosening monitoring, and in particular to a bolt loosening identification method and system based on image feature point matching. Background Art

[0002] Bolt connections are widely used in civil engineering and their large number makes them an important component. Moreover, loosening of bolts at key locations may lead to serious safety risks. In the prior art, bolt loosening detection mainly relies on manual maintenance. However, due to the large scale of the project, high labor costs, subjectivity in the maintenance process, and the danger of maintenance conditions, there are problems such as low detection efficiency, large errors, and easy maintenance safety accidents.

[0003] At present, the research on bolt loosening detection is mainly divided into three categories: the first category is the method based on tapping, the second category is the method based on sensor, and the third category is the method based on computer vision. There are two main types of bolt loosening detection methods based on tapping. The first is the traditional manual tapping method, but the traditional manual tapping method requires a lot of manpower input, low efficiency, and is easily affected by subjective factors; the second is the acoustic wave recognition method based on sensors and combined with deep learning technology. The vibration-guided sound reconstruction method is used to achieve tapping-based bolt loosening identification. Since the acoustic wave recognition method needs to record a large amount of sound data and is easily affected by the noise of the surrounding environment during the detection, it is easy to cause misjudgment. In addition, acoustic wave recognition may cause damage to the structure because sufficient tapping force is required to make the structure produce a clear and full sound for analysis. The sensors used to detect bolt loosening mainly include strain gauges. Under complex working conditions, the arrangement of sensors may become quite difficult, requiring precise installation and adjustment. The detection of looseness of structural bolt groups requires the deployment of a large number of sensors. Secondly, in complex working environments, sensors are often easily damaged, which directly affects the accuracy and reliability of measurement, limiting the widespread application of sensor-based bolt loosening detection methods. Computer vision-based methods can be divided into two main categories: one is to extract the edge line of the bolt and compare it with the front and rear edge lines to obtain the rotation angle; the other is to identify the bolt rotation angle through feature extraction and matching methods, taking two images of the monitored steel joint during different inspections, and then judging whether the bolt has rotated by whether the two images can be aligned.

[0004] The current research on loose bolt identification methods based on computer vision has the following defects: First, when extracting the edge line of the bolt, if the edge features of the bolt are not obvious, it will be difficult to accurately extract its edge line, which may lead to inaccurate detection results, and the detection range of such methods is limited to [0°, 60°]; Second, the correction of the shooting angle usually requires manual assistance, such as drawing auxiliary lines, character markings, or installing auxiliary structures such as rectangular gaskets, which increases the complexity of the operation and dependence on manual intervention. In addition, most existing studies have not fully considered the actual application. When segmenting a single bolt sub-image from an image, low resolution may have an adverse effect on the calculation of the loose angle, and ensuring accuracy is very important in this regard. A task worthy of attention; third, usually, the features or identifications required for the front and rear bolt comparison need to be manually marked on the bolts, or rely on the characters on the bolts for matching. This process is cumbersome and relies on manual operation. In addition, in actual engineering applications, the characters on the bolts may not be clear, covered by the paint on the surface of the bolts, or there may be no characters at all, making automated detection more difficult, or it is necessary to continuously record videos to track and record the changes in the bolts, which is difficult to achieve; fourth, the existing methods are difficult to accurately identify the preload state of the bolts, and the change value of the bolt preload is an important indicator for determining the looseness of the bolts, which is very important information in determining the bolt state in key parts of the structure.

[0005] In summary, among the existing conventional means of detecting bolt loosening, the detection method based on manual tapping is inefficient and easily affected by the surrounding environmental noise, leading to misjudgment; the detection method based on sensors requires a large number of sensors, which are difficult to deploy in large monitoring structures, and the sensors have a short service life and are easily damaged; the method based on traditional computer vision is not good at image processing and is difficult to accurately identify, and can only identify loose angles within a range of 60°.

[0006] In view of this, it is necessary to propose a bolt loosening identification method and system based on image feature point matching to solve or at least alleviate some of the above-mentioned defects. Summary of the invention

[0007] The main purpose of the present invention is to provide a method and system for identifying loose bolts based on image feature point matching, aiming to solve the technical problem that the computer vision-based loose bolt identification method in the prior art can only identify loose angles within a range of 60°.

[0008] To achieve the above-mentioned purpose, the present invention provides a method for identifying loose bolts based on image feature point matching, comprising the following steps: S10, obtaining two frames of initial bolt group images at different times, wherein the initial bolt group images include a plurality of fastening bolts; S20, determining the row-column ratio of the fastening bolts arranged in the initial bolt group image, performing perspective transformation according to the row-column ratio to obtain a corrected bolt group image after perspective correction; performing image cutting and super-resolution reconstruction processing on each frame of the corrected bolt group image to obtain a plurality of bolt sub-images corresponding to each frame; S30, performing semantic segmentation on the bolt sub-image, eliminating non-target areas, determining target areas to segment out the fastening bolts; S40, matching and pairing the segmented fastening bolts at different times to determine a target bolt matching pair, and the target bolt matching pair The invention is to match the same fastening bolt in the front image and the rear image; the first feature descriptor and the second feature descriptor of each fastening bolt in the matching pair are generated by using the SIFT algorithm, wherein the first feature descriptor is the feature description of the feature points of the target area in the front image, and the second feature descriptor is the feature description of the feature points in the target area in the rear image. The SIFT algorithm generates the feature description of the feature points by means of scale space extreme value detection, key point positioning, direction allocation and feature point description; the first feature descriptor and the second feature descriptor are matched by the Brutal-Force matcher, and the homography matrix is ​​calculated. By transforming the homography matrix, the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image is determined.

[0009] Furthermore, in step S40, the homography matrix satisfies the rigid body transformation matrix in the form of:

[0010]

[0011] in,[ ] represents the horizontal displacement, vertical displacement and overall translation of the projection transformation between the two images; [ ] represents the rotation and scaling effects of the projection transformation; [ ] is used to represent the perspective effect and ensure it is an affine transformation. , are the horizontal and vertical translations respectively Expressed as a rotation angle.

[0012] Further, in step S20, the method for obtaining the corrected bolt group image is specifically as follows: using the trained YOLOv8-P6 algorithm to perform bolt target detection on the initial bolt group image, and obtaining two sets of bolt detection data before and after, wherein the bolt detection data includes bolt category, center coordinates and size of the detection frame, the bolt category includes outer sleeve bolt, embedded bolt and missing bolt, the outer sleeve bolt corresponds to an inner round screw, the center coordinates of the detection frame include the center coordinates x of the detection frame and the center coordinates y of the detection frame, and the size of the detection frame includes the length l of the detection frame and the width w of the detection frame; traversing the The center coordinates of the detection frame corresponding to each fastening bolt in a set of bolt detection data are calculated, the Euclidean distance between the center coordinates of any two detection frames is calculated, all Euclidean distances in the bolt detection data are traversed to obtain, and the four vertices of the bolt group are determined based on the two largest sets of Euclidean distances; the four vertices are connected into a quadrilateral in sequence, and the number of rows and columns of the outermost fastening bolts in the initial bolt group image is determined according to the number of detection frames passed by the four sides of the quadrilateral; the row-column ratio of the bolt arrangement in the initial bolt group image is determined; and a perspective transformation is performed according to the row-column ratio to obtain a corrected bolt group image after perspective correction.

[0013] Furthermore, the specific method of determining the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image is as follows: if it is an outer sleeve bolt, the rotation angles of the outer sleeve bolt and the inner circle screw in the target bolt matching alignment are respectively calculated, and the relative rotation angle calculated based on the rotation angles of the outer sleeve bolt and the inner circle screw is used as the rotation angle; if it is an embedded bolt, the rotation angle of the embedded bolt is calculated as the rotation angle.

[0014] Furthermore, the specific method of performing image cutting and super-resolution reconstruction processing on the corrected bolt group image to obtain multiple bolt sub-images corresponding to each frame is as follows: performing target detection on the corrected bolt group image to obtain a detection frame, cutting the bolt sub-image detected in the corrected bolt group image according to the coordinates of the detection frame to obtain the bolt sub-image, and using the SRGAN algorithm to perform super-resolution reconstruction on the cut bolt sub-image to improve the clarity and details of the image.

[0015] Further, in step S40, the specific steps of the SIFT algorithm generating a feature description of a feature point by means of scale space extreme value detection, key point positioning, direction allocation and feature point description are as follows: S421, applying Gaussian kernels with different standard deviations to the initial bolt group image for convolution to generate a series of blurred images with different blurring degrees, and obtaining the scale space. Each blurred image represents the blurring degree at different scales. The Gaussian convolution kernel is shown in the following formula:

[0016]

[0017] in , Represent the horizontal and vertical coordinates of the pixel points respectively. , Represent the length and width of the initial bolt group image, The parameters are the variance (Gaussian radius), The value ranges from 1.2 to 1.6;

[0018] S422, subtract two adjacent images with high blur levels in the scale space to generate a Gaussian difference image; S423, mark candidate key points based on the Gaussian difference image; S424, perform threshold screening on the detected candidate key points, remove candidate key points and edge response points with low contrast, and obtain a preliminary detected key point set as shown in the following formula:

[0019]

[0020] Among them, T is the threshold, n is the number of images to be extracted features, is the pixel value;

[0021] S425, on the set of key points initially detected, accurately locate the position and scale of the key points by fitting a second-order Gaussian function, and determine the exact position of the key points; S426, determine the main direction of the key points to ensure the rotation invariance of the key points; S427, perform mathematical feature description on the key points to obtain feature description of the feature points, wherein the following formula is used

[0022]

[0023]

[0024] Processing is performed to obtain feature descriptions (descriptors) of feature points, where: is the L2 norm of the feature descriptor, is the feature descriptor vector, is a component of the feature descriptor vector, n is the vector dimension, is the normalized feature descriptor vector.

[0025] Furthermore, T is set to 0.5, or T is the local threshold obtained by adjusting the adaptive Gaussian threshold method. According to the local pixel distribution of each small block, a local Gaussian weight is calculated, and the threshold is adjusted by the adaptive Gaussian threshold method, as shown in the formula,

[0026]

[0027] in Representing pixels At the local threshold, is the average gray value of pixels in adjacent local areas, is the standard deviation of pixels in adjacent local regions.

[0028] Furthermore, in step S40, the specific method for matching the first feature descriptor and the second feature descriptor is as follows: compare the first feature descriptor and the second feature descriptor one by one, and calculate the square of the difference of each corresponding component of the feature vector of the two feature points one by one; add the squares of the differences of all corresponding components, and take the square root thereof to calculate the vector distance; determine that the feature points whose vector distance is less than a preset distance threshold are suitable matching points; wherein, a dynamic adjustment method is used to determine the preset distance threshold, and an initial distance threshold is set. If the number of feature point matching pairs is less than the minimum number of feature points of the two images, the preset distance threshold is increased by a preset step size s so that the number of matching points meets the number requirement of the matcher; after the number of matching points meets the number requirement of the matcher, the RANSAC algorithm is used to screen the matching points, and then the wrong matching pairs are eliminated, wherein the filtering threshold of the RANSAC algorithm is dynamically adjusted for screening, and the filtering threshold is gradually reduced until the obtained homography matrix satisfies the rigid body transformation matrix after matrix transformation. in the form of.

[0029] Further, the preload change value of the fastening bolt is determined based on the rotation angle of the fastening bolt and the rotation angle-preload change relationship data.

[0030] The present invention also provides a bolt loosening identification system based on image feature point matching, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the above method is implemented when the processor executes the computer program.

[0031] Compared with the prior art, the bolt loosening identification method based on image feature point matching provided by the present invention has the following beneficial effects:

[0032] The present invention provides a method for identifying loose bolts based on image feature point matching. First, two frames of initial bolt group images at the time of front and back interlocking are obtained; perspective transformation is performed according to the row-column ratio of the arrangement of the fastening bolts to obtain a corrected bolt group image after perspective correction, and image cutting is performed to obtain each bolt sub-image in the front and back corrected bolt group images; semantic segmentation is performed after super-resolution reconstruction to determine the target area that needs to be feature matched; matching and pairing are performed in the target area to complete the matching of the same fastening bolt in the front image and the back image; the first feature descriptor and the second feature descriptor of the fastening bolt in the matching pair are generated by using the SIFT algorithm; and the Bru The Brutal-Force matcher is used to match the first feature descriptor and the second feature descriptor, and the homography matrix is ​​calculated. By transforming the homography matrix, the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image is determined. The SIFT algorithm is combined with the Brutal-Force matcher, and the bolt rotation angle is calculated based on the SIFT-Bolt method, which realizes the detection of the bolt's 360° full cycle, breaking through the bottleneck of the [0°, 60°] detection range, and solving the technical problem that the existing computer vision-based bolt loosening identification method can only identify the loose rotation angle within 60°. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0034] Figure 1 This is one of the flow charts of a method for identifying loose bolts based on image feature point matching in one embodiment of the present invention;

[0035] Figure 2 : is a model architecture diagram of YOLOv8-P6 in one embodiment of the present invention;

[0036] Figure 3 It is a YOLOv8-BoltSeg model architecture diagram in one embodiment of the present invention;

[0037] Figure 4 Schematic diagrams of three typical detection target objects in an embodiment of the present invention, wherein a is a schematic diagram of an outer sleeve bolt, b is a schematic diagram of an embedded bolt, and c is a schematic diagram of a bolt missing;

[0038] Figure 5is a schematic diagram of image correction in one embodiment of the present invention, wherein a is a schematic diagram of finding two maximum Euclidean distances, b is a schematic diagram before perspective transformation, and c is a schematic diagram after perspective transformation according to row-column ratio;

[0039] Figure 6 1 is a schematic diagram of the semantic segmentation principle in an embodiment of the present invention, wherein a is a schematic diagram of the inner circle cutting principle, b is a schematic diagram of the outer hexagonal cutting principle in one form, and c is a schematic diagram of the outer hexagonal cutting principle in another form;

[0040] Figure 7 is a schematic diagram of bolt sub-graph cutting in one embodiment of the present invention, wherein a is one frame of the calibrated bolt group image, and b is a schematic diagram of the bolt after cutting;

[0041] Figure 8 This is a second flow chart of a method for identifying loose bolts based on image feature point matching in one embodiment of the present invention;

[0042] Fig. 9 yes Figure 8 Schematic diagram of the Sift-Bolt algorithm flow;

[0043] Fig.10 It is a schematic diagram of a bolt loosening identification unit based on image feature point matching in one embodiment of the present invention.

[0044] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0048] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0049] Please refer to the attached Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 as well as Fig. 9 The present invention provides a method for identifying loose bolts based on image feature point matching, comprising the following steps: S10, obtaining two frames of initial bolt group images at different times, wherein the initial bolt group images include a plurality of fastening bolts; S20, determining the row-column ratio of the arrangement of the fastening bolts in the initial bolt group image, performing perspective transformation according to the row-column ratio to obtain a corrected bolt group image after perspective correction; performing image cutting and super-resolution reconstruction processing on each frame of the corrected bolt group image to obtain a plurality of bolt sub-images corresponding to each frame; S30, performing semantic segmentation on the bolt sub-image, eliminating non-target areas, determining target areas to segment out the fastening bolts; S40, matching and pairing the segmented fastening bolts at different times to determine a target bolt matching pair, wherein the target bolt matching pair is a pair of images in the previous image and a pair of images in the subsequent image. Matching of the same fastening bolt in the picture; using the SIFT algorithm to generate the first feature descriptor and the second feature descriptor of each fastening bolt in the matching pair, wherein the first feature descriptor is the feature description of the feature points of the target area in the previous image, and the second feature descriptor is the feature description of the feature points of the target area in the subsequent image, and the SIFT algorithm generates the feature description of the feature points through the methods of scale space extreme value detection, key point positioning, direction allocation and feature point description; using the Brutal-Force matcher, the first feature descriptor and the second feature descriptor are matched, and the homography matrix is ​​calculated, and by transforming the homography matrix, the rotation angle of the corresponding fastening bolt in the subsequent image relative to the fastening bolt in the previous image is determined.

[0050] The bolt loosening identification method based on image feature point matching provided by the present invention first obtains two frames of initial bolt group images at the time of front and back disconnection; performs perspective transformation according to the row-column ratio of the arrangement of the fastening bolts to obtain the corrected bolt group image after the perspective correction, and performs image cutting to obtain each bolt sub-image in the front and back corrected bolt group images; performs semantic segmentation after super-resolution reconstruction to determine the target area that needs to be feature matched; performs matching and pairing in the target area to complete the matching of the same fastening bolt in the front image and the back image; generates the first feature descriptor and the second feature descriptor of the fastening bolt in the matching pair by using the SIFT algorithm; and obtains the bolt sub-images in the front and back corrected bolt groups by using the Bruta l-Force matcher, matches the first feature descriptor and the second feature descriptor, calculates the homography matrix, and determines the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image by transforming the homography matrix. The SIFT algorithm is combined with the Brutal-Force matcher, and the bolt rotation angle is calculated based on the SIFT-Bolt method, realizing the detection of the 360° full cycle of the bolt, breaking through the bottleneck of the [0°, 60°] detection range, and solving the technical problem that the existing computer vision-based bolt loosening identification method can only identify the loose rotation angle within the range of 60°.

[0051] It can be understood that the two frames of initial bolt group pictures are photos of the bolt group taken at the previous moment and the next moment, and the shooting angles at the next moment and the previous moment may be consistent or inconsistent; the bolt group can be composed of fastening bolts arranged in two rows and three columns, or a bolt group arranged in three rows and four columns, or a bolt group composed of other rows and other columns. The bolt group is arranged according to the actual project and will not be elaborated here.

[0052] It can be understood that in order to further adapt to the calculation of the bolt rotation angle based on the SIFT-Bolt method and more accurately calculate the bolt rotation angle, the above steps are further optimized in the present invention.

[0053] It can be understood that in a preferred embodiment of the present invention, the SIFT algorithm (Scale-invariant feature transform, i.e., scale-invariant feature extraction method) is used to generate the first feature descriptor and the second feature descriptor of each fastening bolt in the matching pair; and the first feature descriptor and the second feature descriptor are matched by the Brutal-Force matcher.

[0054] The present invention adopts the integration of multiple technologies such as computer vision, neural network, deep learning, target detection, perspective correction, super-resolution reconstruction, etc.

[0055] Among them, computer vision is a kind of computer technology and algorithms that enable computer systems to understand and interpret the content of images or videos; it involves tasks such as extracting information from images or videos, identifying objects, detecting motion, and measuring the size and shape of objects; computer vision usually uses digital image processing and pattern recognition technology, including feature extraction, pattern matching, machine learning, and deep learning methods.

[0056] Among them, a neural network is a computational model that mimics the way the human nervous system works and is used in the fields of machine learning and artificial intelligence. It consists of a large number of artificial neurons (or nodes), which transmit information through connections. The connections have weights that can be adjusted to affect the transmission of information in the network. Neural networks are usually divided into multiple layers, including input layers, hidden layers, and output layers. The input layer receives data from the external environment; the output layer generates predictions or results of the model, and the hidden layer performs information processing between the input and output. Each node performs weighted processing on the input data and generates output through an activation function. The neural network adjusts the connection weights by learning the training data set to optimize the performance of the model.

[0057] Among them, deep learning is a branch of machine learning that attempts to achieve abstract learning of data by simulating the neural network structure of the human brain. The core of deep learning is to learn the feature representation of data through multi-level neural networks, so as to achieve efficient processing and analysis of complex data; each layer of these neural networks will perform some transformations and extract features on the data, gradually forming a more abstract and advanced representation of the input data.

[0058] Among them, object detection is a task in the field of computer vision, which aims to identify specific objects in images or videos and determine their locations. Unlike simple image classification tasks, object detection requires determining the bounding box (detection box) of the object and classifying it into predefined categories.

[0059] Among them, the detection box is a common representation in the target detection task, which is used to identify the location of the target object in the image. It is usually a rectangular box consisting of four boundaries, which is used to surround the target object.

[0060] Among them, perspective correction is an image processing technology used to correct the perspective or observation angle of objects in an image so that they appear straight or aligned with a specific angle; perspective correction is commonly used in applications such as image correction, three-dimensional reconstruction, and virtual reality.

[0061] Among them, super-resolution reconstruction is an image processing technology that aims to reconstruct high-resolution images from low-resolution images. In image processing, sometimes due to equipment limitations or other factors, we can only obtain low-resolution images, and super-resolution reconstruction technology can use some algorithms and techniques to improve the resolution of the image, making it look clearer and richer in details.

[0062] Among them, backpropagation is an optimization algorithm for training neural networks. It is a variant of the gradient descent algorithm. The backpropagation algorithm calculates the gradient of each parameter in the neural network to the loss function, and then updates the parameters in the opposite direction of the gradient, thereby gradually reducing the loss function and realizing network training. The back propagation algorithm mainly includes the following steps: 1. Forward Propagation: Pass the input data through the neural network and calculate it layer by layer until the output result is obtained. In this process, the input of each layer is processed by weighting and activation function to obtain the output of the next layer; 2. Calculate the loss function: Compare the network output with the true label and calculate the value of the loss function to evaluate the accuracy of the model prediction; 3. Backward Propagation of Gradients: Starting from the output layer, calculate the gradient of each parameter to the loss function layer by layer according to the chain rule. This process can be achieved by calculating the error gradient of each layer and propagating it to the previous layer; Parameter Update: According to the calculated gradient, update each parameter in the opposite direction of the gradient to reduce the loss function. The gradient descent algorithm or its variants are usually used for parameter update; 5. Iterative Process: The back propagation algorithm usually requires multiple iterations. Each iteration is to forward and backward propagate a batch of data (a mini-batch) through the network, and then update the parameters according to the gradient until the stopping condition is reached (such as reaching the maximum number of iterations or the loss function converges). During the back propagation process, the gradient decreases layer by layer and eventually tends to zero, resulting in the inability to effectively update the parameters close to the input layer, which in turn affects the training effect of the network.

[0063] Among them, momentum gradient random descent can be more stable and efficient when updating parameters, which helps to jump out of local minima and accelerate convergence, while also reducing the oscillation of parameter updates; the momentum gradient descent algorithm usually converges faster than the ordinary stochastic gradient descent algorithm and can train deep neural networks more effectively.

[0064] Among them, Brute-ForceMatcher is a feature matching method commonly used in the field of computer vision. It is used to find similar feature points in two images. The basic idea is to compare the feature descriptors of each feature point in one image with all the feature points in the other image, and then select the most similar feature point as the matching result.

[0065] Among them, RANSAC (Random Sample Consensus) is an iterative method for fitting models and removing outliers in data. In the RANSAC algorithm, a threshold needs to be set to determine which data points are considered to be inliers that conform to the model and which are considered outliers (outliers). The choice of threshold is usually determined based on the specific application scenario and data characteristics. In RANSAC, the threshold determines the degree of fit between the data point and the fitted model, and data points exceeding the threshold are considered outliers; generally speaking, the smaller the threshold, the better the robustness of the model, but it may also cause more inliers to be misjudged as outliers; on the contrary, the larger the threshold, the worse the robustness of the model, but more inliers can be retained.

[0066] Among them, the characteristic of rigid body transformation is that it does not change the shape and size of the object, but only translates, rotates and scales in space, thereby maintaining the rigidity of the object; rigid body transformation can be described as a Euclidean transformation, which keeps the distance and angle between points in space unchanged.

[0067] Among them, the homography matrix (HomographyMatrix), also known as the homography matrix or homography transformation matrix, is an important concept in the field of computer vision used to describe the projection transformation relationship between two images. In two-dimensional space, the homography matrix can describe the projection transformation relationship between one image and another image, and is usually used in tasks such as image registration, image correction, and image stitching; assuming that there are two images A and B, the homography matrix H between them defines the projection transformation relationship between a point in image A and the corresponding point in image B; specifically, for a two-dimensional point (x, y) in image A, its corresponding point (x', y') in image B can be expressed by the multiplication relationship between the homography matrix H and the homogeneous coordinates as follows:

[0068]

[0069] Among them, H is a 3x3 matrix, called the homography matrix, which contains a series of transformation parameters, such as rotation, translation, scaling and projection; by solving the homography matrix, operations such as registration, alignment and splicing between images can be achieved. In practical applications, a common method is to estimate the homography matrix by matching feature points, and then use the estimated homography matrix to perform image projection transformation. Various methods can be used to estimate the homography matrix, such as direct linear transformation (DLT), least squares (LeastSquares), random sampling consensus (RANSAC), etc.

[0070] Among them, bolt preload force refers to the force applied to the bolt during the bolt installation process, which is used to generate friction and pressure to form a tight connection between the bolt and the connecting part; preload force is a tightening force that can ensure the tightness and stability of the bolt connection to prevent loosening and failure.

[0071] Furthermore, the homography matrix satisfies the rigid body transformation matrix in the form of:

[0072]

[0073] in,[ ] represents the horizontal displacement, vertical displacement and overall translation of the projection transformation between the two images; [ ] represents the rotation and scaling effects of the projection transformation; [ ] is used to represent the perspective effect and ensure it is an affine transformation. , are the horizontal and vertical translations respectively In the present invention, These are the 9 elements of the obtained homography matrix.

[0074] Please refer again Fig. 9 In an optional embodiment of the present invention, the bolt loosening identification method based on image feature point matching mainly includes several steps such as feature point generation, feature point matching, matching pair screening and angle calculation.

[0075] In a specific embodiment of the present invention, the two bolts with the shortest Euclidean distance of the center coordinates of the bolt sub-images of the front and back pictures are calculated and paired into a group pair, and we obtain the matching of the same bolt in the front and back pictures. The feature descriptors (feature points) of each bolt in the group are generated by the improved SIFT algorithm of the present invention, and then all the generated feature descriptors are matched by the Brutal-Force matcher. The specific matching steps are: first, the current feature descriptor is compared with the feature descriptor of the second group, the square of the difference between each corresponding component of the two feature vectors is calculated, and then all the square differences are added and the square root is taken to calculate the vector distance. At the same time, a distance threshold is set. Only feature descriptors with a distance less than the threshold are considered to be suitable matches. In order to make the number of feature matching pairs the minimum number of feature points in the two bolts, the present invention dynamically adjusts the distance threshold. First, an initial distance threshold is set. If the number of feature point matching pairs is less than the minimum number of feature points in the two images, the distance threshold is increased by a certain step size s until all feature points are matched.

[0076] At this point, matching pairs between the first set of feature descriptors and the second set of feature descriptors are obtained. Since in this working condition, the front and rear bolt images after perspective correction only contain rigid body transformations, this matrix allows us to perform a matrix transformation of the obtained homography matrix. Use OpenCV to calculate the homography matrix, and use the RANSAC algorithm to screen the matching points. However, since in this process, the matching points that meet the digital image processing conditions may not necessarily be true matching points, in order to enhance the robustness of the algorithm, the threshold of the RANSAC algorithm is dynamically adjusted in the solution of the present invention, the number of inliers is increased, and the mismatched point pairs are deleted. The optimal subset of inliers is selected to calculate the homography matrix until the obtained homography matrix can satisfy the rigid body transformation matrix after matrix transformation. The form of the calculation meets the actual conditions under this working condition, thereby determining the correct rotation angle of the bolt. Determining the rotation angle based on the characteristic points removes the existing limitation that only 60-degree rotation angle can be calculated.

[0077] .

[0078] Further, in step S20, the method for obtaining the corrected bolt group image is specifically as follows: using the trained YOLOv8-P6 algorithm to perform bolt target detection on the initial bolt group image, and obtaining two sets of bolt detection data before and after, wherein the bolt detection data includes bolt category, center coordinates and size of the detection frame, the bolt category includes outer sleeve bolt, embedded bolt and missing bolt, the outer sleeve bolt corresponds to an inner round screw, the center coordinates of the detection frame include the center coordinates x of the detection frame and the center coordinates y of the detection frame, and the size of the detection frame includes the length l of the detection frame and the width w of the detection frame; traversing one set of bolts The center coordinates of the detection frame corresponding to each fastening bolt in the bolt detection data are obtained, the Euclidean distance between the center coordinates of any two detection frames is calculated, all Euclidean distances in the bolt detection data are traversed to obtain, and the four vertices of the bolt group are determined based on the two largest sets of Euclidean distances (diagonal distances); the four vertices are connected into a quadrilateral in sequence, and the number of rows and columns of the outermost fastening bolts in the initial bolt group image is determined according to the number of detection frames passed by the four sides of the quadrilateral; the row-column ratio of the bolt arrangement in the initial bolt group image is determined; and a perspective transformation is performed according to the row-column ratio to obtain a corrected bolt group image after perspective correction.

[0079] Among them, the YOLOv8 algorithm is a target detection algorithm based on deep learning. It is widely used in real-time object detection tasks. The name of the algorithm "YOLO" stands for "You Only Look Once", which emphasizes its fast detection speed and efficient calculation method. In the YOLOv8-P6 network model architecture adopted by the present invention, Backbone (backbone network / backbone network) is responsible for extracting features from the input image and converting the image into a feature representation with rich semantic information; Neck (connection / neck module) is an intermediate layer used to fuse features from Backbone to improve the performance of the model; Head (task head / row head module) is the last layer of the model, and its structure varies according to different tasks. In the target detection task of the present invention, we use a bounding box regressor and a classifier as Head. YOLOv8-P6 achieves fast and accurate target detection by dividing the entire image into smaller grid cells and predicting the target in each grid cell at the same time. The network training of YOLOv8-P6 is as follows. The collected target bolt photos are used as the data set, which includes three categories: embedded bolts (bolt_embedded), jacket bolts (bolt_jacketed) and missing bolts (missing). Typical examples are as follows Figure 4As shown in the figure; data enhancement techniques such as cropping, adding noise, rotating and changing color channels are used to process the target object so that the target object is still included in the enhanced image. Finally, a bolt detection dataset of 18779 images is generated, which is divided into training set, validation set and test set in the ratio of 7:1.5:1.5, of which the training set is 18779×70%≈13145 images, and the validation set and test set are 18779×15%≈2817 images; YOLOv8-P6 is trained by back propagation and momentum stochastic gradient descent (SGD), with a learning rate of 0.01, a mini-batch size of 8, and the maximum number of training epochs is set to 500.

[0080] Furthermore, the specific method for determining the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image is as follows: if it is an outer sleeve bolt, the Sift-Bolt algorithm formed by the SIFT algorithm combined with the Brutal-Force matcher is used to respectively calculate the rotation angles of the outer sleeve bolt and the inner circle screw in the target bolt matching pair, and the relative rotation angles of the two categories are calculated based on the rotation angles of the two categories as the rotation angle; if it is an embedded bolt, the Sift-Bolt algorithm formed by the SIFT algorithm combined with the Brutal-Force matcher is used to calculate the rotation angle of the embedded bolt as the rotation angle.

[0081] Among them, the development of YOLOv8-BoltSeg includes the understanding of the basic principles and the training of YOLOv8-BoltSeg. In the scheme of the present invention, the YOLOv8-Seg model is improved based on the Gather-and-Distribute mechanism (GD), and its network architecture is shown in the figure. First, the image features are extracted by Backbone, and then the GD mechanism is added to the Neck structure to improve the feature fusion mechanism. The GD mechanism is implemented through convolution and self-attention operations, which enhances the multi-scale feature fusion capability. It includes a low-level collection and distribution branch (Low-GD) and a high-level collection and distribution branch (High-GD), which extract and fuse feature information through convolution-based blocks and attention-based blocks respectively. The collection and distribution process corresponds to three modules: feature alignment module (FAM), information fusion module (IFM) and information injection module (Inject). The collection process involves two steps. First, FAM collects and adjusts features at each level, and secondly, IFM fuses aligned features to generate global information. The step of distributing branches is that after obtaining the fused global information from the collection process, the injection module distributes the information to each level and injects it using a simple attention operation, thereby enhancing the segmentation ability of the branch. Finally, the Head part uses a bounding box regressor, a classifier, and a cross entropy loss. The network training of YOLOv8-BoltSeg is as follows. 2,000 close-up bolt photos in the target detection dataset are used as the dataset. Labelme is used to annotate and generate a json file, and the json file is converted into a txt file required for training. The label types are divided into two categories: hexagon and circle. The outer hexagon label represents the head of the embedded bolt and the nut part of the outer sleeve bolt, and the inner circle label represents the screw part of the outer sleeve bolt. Typical examples are as follows. Figure 6 As shown. At the same time, data enhancement techniques such as cropping, adding noise and changing color channels were used to generate a bolt detection data set of 6,000 images. The training set, validation set and test set were also divided into training set, validation set and test set in the ratio of 7:1.5:1.5, so the training set is 6,000×70%=4,200 images, and the validation set and test set are 6,000×15%=900 images. The training of YOLOv8-BoltSeg is achieved through back propagation and momentum stochastic gradient descent (SGD), with a learning rate of 0.01, a mini-batch size of 8, and a maximum number of training epochs of 500.

[0082] Furthermore, the specific method of performing image cutting and super-resolution reconstruction on the corrected bolt group image to obtain multiple bolt sub-images corresponding to each frame is as follows: performing target detection on the corrected bolt group image to obtain a detection frame, cutting the bolt sub-image detected in the corrected bolt group image according to the coordinates of the detection frame to obtain a bolt sub-image, and using the SRGAN algorithm to perform super-resolution reconstruction on the cut bolt sub-image to improve the clarity and details of the image. In the specific implementation, cutting is performed based on the results obtained by YOLOv8-P6 target detection, and segmentation is performed after cutting. YOLOv8-BoltSeg is used to perform semantic segmentation on the cut image, and the bolt area is cut out according to the results of the semantic segmentation, and the redundant background is removed. In the present invention, semantic segmentation is performed after the bolt category is determined.

[0083] In a specific embodiment, the SRGAN algorithm (Super-Resolution Generative Adversarial Network, i.e., an image super-resolution reconstruction algorithm based on a generative adversarial network) is used to perform super-resolution reconstruction on the segmented sub-images to improve the clarity and details of the image. This process helps to restore the detailed information of the bolts more accurately, which is convenient for subsequent feature point generation and matching. At the same time, the YOLOv8-BoltSeg algorithm is used for semantic segmentation, and a coordinate text document obtained by segmentation is generated at the same time. The pixels of the segmented area are cut out through the segmentation results, so that the subsequent generated feature points fall accurately in the target area, further improving the accuracy and reliability of the detection. At this time, the category of this bolt is read according to the text document corresponding to the bolt sub-image. The bolt sub-image coordinates stored in the text documents of all groups in the front and back photos are traversed and read, and the two bolts with the shortest Euclidean distance are obtained as a pair, and they are considered to be the same bolt in the front and back photos. It should be noted that at this time, if there is a missing category in the bolt sub-image pair, the subordinate steps are skipped and the missing (missing) is used as the output value. The bolt pair and the corresponding bolt category are used as input. When it is an outer sleeve bolt, it will be divided into two categories: outer hexagon and inner circle. The Sift-Bolt proposed in the present invention is used to calculate the angles of the outer hexagon and inner circle respectively, and then the relative angles of the two categories are calculated as the loosening angle of the outer sleeve bolt. If it is an embedded bolt, it will be divided into an outer hexagon category. At this time, Sift-Bolt is used to calculate the outer hexagon angle as the loosening angle of the embedded bolt. Finally, the angle numerical results are spliced ​​to the coordinate document corresponding to the bolt sub-graph and stored and output.

[0084] Furthermore, in step S40, the specific steps of generating feature points and feature descriptions by the SIFT algorithm through the method of scale space extreme value detection, key point positioning, direction allocation and feature point description are as follows:

[0085] S421, convolve the initial bolt group image with Gaussian kernels of different standard deviations to generate a series of blurred images with different blur levels, and obtain the scale space. Each blurred image represents the blur level at different scales. The Gaussian convolution kernel is shown in the following formula:

[0086]

[0087] in , Represent the horizontal and vertical coordinates of the pixel points respectively. , Represent the length and width of the initial bolt group image, The parameters are the variance (Gaussian radius), The value ranges from 1.2 to 1.6;

[0088] S422, performing a subtraction operation on two adjacent images with higher blur levels in the scale space to generate a Gaussian difference image;

[0089] S423, marking candidate key points based on the Gaussian difference image;

[0090] S424, threshold screening is performed on the detected candidate key points, and the candidate key points and edge response points with low contrast are removed to obtain a key point as shown in the following formula:

[0091]

[0092] Among them, T is the threshold, n is the number of images to be extracted features, is the pixel value;

[0093] Process all candidate key points to obtain a key point set;

[0094] S425, accurately locating the position and scale of the key points by fitting a second-order Gaussian function on the set of key points initially detected, thereby determining the precise position of the key points;

[0095] S426, determining the main direction of the key point to ensure rotation invariance of the key point;

[0096] S427, performing mathematical feature description on the key points to obtain feature description of the feature points, wherein the following formula is used:

[0097]

[0098]

[0099] Processing is performed to obtain feature descriptions (descriptors) of feature points, where: is the L2 norm of the feature descriptor, is the feature descriptor vector, is a component of the feature descriptor vector, n is the vector dimension, It is the normalized feature descriptor vector, and the length of the descriptor is 1.

[0100] In the present invention, n is a pair of images to be matched, and its value is 2.

[0101] Furthermore, T is set to 0.5, or T is the local threshold obtained by adjusting the adaptive Gaussian threshold method. According to the local pixel distribution of each small block, a local Gaussian weight is calculated, and the threshold is adjusted by the adaptive Gaussian threshold method, as shown in the formula,

[0102]

[0103] in Representing pixels At the local threshold, is the average gray value of pixels in adjacent local areas, is the standard deviation of pixels in adjacent local regions.

[0104] In a specific embodiment of the present invention, SIFT generates feature description of feature points through four steps: scale space extremum detection, key point location, direction allocation and feature point description.

[0105] The first step is the scale space extreme value detection step. In order to make the feature points invariant to scale changes, a series of Gaussian kernels with different standard deviations are applied to the original image for convolution to generate a series of images with different blur levels, called scale space. Each image represents the blur level of the original image at different scales. The Gaussian convolution kernel is shown in formula (1).

[0106]

[0107] in , Represent the horizontal and vertical coordinates of the pixel respectively. , Represent the length and width of the image respectively. The parameter is the variance, also known as the Gaussian radius. The value is 1.6, but considering that the camera has already blurred the image in practice, in the present invention, The value is 1.2. At this time, a Gaussian difference operation is performed on images of adjacent scales. That is, a subtraction operation is performed on two adjacent images with higher blur levels in the scale space to produce a Gaussian difference image. The purpose of this step is to find local extreme points in the scale space, which have stable responses at different scales and are invariant to the scale changes of the image. In the Gaussian difference image, each pixel is compared with the 26 corresponding pixels of its 8 adjacent pixels to determine whether it is a local extreme point. If a pixel value (i.e., grayscale value) is larger (or smaller) than the values ​​of its adjacent pixels, it is marked as a candidate key point. At this time, the detected candidate key points are thresholded to remove low-contrast key points and edge response points, so that the final key point set of this step is obtained as shown in formula (2),

[0108]

[0109] In SIFT, T is usually set to 0.4, and n is the number of images to be extracted. is the pixel value. It should be noted that in order to make the selected key points more accurate and better process images under various lighting conditions, the threshold selection is optimized in the present invention. An adaptive Gaussian threshold method is used to calculate a local Gaussian weight according to the local pixel distribution of each small block. The calculation for adjusting the threshold is shown in formula (3).

[0110]

[0111] in Representing pixels At the local threshold, is the average gray value of pixels in adjacent local areas, is the standard deviation of pixels in adjacent local regions.

[0112] The second step is to locate the key points. On the set of key points initially detected, the position and scale of the key points are accurately located by fitting a second-order Gaussian function. The process first uses the Sobel operator to calculate the amplitude and direction of the pixel gradient around the key point, and then constructs a second-order Gaussian function near the extreme point position as shown in formula (4) to fit the surface shape of the local image.

[0113]

[0114] in, is the gray value at the extreme point, is the gradient vector, is the Hessian matrix. Finally, quadratic interpolation is performed by expanding the Gaussian function into a Taylor series and taking the derivative, that is, solving the extreme points of the Gaussian function to determine the exact position of the key point.

[0115] In the third step, in order to ensure the rotation invariance of the key point, the histogram of the gradient direction obtained in the second step is counted in the neighborhood around the key point. Each bin of the histogram represents an interval of gradient direction, and its value represents the accumulation of gradient amplitude in that direction. The number of bins in the histogram is divided into 36 direction intervals, one interval every 10 degrees. Then traverse the direction histogram to find the direction with the largest gradient amplitude as the main direction of the key point. The main direction selected in this way is usually the most obvious change direction of the key point at this position. At the same time, in order to enhance the robustness of the key point, multiple main directions are usually selected within a 360-degree range around the key point. In addition to the main direction, the direction of the second largest gradient amplitude is selected as the secondary direction. Doing so can make the key point more robust to rotation changes and increase the diversity of feature points.

[0116] In the fourth step, after finding the key point and determining its main direction, the key point will be described mathematically, which is called a feature descriptor, to facilitate subsequent feature matching. First, in the 16x16 pixel area around the key point, this area is divided into 4×4 pixel sub-areas, and then the gradient amplitude and direction of each pixel in the pixel sub-area are calculated. Next, the gradient direction of the pixels in each sub-area is assigned to the corresponding gradient direction histogram. This histogram has 8 directions, and the angle between every two directions is 45°. Subsequently, these gradient direction histograms are connected in series to form a long feature vector, namely the key point descriptor (feature descriptor). For each small block, the values ​​of its 8 gradient directions are connected in sequence to form a sub-vector of length 8. All sub-vectors are connected to obtain a feature vector of length 4×4×8=128. Finally, in order to increase the robustness of the descriptor, the present invention performs L2 norm normalization on the entire descriptor vector. First, for each element in the descriptor vector, its square is calculated and summed as shown in formula (5),

[0117]

[0118] Get the L2 norm of the descriptor, where Descriptor vector, is a component of the descriptor vector, n is the vector dimension, and its value is 128. Then each element in the descriptor vector is divided by its L2 norm as shown in equation (6),

[0119]

[0120] The normalized descriptor vector is obtained. This step makes the length of the descriptor become a fixed value of 1, thereby further enhancing the rotation invariance and scale invariance of the descriptor.

[0121] Furthermore, in step S40, the specific method for matching the first feature descriptor and the second feature descriptor is as follows: compare the first feature descriptor and the second feature descriptor one by one, and calculate the square of the difference between each corresponding component of the feature vectors of the two feature points one by one; add the squares of the differences of all corresponding components, and take the square root thereof to calculate the vector distance; determine that the feature descriptor (feature point) whose vector distance is less than a preset distance threshold is a suitable matching point; wherein, a dynamic adjustment method is used to determine the preset distance threshold, and an initial distance threshold is set. If the number of feature point matching pairs is less than the minimum number of feature points of the two images, the preset distance threshold is increased by a preset step size s until all feature points are matched one by one, so that the number of matching points meets the number requirement of the matcher; after the number of matching points meets the number requirement of the matcher, the RANSAC algorithm is used to screen the matching points, and then the wrong matching pairs are eliminated, wherein the filtering threshold of the RANSAC algorithm is dynamically adjusted for screening, and the filtering threshold is increased step by step until the obtained homography matrix satisfies the rigid body transformation matrix after matrix transformation. in the form of.

[0122] In the present invention, the number of erroneous matching pairs eliminated is the largest in the initial state, and the remaining matching pairs are the smallest. Then, the filtering threshold becomes smaller and smaller, fewer and fewer are eliminated, and more and more remaining matching pairs are found until the steel body transformation matrix form is satisfied.

[0123] Further, the preload change value of the fastening bolt is determined based on the rotation angle of the fastening bolt and the rotation angle-preload change relationship data.

[0124] In a specific embodiment of the present invention, after the rotation angle is obtained, the operator will receive a prompt whether to perform preload estimation. When performing preload estimation, the operator will select the rotation angle and preload relationship formula proposed by Shoberg

[32] that was input in advance, as shown in formula (7). When the bolt rotation angle is When the preload force changes for:

[0125]

[0126] in is the bolt stiffness, is the stiffness of the connection, and P is the pitch. Or you can enter the formula yourself, and enter the bolt model, connection stiffness and initial preload value of the bolt. At this time, you can calculate the current preload change value based on the obtained bolt rotation angle and rotation angle-preload change relationship formula. It should be noted that when When the initial preload value is greater than the input value, the output loss preload That is the preload loss value, and the output value is added to the text document corresponding to the bolt sub-diagram and stored. Then the document information of the bolt sub-diagram at this time should be as shown in Table 1.

[0127] Table 1

[0128]

[0129] The beneficial effects of the present invention are as follows:

[0130] 1. It can reduce the labor cost in detection and realize fast, efficient and accurate bolt loosening detection. It is a non-contact detection and does not require the deployment of sensors.

[0131] 2. Automatically correct the viewing angle of bolt group photos without auxiliary means;

[0132] 3. Added image super-resolution reconstruction, which can also provide accurate detection results for blurred photos;

[0133] 4. Developed the YOLOv8-BoltSeg network for fast and accurate bolt semantic segmentation, so that the matching of feature points is within the target area and not on the background;

[0134] 5. SIFT-Bolt is proposed to calculate the bolt rotation angle, which realizes the detection of the bolt in 360° full cycle, breaking through the bottleneck of [0°, 60°] detection range;

[0135] 6. The change value of bolt preload can be calculated based on computer vision, providing a more critical reference value for bolt loosening

[0136] Furthermore, a loose bolt identification system based on image feature point matching includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned loose bolt identification methods based on image feature point matching is implemented.

[0137] Please refer to Fig.10 The present invention provides a bolt loosening identification unit based on image feature point matching, including a data input module for executing step S10; a target detection and super-resolution reconstruction module for executing step S20; a target detection and super-resolution reconstruction module for executing step S20; a semantic segmentation module for executing step S30; a rotation angle and prestress calculation module for executing step S40; and a result and output display module for outputting and displaying the calculation results.

[0138] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying loose bolts based on image feature point matching, characterized in that: The steps include: S10, acquiring two frames of initial bolt group images at different times, wherein the initial bolt group images include a plurality of fastening bolts; S20, determining the row-column ratio of the arrangement of the fastening bolts in the initial bolt group image, and performing perspective transformation according to the row-column ratio to obtain a corrected bolt group image after perspective correction; Performing image segmentation and super-resolution reconstruction processing on each frame of the calibrated bolt group image to obtain a plurality of bolt sub-images corresponding to each frame; S30, performing semantic segmentation on the bolt sub-graph, eliminating non-target areas, and determining target areas to segment the fastening bolts; S40, matching the segmented fastening bolts at different times to determine a target bolt matching pair, where the target bolt matching pair is a match of the same fastening bolt in the previous image and the next image; The SIFT algorithm is used to generate a first feature descriptor and a second feature descriptor of each fastening bolt in the matching pair, wherein the first feature descriptor is a feature description of a feature point of the target area in the previous image, and the second feature descriptor is a feature description of a feature point of the target area in the subsequent image. The SIFT algorithm generates a feature description of a feature point by using a method of scale space extreme value detection, key point positioning, direction allocation and feature point description; Matching the first feature descriptor and the second feature descriptor by a Brutal-Force matcher, calculating a homography matrix, and determining a rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image by transforming the homography matrix; In step S20, the calibration bolt group image is obtained in the following manner: Use the trained YOLOv8-P6 algorithm to perform bolt target detection on the initial bolt group image, and obtain two groups of bolt detection data, wherein the bolt detection data includes bolt categories, center coordinates and sizes of detection frames. The bolt categories include outer sleeve bolts, embedded bolts and missing bolts. The outer sleeve bolts correspond to inner round screws. The center coordinates of the detection frame include the center coordinates x and y of the detection frame. The size of the detection frame includes the length l and the width w of the detection frame. Traversing the center coordinates of the detection frame corresponding to each fastening bolt in one set of the bolt detection data, calculating the Euclidean distance between the center coordinates of any two of the detection frames, traversing and obtaining all the Euclidean distances in the bolt detection data, and determining four vertices of the bolt group based on the two largest sets of Euclidean distances; Connect the four vertices in sequence to form a quadrilateral, and determine the number of rows and columns of the outermost fastening bolts in the initial bolt group image according to the number of detection frames passed by the four sides of the quadrilateral; Determining the row and column ratio of the bolt arrangement in the initial bolt group image; Performing perspective transformation according to the row-column ratio to obtain a correction bolt group image after viewing angle correction; In step S40, the specific steps of the SIFT algorithm generating a feature description of a feature point by means of scale space extreme value detection, key point location, direction allocation and feature point description are as follows: S421, convolve the initial bolt group image with Gaussian kernels of different standard deviations to generate a series of blurred images with different blur levels, and obtain the scale space. Each blurred image represents the blur level at different scales. The Gaussian convolution kernel is shown in the following formula: in , Represent the horizontal and vertical coordinates of the pixel points respectively. , Represent the length and width of the initial bolt group image, The parameter is the variance, The value ranges from 1.2 to 1.6; S422, performing a subtraction operation on two adjacent images with high blur levels in the scale space to generate a Gaussian difference image; S423, marking candidate key points based on the Gaussian difference image; S424, performing threshold T screening on the detected candidate key points, removing candidate key points and edge response points with low contrast, and obtaining a preliminary detected key point set; S425, accurately locating the position and scale of the key points by fitting a second-order Gaussian function on the set of key points initially detected, thereby determining the precise position of the key points; S426, determining the main direction of the key point to ensure rotation invariance of the key point; S427, performing mathematical feature description on the key points to obtain feature description of the feature points; The specific manner of matching the first feature descriptor and the second feature descriptor in step S40 is: Compare the first feature descriptor and the second feature descriptor one by one, and calculate the square of the difference between each corresponding component of the feature vectors of the two feature points one by one; Add the squares of the differences between all corresponding components and take the square root to calculate the vector distance; Determine that the feature point whose vector distance is less than a preset distance threshold is a suitable matching point; Wherein, a dynamic adjustment method is used to determine a preset distance threshold, an initial distance threshold is set, and if the number of feature point matching pairs is less than the minimum number of feature points of the two images, the preset distance threshold is increased by a preset step length s so that the number of matching points meets the number requirement of the matcher; After the number of matching points meets the number requirement of the matcher, the RANSAC algorithm is used to screen the matching points and then eliminate the wrong matching pairs. The filtering threshold of the RANSAC algorithm is dynamically adjusted for screening, and the filtering threshold is gradually reduced until the obtained homography matrix satisfies the rigid body transformation matrix after matrix transformation. in the form of , are the horizontal and vertical translations, respectively. Expressed as a rotation angle.

2. The method for identifying loose bolts based on image feature point matching according to claim 1, characterized in that: In step S40, the homography matrix is ​​transformed into a rigid body transformation matrix in the form of: in,[ ] represents the horizontal displacement, vertical displacement and overall translation of the projection transformation between the two images; [ ] represents the rotation and scaling effects of the projection transformation; [ ] is used to represent the perspective effect and ensure it is an affine transformation. , are the horizontal and vertical translations, respectively. Expressed as a rotation angle.

3. The method for identifying loose bolts based on image feature point matching according to claim 1, characterized in that: The specific method of determining the rotation angle of the corresponding fastening bolt in the rear image relative to the fastening bolt in the front image is: If it is an outer sleeve bolt, the rotation angles of the outer sleeve bolt and the inner screw in the target bolt matching alignment are calculated respectively, and the relative rotation angle calculated based on the rotation angles of the outer sleeve bolt and the inner screw is used as the rotation angle; If it is an embedded bolt, the rotation angle of the embedded bolt will be calculated as the rotation angle.

4. The method for identifying loose bolts based on image feature point matching according to any one of claims 1 or 2, characterized in that: The specific method of performing image segmentation and super-resolution reconstruction on each frame of the corrected bolt group image to obtain multiple bolt sub-images corresponding to each frame is as follows: Perform target detection on the calibration bolt group image to obtain a detection frame, According to the coordinates of the detection frame, the bolt sub-image detected in the correction bolt group image is cut to obtain the bolt sub-image, The SRGAN algorithm is used to perform super-resolution reconstruction on the cut bolt sub-image.

5. The method for identifying loose bolts based on image feature point matching according to claim 1, characterized in that: T is 0.5, or T is the local threshold obtained by adjusting the adaptive Gaussian threshold method , according to the local pixel distribution of each small block, a local Gaussian weight is calculated, and the threshold is adjusted using the adaptive Gaussian threshold method, as shown in the formula, in Representing pixels At the local threshold, is the average gray value of pixels in adjacent local areas, is the standard deviation of pixels in adjacent local regions.

6. The method for identifying loose bolts based on image feature point matching according to any one of claims 1 or 2, characterized in that: The preload change value of the fastening bolt is determined based on the rotation angle of the fastening bolt and the rotation angle-preload change relationship data.

7. A bolt loosening identification system based on image feature point matching, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

Images