Bolt disease detection method and device based on phase motion estimation

Abstract

The invention provides a bolt disease detection method and device based on phase motion estimation, and belongs to the technical field of bridge engineering, and the method comprises the steps: obtaining vibration videos of different types of bolt diseases; based on an image processing technology, performing phase motion estimation on a target area in each frame of image in the vibration video to obtain a displacement signal; converting the displacement signal into a time-frequency image based on a time-frequency conversion algorithm; training a machine learning model by using the time-frequency image and the corresponding disease classification label, and taking the trained machine learning model as a bolt disease classification model; and detecting bolt disease types by using the bolt disease classification model. According to the bolt disease detection method and device based on phase motion estimation provided by the invention, the speed and accuracy of bolt disease detection are improved, the cost and risk of manual inspection are reduced, and meanwhile, a scientific maintenance decision basis is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge engineering, and in particular to a method and device for detecting bolt defects based on phase motion estimation. Background Art

[0002] Steel box girders are a key part of bridge construction. The connections between different parts of steel structure bridges mainly rely on three methods: rivets, welds and bolts. Among them, bolt connections are widely used because of their simple construction, convenient disassembly and assembly, and small amount of steel. However, in actual use, bolt connections are often affected by unfavorable factors such as vibration of the fastened structure, impact of wind and corrosion of rainwater, and are prone to loosening, deformation and even falling off, which will have adverse consequences on the overall structure of the bridge and pose a hidden danger to the safe operation of the bridge and the safety of people's lives and property. Therefore, regular inspections of bolts at bridge joints have become an indispensable task in bridge maintenance.

[0003] The internal structure of steel box girders is complex. Currently, all steel box girder inspection tasks are mainly carried out manually. Operators observe and judge each section with the naked eye, which makes it difficult to quickly detect and identify common problems inside the steel box girders. Therefore, there are disadvantages such as low detection efficiency, incomplete coverage, high cost and inaccuracy. Summary of the invention

[0004] The present invention provides a bolt defect detection method and device based on phase motion estimation, aiming to solve the defects of low efficiency, incomplete coverage, high cost and insufficient accuracy of relying on manual inspection in the prior art, and to realize efficient, accurate, automated and real-time defect detection and classification of bolt connection parts in structures such as bridges.

[0005] In a first aspect, the present invention provides a method for detecting bolt defects based on phase motion estimation, comprising: obtaining vibration videos of different types of bolt defects; based on image processing technology, performing phase motion estimation on a target area in each frame of the vibration video to obtain a displacement signal; based on a time-frequency transformation algorithm, transforming the displacement signal into a time-frequency image; using the time-frequency image and corresponding defect classification labels to train a machine learning model, and using the trained machine learning model as a bolt defect classification model; and using the bolt defect classification model to detect bolt defect types.

[0006] According to the bolt defect detection method based on phase motion estimation provided by the present invention, based on image processing technology, phase motion estimation is performed on the target area in each frame image in the vibration video to obtain a displacement signal, including: obtaining the spatial position information of each pixel in the target area of ​​each frame image in the vibration video at a corresponding moment; based on the spatial position information of each pixel point in the target area of ​​the images of consecutive frames in the vibration video at each moment, constructing a displacement signal.

[0007] According to the bolt defect detection method based on phase motion estimation provided by the present invention, the machine learning model is a convolutional neural network.

[0008] According to the bolt disease detection method based on phase motion estimation provided by the present invention, the convolutional neural network includes an input layer, multi-layer convolutional layers, multi-layer pooling layers corresponding to the convolutional layers, a fully connected layer and an output layer; the input layer is used to receive the time-frequency image; the convolutional layer is used to perform a convolution operation on the time-frequency image to output a feature map of the time-frequency image; each pooling layer is located after each convolutional layer, and is used to perform dimensionality reduction processing on the feature map output by the convolutional layer by using the maximum pooling method; the fully connected layer is used to map the features after dimensionality reduction processing into a feature vector; the output layer uses a classifier to process the feature vector to obtain a classification result; the training of the convolutional neural network adopts a batch sample input method.

[0009] According to the bolt defect detection method based on phase motion estimation provided by the present invention, vibration videos of different types of bolt defects are obtained, including: using a video acquisition device to respectively obtain vibration videos of bolt displacement, falling off and loosening.

[0010] According to the bolt defect detection method based on phase motion estimation provided by the present invention, the time-frequency image is an RGB image.

[0011] In a second aspect, the present invention further provides a bolt defect detection device based on phase motion estimation, comprising:

[0012] The first processing module is used to obtain vibration videos of different types of bolt diseases;

[0013] The second processing module is used to estimate the phase motion of the target area in each frame of the vibration video based on the image processing technology to obtain the displacement signal;

[0014] A third processing module is used to transform the displacement signal into a time-frequency image based on a time-frequency transformation algorithm;

[0015] A fourth processing module is used to train a machine learning model using the time-frequency image and the corresponding disease classification labels, and use the trained machine learning model as a bolt disease classification model;

[0016] The fifth processing module is used to detect the bolt disease type by using the bolt disease classification model.

[0017] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned methods for detecting bolt defects based on phase motion estimation are implemented.

[0018] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described bolt defect detection methods based on phase motion estimation.

[0019] The bolt defect detection method and device based on phase motion estimation provided by the present invention have the following beneficial effects compared with the prior art:

[0020] (1) The present invention accurately captures subtle vibration changes through phase motion estimation technology, which can detect potential defects of bolts, such as looseness, cracks or corrosion, earlier, which is more accurate than traditional manual visual inspection.

[0021] (2) The present invention automatically processes video data and performs real-time analysis, greatly shortening the detection cycle, reducing the time and resources required for manual inspections, and can quickly cover large areas. It reduces the reliance on professional technicians and the need to frequently dispatch personnel for on-site inspections, thereby saving human resources and related costs.

[0022] (3) The present invention can complete the inspection work without directly contacting the bolts, and is particularly suitable for bolt inspection in difficult-to-access or dangerous locations, thereby improving the safety of the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flow chart of a bolt defect detection method based on phase motion estimation provided by the present invention;

[0025] Figure 2 It is a schematic diagram of classification using a convolutional neural network provided by the present invention;

[0026] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0029] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0030] Combine the following Figure 1-Figure 3 The present invention describes a method and device for detecting bolt defects based on phase motion estimation provided by an embodiment of the present invention.

[0031] Figure 1 is a flow chart of a bolt defect detection method based on phase motion estimation provided by the present invention, such as Figure 1 As shown, including but not limited to the following steps:

[0032] The bolt defect detection method based on phase motion estimation provided by the present invention specifically comprises the following steps:

[0033] Step 101: Obtain vibration videos of different types of bolt damage.

[0034] The purpose of this step is to collect the vibration characteristics of the bolts under different disease states. Optionally, use video acquisition equipment to obtain vibration videos of bolts under different diseases such as displacement, falling off, and loosening. Make sure to cover a variety of possible disease types so that subsequent analysis can identify various conditions.

[0035] Step 102: Based on image processing technology, phase motion estimation is performed on the target area in each frame of the vibration video to obtain a displacement signal.

[0036] Extract signals that reflect the vibration characteristics of the bolt from the video data. The present invention uses advanced image processing algorithms, such as optical flow and feature point tracking, to calculate the position offset of each pixel over time, and then obtain phase motion information. This process can capture subtle vibration patterns and provide high-precision data support for subsequent analysis.

[0037] Based on the content of the above embodiment, as an optional embodiment, the bolt defect detection method based on phase motion estimation provided by the present invention performs phase motion estimation on the target area in each frame image in the vibration video based on image processing technology to obtain a displacement signal, including the following steps:

[0038] Firstly, the vibration video is decomposed into images frame by frame, and the Gabor filter is used to perform convolution operation on the decomposed images frame by frame to determine the target area;

[0039] Then, the spatial position information (which can be called the spatiotemporal information of the pixel point) of each pixel in the target area (the area where the bolt is located) of each frame image at the corresponding time is obtained;

[0040] Finally, a displacement signal is constructed based on the spatial position information of each pixel in the target area of ​​the image of the continuous frames in the vibration video at each moment.

[0041] That is, based on the spatiotemporal information of each pixel in the target area of ​​the image of the continuous frames in the vibration video, a displacement signal (ie, the collection of spatiotemporal information at all moments in the vibration video) is constructed.

[0042] The phase motion estimation-based method provided by the present invention has high spatial resolution for displacement signal acquisition, that is, each pixel is used as a sensor to obtain a large amount of data at the same time; the dynamic characteristics do not change, that is, it is a non-contact measurement, and the displacement signal of the acquisition area can be obtained more accurately.

[0043] Step 103: Based on a time-frequency transformation algorithm, transform the displacement signal into a time-frequency image.

[0044] Optionally, the specific implementation of step 103 is as follows:

[0045] (1) For each individually processed pixel in the target area, its time-frequency image is obtained through time-frequency transformation.

[0046] The time-frequency transform algorithm may be a continuous wavelet transform algorithm. The present invention may select Daubechies wavelet with good approximate symmetry, tight support and smoothness as the wavelet basis function of the continuous wavelet transform.

[0047] (2) Based on the time-frequency image of each pixel in the target area, a time-frequency image used in the final step 104 is formed; the optional method is as follows:

[0048] 1) Stitching: The time-frequency images of each pixel in the target area are stitched together according to the original spatial layout to form a complete time-frequency image covering the entire target area. Since the complete time-frequency image is too large, a maximum pooling process can be performed to generate a final time-frequency image of smaller size.

[0049] 2) Aggregation: Another way is to construct a time-frequency image that reflects the overall vibration characteristics by statistically summarizing the time-frequency characteristics of each pixel (such as finding the average, maximum value, total energy, etc.).

[0050] 3) Superposition: The displacement signals of the pixels in the target area are superimposed to form an average displacement signal, and the averaged displacement signal is directly converted into a time-frequency image as the final result.

[0051] Step 104: Use the time-frequency images and the corresponding disease classification labels to train a machine learning model, and use the trained machine learning model as a bolt disease classification model.

[0052] Select a suitable machine learning or deep learning framework (such as convolutional neural network CNN), use the time-frequency images with disease classification labels as input, and train the model to learn different disease characteristics. After sufficient training, the model can accurately distinguish between normal and abnormal states, and further subdivide them into specific disease types.

[0053] Step 105: Detect the bolt disease type using the bolt disease classification model.

[0054] The present invention can deploy the trained bolt disease classification model in the actual monitoring environment, continuously receive new vibration video data, quickly process and output the diagnosis results. This enables maintenance personnel to detect problems at an early stage and take timely measures to ensure the safety and reliability of the structure.

[0055] The bolt defect detection method based on phase motion estimation provided by the present invention not only improves the speed and accuracy of bolt defect detection, but also reduces the cost and risk of manual inspection, and provides a scientific basis for maintenance decision-making.

[0056] Figure 2 is a schematic diagram of classification using a convolutional neural network provided by the present invention, such as Figure 2 As shown, the convolutional neural network includes an input layer, multi-layer convolutional layers, multi-layer pooling layers corresponding to the convolutional layers, a fully connected layer and an output layer; the input layer is used to receive the time-frequency image; the convolutional layer is used to perform a convolution operation on the time-frequency image to output a feature map of the time-frequency image; each pooling layer is located after each convolutional layer, and is used to reduce the dimension of the feature map output by the convolutional layer by using the maximum pooling method; the fully connected layer is used to map the features after the dimensionality reduction process to a feature vector; the output layer uses a classifier to process the feature vector to obtain a classification result; the training of the convolutional neural network adopts a batch sample input method.

[0057] Specifically, the time-frequency image of the displacement signal is learned and classified through a convolutional neural network, and the RGB three-channel time-frequency image is input into the input layer. The convolution layer convolves the feature map of the upper layer with a specific convolution kernel and adds a bias value, and then obtains the output feature map of this layer through the activation function relu function. The pooling layer controls the amount of calculation by downsampling, and the pooling method is maximum pooling. Finally, the data is flattened and passed through a fully connected layer, and the softmax classifier is used to complete the decision classification of the samples. The training of the convolutional neural network adopts the method of batch sample input, which mainly includes two processes: forward propagation of data and back propagation of errors. The Adam optimizer is used, and the learning rate is set to 0.001. Finally, the bolt connection status of the structure is classified.

[0058] On the other hand, the present invention also provides a bolt defect detection device based on phase motion estimation, the device comprising:

[0059] The first processing module is used to obtain vibration videos of different types of bolt diseases;

[0060] The second processing module is used to estimate the phase motion of the target area in each frame of the vibration video based on the image processing technology to obtain the displacement signal;

[0061] A third processing module is used to transform the displacement signal into a time-frequency image based on a time-frequency transformation algorithm;

[0062] A fourth processing module is used to train a machine learning model using the time-frequency image and the corresponding disease classification labels, and use the trained machine learning model as a bolt disease classification model;

[0063] The fifth processing module is used to detect the bolt disease type by using the bolt disease classification model.

[0064] It should be noted that the bolt defect detection device based on phase motion estimation provided in the embodiment of the present invention can execute the bolt defect detection method based on phase motion estimation described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0065] In summary, the bolt defect detection method and device based on phase motion estimation provided by the present invention have the following beneficial effects compared with the prior art:

[0066] (1) The present invention accurately captures subtle vibration changes through phase motion estimation technology, which can detect potential defects of bolts, such as looseness, cracks or corrosion, earlier, which is more accurate than traditional manual visual inspection.

[0067] (2) The present invention automatically processes video data and performs real-time analysis, greatly shortening the detection cycle, reducing the time and resources required for manual inspections, and can quickly cover large areas. It reduces the reliance on professional technicians and the need to frequently dispatch personnel for on-site inspections, thereby saving human resources and related costs.

[0068] (3) The present invention can complete the inspection work without directly contacting the bolts, and is particularly suitable for bolt inspection in difficult-to-access or dangerous locations, thereby improving the safety of the operation.

[0069] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute a bolt disease detection method based on phase motion estimation, the method comprising: obtaining vibration videos of different types of bolt diseases; based on image processing technology, performing phase motion estimation on the target area in each frame image in the vibration video to obtain a displacement signal; based on a time-frequency conversion algorithm, converting the displacement signal into a time-frequency image; using the time-frequency image and the corresponding disease classification label to train a machine learning model, and using the trained machine learning model as a bolt disease classification model; and using the bolt disease classification model to detect the type of bolt disease.

[0070] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0071] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the bolt disease detection method based on phase motion estimation provided by the above-mentioned embodiments, and the method includes: obtaining vibration videos of different types of bolt diseases; based on image processing technology, performing phase motion estimation on the target area in each frame image in the vibration video to obtain a displacement signal; based on a time-frequency transformation algorithm, transforming the displacement signal into a time-frequency image; using the time-frequency image and the corresponding disease classification label to train a machine learning model, and using the trained machine learning model as a bolt disease classification model; and using the bolt disease classification model to detect the type of bolt disease.

[0072] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the bolt disease detection method based on phase motion estimation provided in the above-mentioned embodiments, the method comprising: obtaining vibration videos of different types of bolt diseases; based on image processing technology, performing phase motion estimation on the target area in each frame image in the vibration video to obtain a displacement signal; based on a time-frequency transformation algorithm, transforming the displacement signal into a time-frequency image; using the time-frequency image and the corresponding disease classification labels to train a machine learning model, and using the trained machine learning model as a bolt disease classification model; and using the bolt disease classification model to detect the type of bolt disease.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A bolt defect detection method based on phase motion estimation, characterized in that: include: Get vibration videos of different types of bolt diseases; Based on image processing technology, the phase motion of the target area in each frame of the vibration video is estimated to obtain the displacement signal; Based on the time-frequency transformation algorithm, the displacement signal is transformed into a time-frequency image; The time-frequency images and corresponding disease classification labels are used to train the machine learning model, and the trained machine learning model is used as the bolt disease classification model; The bolt disease classification model is used to detect the bolt disease type.

2. The bolt defect detection method based on phase motion estimation according to claim 1 is characterized in that: Based on image processing technology, the phase motion of the target area in each frame of the vibration video is estimated to obtain the displacement signal, including: Obtaining spatial position information of each pixel in the target area of ​​each frame image in the vibration video at a corresponding time; A displacement signal is constructed based on the spatial position information of each pixel in the target area of ​​the image of the continuous frames in the vibration video at each moment.

3. The bolt defect detection method based on phase motion estimation according to claim 1 is characterized in that: The machine learning model is a convolutional neural network.

4. The bolt defect detection method based on phase motion estimation according to claim 3 is characterized in that: The convolutional neural network includes an input layer, a multi-layer convolutional layer, a multi-layer pooling layer corresponding to the convolutional layer, a fully connected layer and an output layer; The input layer is used to receive the time-frequency image; The convolution layer is used to perform a convolution operation on the time-frequency image to output a feature map of the time-frequency image; Each pooling layer is located after each convolutional layer and is used to reduce the dimension of the feature map output by the convolutional layer by using the maximum pooling method; The fully connected layer is used to map the features after dimensionality reduction processing into feature vectors; The output layer processes the feature vector using a classifier to obtain a classification result; The convolutional neural network is trained by batch sample input.

5. The bolt defect detection method based on phase motion estimation according to claim 1 is characterized in that: Get vibration videos of different types of bolt failures, including: The video acquisition equipment is used to obtain vibration videos when the bolts are displaced, fallen off, and loosened.

6. The bolt defect detection method based on phase motion estimation according to claim 1 is characterized in that: The time-frequency image is an RGB image.

7. A bolt defect detection device based on phase motion estimation, characterized in that: include: The first processing module is used to obtain vibration videos of different types of bolt diseases; The second processing module is used to estimate the phase motion of the target area in each frame of the vibration video based on the image processing technology to obtain the displacement signal; A third processing module is used to transform the displacement signal into a time-frequency image based on a time-frequency transformation algorithm; A fourth processing module is used to train a machine learning model using the time-frequency image and the corresponding disease classification labels, and use the trained machine learning model as a bolt disease classification model; The fifth processing module is used to detect the bolt disease type by using the bolt disease classification model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the bolt defect detection method based on phase motion estimation as described in any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the bolt defect detection method based on phase motion estimation as claimed in any one of claims 1 to 6 are implemented.

Images