Method for detecting loose bolts, convolutional neural network, training method and system

Abstract

The present application discloses a method for detecting bolt looseness, a convolutional neural network, a training method and a system, wherein the training method includes: obtaining a vibration image sequence of a test structure under different bolt looseness state tests; extracting vibration displacement signals of a preset number of pixel points in a target area in the vibration image sequence; converting the extracted vibration displacement signal of each pixel point into a time-frequency image; constructing a training sample; using the time-frequency image in the training sample as input and the predicted bolt looseness state category as output, training the convolutional neural network to be trained until convergence. In the present application, a video sample can provide thousands of vibration displacement signals for the training of the convolutional neural network. Sufficient training data can greatly improve the prediction accuracy of the convolutional neural network, thereby solving the problem of shortage of engineering measured data in the training of the convolutional neural network.

Description

Technical Field

[0001] The present application belongs to the technical field of bolt fault detection, and in particular, relates to a method for detecting bolt loosening, a convolutional neural network, a training method and a system. Background Art

[0002] If bolts on mechanical structures become loose, corroded, or creep damaged during use, it may cause catastrophic accidents, immeasurable economic losses, and casualties. Therefore, structural health monitoring (SHM), that is, damage detection strategies and processes that identify features through measurement data recorded in engineering structures, has become increasingly important.

[0003] Among them, the occurrence of damage will affect the inherent properties of the mechanical structure, such as mass, damping ratio and stiffness, and the vibration response of the mechanical structure can reflect these damage characteristics, and these vibration responses will change with different damages. Therefore, in the prior art, structural damage is often identified by analyzing the changes in vibration characteristics.

[0004] In recent years, convolutional neural networks have been widely used to extract and classify vibration features to identify structural damage. However, damage detection using convolutional neural networks is highly dependent on high-quality learning data. In other words, obtaining sufficient training data is crucial to obtaining a convolutional neural network with high accuracy. However, existing methods for obtaining structural vibration features are mainly obtained through accelerometers, strain gauges, laser Doppler vibrometers, etc. These methods are difficult to obtain sufficient vibration feature data sets under different damage conditions in actual engineering scenarios. Summary of the invention

[0005] In order to solve the problem in the prior art that it is difficult to obtain sufficient vibration feature data sets containing different damage conditions, the present application provides a method for detecting bolt loosening, a convolutional neural network, a training method and a system.

[0006] In a first aspect, the present application provides a method for training a convolutional neural network for detecting bolt loosening, comprising:

[0007] Acquire a vibration image sequence of the test structure under different bolt looseness state tests, wherein the vibration image sequence includes multiple frames of vibration images arranged in time series;

[0008] Extracting vibration displacement signals of a preset number of pixel points in the target area of ​​the vibration image sequence;

[0009] Convert the extracted vibration displacement signal of each pixel point into a time-frequency image;

[0010] Constructing a training sample, wherein the training sample includes all bolt loosening state categories tested on the test structure, and a time-frequency image corresponding to each bolt loosening state category;

[0011] The time-frequency image in the training sample is used as input, the predicted bolt loosening state category is used as output, and the convolutional neural network to be trained is trained until convergence.

[0012] In one achievable manner, the method for extracting the vibration displacement signal of the first pixel point in the vibration image sequence includes:

[0013] Extracting grayscale information of a first pixel in each frame of the vibration image, where the first pixel is any pixel in a target area of ​​the vibration image;

[0014] Filtering the extracted grayscale information of each of the first pixels;

[0015] Calculating the phase of the first pixel in each frame of the vibration image according to the filtered grayscale information;

[0016] respectively calculating the phase difference between the first pixel point in each vibration image and the first frame vibration image;

[0017] A vibration displacement signal of the first pixel is generated according to the phase difference of the first pixel in each frame of the vibration image.

[0018] In one achievable manner, converting the vibration displacement signal of each pixel point into a time-frequency image includes:

[0019] Through continuous wavelet transform, the time-frequency image corresponding to the vibration displacement signal of each pixel is drawn.

[0020] In one achievable manner, if the target area includes a sub-area, a time-frequency image in the training sample is used as input, and the predicted bolt loosening state category is used as output, and the convolutional neural network to be trained is trained until convergence.

[0021] In an implementable manner, if the target region includes multiple sub-regions, a time-frequency image is selected from each sub-region to obtain a time-frequency image sequence;

[0022] The time-frequency image sequence is used as input, the predicted bolt loosening state category is used as output, and the convolutional neural network to be trained is trained until convergence.

[0023] In an implementable manner, taking the time-frequency image in the training sample as input and the loose state category as output, training the convolutional neural network to be trained until convergence includes:

[0024] Inputting the time-frequency image in the training sample into the convolutional neural network to be trained;

[0025] After being processed by the convolutional neural network to be trained, outputting a predicted bolt looseness state category;

[0026] Calculating a loss value according to the predicted bolt loosening state category and the bolt loosening state category in the training sample;

[0027] Determine whether the loss value converges, if so, the training ends; or, if not, continue training until the loss value converges.

[0028] In one implementable manner, the convolutional neural network to be trained includes an input layer, a plurality of groups of convolutional layers and pooling layers alternately stacked, a fully connected layer, a classifier, and an output layer;

[0029] Input the time-frequency image in the training sample through the input layer;

[0030] Through the multiple groups of alternately stacked convolutional layers and pooling layers, convolution calculation and downsampling processing are performed on the time-frequency image input to the convolutional neural network to be trained, and a first feature map is output;

[0031] Through the fully connected layer, the first feature map is fully connected to output a second feature map;

[0032] Classifying the second feature map by the classifier;

[0033] Through the output layer, the predicted bolt looseness state category is output.

[0034] In a second aspect, the present application provides a convolutional neural network training system for detecting the loose state of a bolt, comprising:

[0035] A first acquisition module is used to acquire a vibration image sequence of the test structure under different bolt looseness state tests, wherein the vibration image sequence includes multiple frames of vibration images arranged in time series;

[0036] A first extraction module, used to extract vibration displacement signals of a preset number of pixel points within a target area in the vibration image sequence;

[0037] A first conversion module, used for converting the extracted vibration displacement signal of each pixel point into a time-frequency image;

[0038] A training sample construction module, used to construct a training sample, wherein the training sample includes all bolt loosening state categories tested on the test structure, and a time-frequency image corresponding to each bolt loosening state category;

[0039] The training module is used to take the time-frequency image in the training sample as input and the predicted bolt loosening state category as output, and train the convolutional neural network to be trained until convergence.

[0040] In a third aspect, the present application provides a convolutional neural network for detecting the loose state of bolts, comprising: an input layer, a plurality of groups of convolutional layers and pooling layers alternately stacked, a fully connected layer, a classifier, and an output layer;

[0041] Wherein, the input layer is used to input the time-frequency image in the training sample;

[0042] The multiple groups of convolutional layers and pooling layers alternately stacked are used to perform convolution calculation and downsampling processing on the time-frequency image input to the convolutional neural network to be trained, and output a first feature map;

[0043] The fully connected layer is used to perform full connection processing on the first feature map and output a second feature map;

[0044] The classifier is used to classify the second feature map;

[0045] The output layer is used to output the predicted bolt looseness state category.

[0046] In a fourth aspect, the present application provides a method for detecting a loose state of a bolt, comprising:

[0047] Acquire a vibration image sequence of a target structure, wherein the vibration image sequence includes a plurality of frames of vibration images arranged in a time series;

[0048] Extracting vibration displacement signals of a preset number of pixel points in the target area of ​​the vibration image sequence;

[0049] Convert the extracted vibration displacement signal of each pixel point into a time-frequency image;

[0050] Inputting the time-frequency images one by one into a convolutional neural network, wherein the convolutional neural network is a convolutional neural network trained by the training method for a convolutional neural network for detecting bolt loosening described in the first aspect;

[0051] Output the predicted bolt loosening state category corresponding to each time-frequency image;

[0052] The bolt loosening state category of the target structure is determined, wherein the bolt loosening state category of the target structure is the bolt loosening state category with the largest proportion among all the output predicted bolt loosening state categories.

[0053] In a fifth aspect, the present application provides a system for detecting a loose state of a bolt, comprising:

[0054] A second acquisition module is used to acquire a vibration image sequence of the target structure, wherein the vibration image sequence includes a plurality of frames of vibration images arranged in a time series;

[0055] A second extraction module is used to extract vibration displacement signals of a preset number of pixel points within a target area in the vibration image sequence;

[0056] A second conversion module, used for converting the extracted vibration displacement signal of each pixel point into a time-frequency image;

[0057] A convolutional neural network calculation module, used to input the time-frequency images into a convolutional neural network one by one, and output a predicted bolt loosening state category corresponding to each time-frequency image, wherein the convolutional neural network is a convolutional neural network trained by the training method of the convolutional neural network for detecting bolt loosening described in the first aspect;

[0058] The determination module is used to determine the bolt loosening state category of the target structure, wherein the bolt loosening state category of the target structure is the bolt loosening state category with the largest proportion among all the output predicted bolt loosening state categories.

[0059] In summary, the present application provides a method for detecting bolt loosening, a convolutional neural network, a training method and a system, which can obtain a convolutional neural network with reliable accuracy by using only one measured video sample for each bolt loosening state. Among them, one video sample can provide thousands of vibration displacement signals for the training of the convolutional neural network. Sufficient training data can greatly improve the prediction accuracy of the convolutional neural network, thereby solving the problem of shortage of engineering measured data in the training of the convolutional neural network. It can also be seen from the experiment that the training method provided by the present application can not only detect the presence of bolt loosening damage, but also identify different degrees of bolt loosening. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0061] Figure 1 A workflow diagram of a convolutional neural network training method for detecting bolt loosening provided in an embodiment of the present application;

[0062] Figure 2A A schematic diagram of a two-layer steel structure provided in an embodiment of the present application;

[0063] Figure 2BA schematic diagram of a two-layer steel structure connected by bolts provided in an embodiment of the present application;

[0064] Figure 2C A schematic diagram of a structure in which a bolt is loosened provided in an embodiment of the present application;

[0065] Figure 3A A flowchart of a method for extracting a vibration displacement signal of a first pixel point in a vibration image sequence provided in an embodiment of the present application;

[0066] Figure 3B A schematic diagram of extracting a vibration displacement signal of a pixel point in a certain frame of a vibration image is provided for an embodiment of the present application;

[0067] Figure 3C A schematic diagram of vibration displacement signals of the same pixel point in area A under five bolt loosening state categories and vibration displacement signals of the same pixel point in area B under five bolt loosening state categories provided by an embodiment of the present application;

[0068] Figure 4 For Figure 3C Time-frequency images corresponding to each vibration displacement signal;

[0069] Figure 5A A schematic diagram of the structure of a convolutional neural network to be trained provided in an embodiment of the present application;

[0070] Figure 5B The confusion matrix of the predicted classification of the video samples in Experiment 1 of the embodiment of the present application;

[0071] Figure 5C The confusion matrix of the predicted classification of the video samples in Experiment 2 of the embodiment of the present application;

[0072] Figure 5D The confusion matrix of the predicted classification of the video samples in Experiment 3 of the embodiment of the present application;

[0073] Figure 6 A flowchart of a method for detecting a loose state of a bolt provided in an embodiment of the present application;

[0074] Figure 7 A structural block diagram of a convolutional neural network training system for detecting the loose state of bolts provided in an embodiment of the present application;

[0075] Figure 8 A structural block diagram of a system for detecting the loose state of a bolt provided in an embodiment of the present application.

[0076] Description of Reference Numerals

[0077] 10-bottom of two-layer steel structure, 20-first layer plate, 30-second layer plate, 40-vertical arm, 50-corner block. DETAILED DESCRIPTION

[0078] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0079] like Figure 1 As shown, the present application provides a training method for a convolutional neural network for detecting bolt loosening, comprising the following steps:

[0080] Step 100: Acquire a vibration image sequence of a test structure under different bolt looseness state tests, wherein the vibration image sequence includes a plurality of frames of vibration images arranged in time series.

[0081] First of all, it should be noted that the different bolt looseness states described in this application can be the looseness of bolts at different positions in the test structure and / or different degrees of looseness of bolts at the same position. The specific settings can be made according to the actual application scenario, and this application does not limit this.

[0082] Secondly, it should be noted that the test structure in the present application refers to a structure including bolt connections, for example, it can be a one-layer steel structure connected by bolts, or it can be a multi-layer steel structure connected by bolts.

[0083] The present application can use a camera to capture video samples of the test structure in different bolt looseness state tests, wherein each video sample is equivalent to a vibration image sequence formed by multiple frames of vibration images arranged in time series.

[0084] The following is an illustrative description using a two-layer steel structure as an example.

[0085] Figure 2A A schematic diagram of a two-layer steel structure is shown. Figure 2A The middle screw hole is used to install the bolt. Figure 2B Shows Figure 2A The structural diagram after installing the bolts is as follows: Figure 2A and Figure 2BAs shown, the bottom 10 of the two-layer steel structure is fixedly connected to the vibration table by M10 bolts, and the two upper layers (the first layer 20 and the second layer 30) are connected by M6 bolts through four vertical arms 40 and corner blocks 50. There are four M6 bolts at the connection of each corner block 50, of which two are axially in the horizontal direction and the other two are axially in the vertical direction.

[0086] As shown in Table 1, five bolt looseness status categories were set in the test. Figure 2B As shown, position 1 is on the first layer 20, and position 2 is on the second layer 30. Figure 2C As shown in the figure, bolt loosening damage is introduced by artificially loosening the two bolts at positions 1 and 2 with their axes in the horizontal direction to different degrees at the same time, because existing studies have shown that vibrations perpendicular to the axis of the bolts are more likely to cause bolt loosening. The vibration table produces vibrations in the vertical direction, so bolts with their axes in the horizontal direction are more likely to loosen, so we artificially introduce this type of fault. The five bolt loosening state categories set can evaluate the loosening of bolts at different positions and the different degrees of loosening of bolts at the same position.

[0087] Table 1 Bolt looseness status category settings

[0088]

[0089] During the test, the vibration table generates vibrations, which excite the two-layer steel structure fixed to it and cause it to vibrate. Among them, the excitation of the vibration table is not limited to a specific signal, because this application focuses on the difference in vibration signals generated by the two-layer steel structure after being excited under different bolt looseness states, so it is only necessary to ensure the consistency of the excitation signal. In one implementable method, this experiment uses a logarithmic sweep signal with a frequency from 2Hz to 100Hz and a single-pass scanning cycle of 30s. This application can use a Basler acA720-520um camera to shoot a vibration video of the two-layer steel structure, shooting 500 frames per second. Each captured vibration video includes two scanning cycles of forward scanning and reverse scanning with a frequency from 2Hz to 100Hz, totaling 60s.

[0090] It should be noted that in order to train the convolutional neural network for detecting bolt loosening to convergence, only one video sample is collected for each bolt loosening state in the experiment. In order to further verify the effectiveness and accuracy of the training method proposed in this application, 32 video samples are collected for each bolt loosening state in the experiment, among which 1 video sample collected for each bolt loosening state is used to train the convolutional neural network for detecting bolt loosening, and the remaining 31 video samples are used to test whether the trained convolutional neural network is accurate. In this way, there are a total of 32×5=160 video samples in this experiment. That is, in the training method provided in the embodiment of the present application, only 1 video sample for each bolt loosening state can be processed in the subsequent steps 200 to 500. .

[0091] Step 200: extract vibration displacement signals of a preset number of pixel points in a target area in a vibration image sequence.

[0092] The present application does not limit the number and location of the target area, wherein the number of the target area can be one or more, and the location of the target area can include any area on the test structure, for example, the target area is Figure 2B The edge area of ​​the first layer board 20 and the edge area of ​​the second layer board 30 are shown in FIG.

[0093] The present application does not limit the method for determining the target area. For example, the target area can be manually framed or framed by a preset program, and then the vibration displacement signal is extracted based on the vibration image sequence of the framed target area.

[0094] The present application can extract the vibration displacement signal of each pixel point in the target area, or can extract the vibration displacement signal of some pixel points in the target area, which is not limited in the present application. If the vibration displacement signal of some pixel points in the target area is extracted, the vibration displacement signal of a preset number of pixel points in the target area can be randomly extracted.

[0095] The present application can use the same method to extract the vibration displacement signal of any pixel point in the target area. In one possible implementation, the method for extracting the vibration displacement signal of any pixel point in the target area in the vibration image sequence is as follows: Figure 3A As shown, the following steps are included:

[0096] For ease of description, the present application takes the first pixel point in the target area in the vibration image sequence as an example for explanation, wherein the first pixel point may be any pixel point in the target area in the vibration image.

[0097] Step 210: extract the grayscale information of the first pixel in each frame of the vibration image.

[0098] Step 220: Filter the grayscale information of each of the first pixel points extracted.

[0099] Step 230: Calculate the phase of the first pixel in each frame of the vibration image according to the filtered grayscale information.

[0100] Step 240: Calculate the phase difference between the first pixel point in each vibration image and the first frame vibration image.

[0101] Step 250: Generate a vibration displacement signal of the first pixel point according to the phase difference of the first pixel point in each frame of the vibration image.

[0102] The vibration image sequence is composed of multiple frames of continuous vibration images. Since the grayscale value of each pixel of the vibration image will continuously change in the continuous vibration image sequence, a phase-changing time series signal, namely, a vibration displacement signal, can be obtained at each pixel position.

[0103] The following description will be continued by taking the test structure as a two-story steel structure as an example.

[0104] The present application can accurately identify the vibration displacement signal of the test structure from the pixel points near the edge of the test structure. Specifically, the vibration displacement signal can be clearly extracted from all the pixel points on the first layer board 20 and the second layer board 30.

[0105] Figure 3B FIG. 1 shows a schematic diagram of extracting a vibration displacement signal of a pixel point in a certain frame of vibration image. Figure 3B As shown, the target area includes two sub-areas, namely, area A on the first layer board 20 and area B on the second layer board 30. Figure 3B Also shown is a vibration displacement signal of a pixel point extracted from area A, and a vibration displacement signal of a pixel point extracted from area B.

[0106] Figure 3C The diagram shows the vibration displacement signals of the same pixel point in area A under the five bolt loosening state categories in Table 1, and the schematic diagram shows the vibration displacement signals of the same pixel point in area B under the five bolt loosening state categories in Table 1. Figure 3C The middle figure (a) shows the vibration displacement signal of a pixel in area A when the bolt loosening state category is "0". Figure 3C The middle figure (b) shows the vibration displacement signal of a pixel in area B when the bolt loosening state category is "0". Figure 3C The middle figure (c) shows the vibration displacement signal of a pixel in area A when the bolt loosening status category is "1". Figure 3CThe middle figure (d) shows the vibration displacement signal of a pixel in area B when the bolt loosening status category is "1". Figure 3C The middle figure (e) shows the vibration displacement signal of a pixel point in area A when the bolt loosening state category is "2". Figure 3C The middle figure (f) shows the vibration displacement signal of a pixel point in area B when the bolt loosening state category is "2". Figure 3C The middle figure (g) shows the vibration displacement signal of a pixel point in area A when the bolt loosening state category is "3". Figure 3C The middle figure (h) shows the vibration displacement signal of a pixel point in area B when the bolt loosening state category is "3". Figure 3C The middle figure (i) shows the vibration displacement signal of a pixel in area A when the bolt loosening state category is "4". Figure 3C The middle figure (j) shows the vibration displacement signal of a pixel point in area B when the bolt loosening status category is "4". Figure 3C It can be seen that under the same bolt loosening state category, the vibration displacement signals in different areas are different, and under different bolt loosening state categories, the vibration displacement signals in the same area are also different.

[0107] In the experiment, 32 video samples were collected for each bolt loosening state, and a total of 160 video samples were obtained. Therefore, vibration displacement signals of 200 pixels can be randomly extracted from each sub-area in each video sample. In this way, for each bolt loosening state, it includes 200 (number of vibration displacement signals) × 32 (number of video samples) × 2 (number of sub-areas) = ​​12800 vibration displacement signals.

[0108] In summary, this application regards each pixel point on the vibration image sequence as a separate displacement sensor, which is referred to as a "pixel sensor" in this application. In this way, all pixel points on the vibration image sequence can be regarded as sensors to obtain vibration signals. Therefore, a lot of signals can be obtained from a single video sample, which can provide enough training data for the training of the convolutional neural network for detecting bolt loosening.

[0109] Step 300: Convert the extracted vibration displacement signal of each pixel point into a time-frequency image.

[0110] In order to be more conducive to feature extraction and learning of convolutional neural networks, the present application converts the extracted vibration displacement signal of each pixel point into a time-frequency image, and then uses the time-frequency image as input to train the convolutional neural network.

[0111] The present application does not limit the method of converting the vibration displacement signal into the time-frequency image. In one achievable method, the time-frequency image corresponding to the vibration displacement signal of each pixel point can be drawn by continuous wavelet transform, wherein the Daubechies (db) wavelet with good approximate symmetry, tight support and smoothness can be selected as the wavelet basis of the continuous wavelet transform. The drawn time-frequency image contains the joint distribution information of the signal in the time domain and the frequency domain, which is more conducive to the feature extraction and learning of the convolutional neural network.

[0112] Figure 4 Shown with Figure 3C The time-frequency images corresponding to each vibration displacement signal in are given by Figure 4 It can be seen that as the loosening state of the bolt changes, the time-frequency image also changes.

[0113] In the example where the test structure is a two-layer steel structure, for each bolt loosening state, 12,800 time-frequency images are included.

[0114] Step 400: construct a training sample, wherein the training sample includes all bolt loosening state categories tested on the test structure, and time-frequency images corresponding to each bolt loosening state category.

[0115] In the above steps, 32 video samples of each bolt loose state are obtained, but enough vibration displacement signals can be extracted from each video sample. Therefore, using 1 video sample for each bolt loose state in this application can provide sufficient training data for the training of the convolutional neural network.

[0116] In this way, for each bolt loose state, a video sample can be randomly selected from the 32 video samples as the data source of the training sample, and the remaining 31 video samples are used as the data source of the test sample. The constructed training samples include five bolt loose state categories and time-frequency images corresponding to the five bolt loose state categories.

[0117] Step 500: Using the time-frequency image in the training sample as input and the predicted bolt loosening state category as output, the convolutional neural network to be trained is trained until convergence.

[0118] Taking the time-frequency image in the training sample as input and the predicted bolt loosening state category as output, the convolutional neural network to be trained is trained until convergence, which may specifically include the following steps:

[0119] Step 510: Input the time-frequency images in the training samples into the convolutional neural network to be trained.

[0120] It should be noted here that, since the target area may include one or more sub-areas, the time-frequency image in the training sample may be extracted from one sub-area in a video sample, or may be extracted from multiple sub-areas in a video sample. Correspondingly, if the time-frequency image in the training sample is extracted from one sub-area, that is, the target area includes one sub-area, then during training, one time-frequency image in the training sample is used as input each time and input into the convolutional neural network to be trained; if the time-frequency image in the training sample is extracted from multiple sub-areas, that is, the target area includes multiple sub-areas, then during training, one time-frequency image can be selected from each sub-area each time to obtain a time-frequency image sequence, and then the time-frequency image sequence can be used as input and input into the convolutional neural network to be trained.

[0121] Step 520: After being processed by the convolutional neural network to be trained, the predicted bolt looseness state category is output.

[0122] The present application does not limit the structure of the convolutional neural network to be trained, as long as it can learn the characteristics of bolts in different loose states based on the input time-frequency image and classify the loose state of the bolts of the test structure.

[0123] Figure 5A The structure of a convolutional neural network to be trained is shown in FIG. Figure 5A As shown, the convolutional neural network to be trained includes an input layer, multiple groups of alternatingly stacked convolutional layers and pooling layers, a fully connected layer, a classifier, and an output layer.

[0124] Among them, the input layer is used to input the time-frequency image in the training sample; the multiple groups of convolutional layers and pooling layers arranged alternately and stacked are used to perform convolution calculation and downsampling processing on the time-frequency image input to the convolutional neural network to be trained, and output a first feature map; the fully connected layer is used to fully connect the first feature map and output a second feature map; the classifier is used to classify the second feature map; the output layer is used to output the predicted bolt loosening state category.

[0125] Thus, during training, the time-frequency images in the training samples are first input into the input layer, where the time-frequency images include three color channels: R, G, and B. Then, multiple sets of alternately stacked convolutional layers and pooling layers (e.g. Figure 5AThe convolution layer 1, pooling layer 1, convolution layer 2, pooling layer 2, etc. are shown in the figure. The time-frequency image of the convolutional neural network to be trained is convoluted and downsampled, and the first feature map is output. The convolution layer extracts and learns features through convolution calculation. In this way, each convolution layer generates a feature map after convolution, and the pooling layer reduces the dimension of these feature maps to control the amount of data. Then, the first feature map is output, and the second feature map is output after the full connection layer performs full connection processing, wherein the full connection processing can flatten the data. Finally, the second feature map is classified into the loose state of the bolts through the classifier, and the output layer outputs the predicted loose state category of the bolts.

[0126] The structure of the convolutional neural network to be trained provided above is simple and compact, and the hidden layer has only a few convolutional layers, pooling layers and fully connected layers, so its computational efficiency is relatively high. The training method provided in this application does not require an overly complex and deep convolutional neural network structure.

[0127] Step 530: Calculate a loss value according to the predicted loose state category and the loose state category in the training sample.

[0128] Step 540: Determine whether the loss value converges. If so, the training ends; or, if not, continue training until the loss value converges.

[0129] The process of training the convolutional neural network to be trained is the process of continuously optimizing the network parameters in the convolutional neural network to be trained, that is, each time step 510 to step 530 is executed, the network parameters of the convolutional neural network to be trained are obtained. If the loss value does not converge, the network parameters of the convolutional neural network to be trained are updated, and then the convolutional neural network to be trained after the updated network parameters is used to continue training, and the network parameters of the convolutional neural network to be trained are continuously updated until the loss value converges, the training ends, and the trained convolutional neural network is obtained.

[0130] It should be noted that the structure of the trained convolutional neural network is the same as that of the trained convolutional neural network. The difference is that the network parameter values ​​of the trained convolutional neural network are different from those of the trained convolutional neural network, that is, the trained convolutional neural network also includes an input layer, multiple groups of alternatingly stacked convolutional layers and pooling layers, a fully connected layer, a classifier and an output layer.

[0131] Next, we continue to take the test structure as a two-layer steel structure as an example, and use a single time-frequency image and a time-frequency image sequence composed of multiple time-frequency images as input to illustrate the training method of the convolutional neural network to be trained.

[0132] First, three groups of control experiments are set up, as shown in Table 2. In experiment 1, only the vibration displacement signals of the pixels in area A are used to train the convolutional neural network to be trained. In this case, the convolutional neural network to be trained is a two-dimensional convolutional neural network, and the input sample data of the two-dimensional convolutional neural network is a single time-frequency image generated by the vibration displacement signals of the pixels extracted from area A. Experiment 2 is similar to experiment 1, except that area A is changed to area B. In experiment 3, the vibration displacement signals of the pixels in areas A and B are used together to train the convolutional neural network to be trained. In this case, the convolutional neural network to be trained is a three-dimensional convolutional neural network, and the input sample data of the three-dimensional convolutional neural network is a two-frame time-frequency image sequence generated by combining the time-frequency image of the vibration displacement signals of the pixels extracted from area A with the time-frequency image of the vibration displacement signals of the pixels extracted from area B.

[0133] Table 2 Selection of regions in three experiments

[0134]

[0135] Among them, Table 3 and Table 4 list the basic structures and network parameters of the two-dimensional convolutional neural network and the three-dimensional convolutional neural network respectively. The Adam optimizer is used in the training process of the convolutional neural network, and the learning rate is set to 0.001. Before inputting into the convolutional neural network to be trained, each time-frequency image is scaled to a size of 150×150 pixels, and the grayscale values ​​of all pixels are standardized to [0,1].

[0136] Table 3 Structure and network parameters of the two-dimensional convolutional neural network in Experiment 1 and Experiment 2

[0137]

[0138] Table 4 Structure and network parameters of the 3D convolutional neural network in Experiment 3

[0139]

[0140] Next, we further verify the effectiveness and accuracy of the trained convolutional neural network through test samples.

[0141] For each bolt loosening state, there are 31 video samples that can be used as test samples. In this way, the time-frequency image corresponding to each video sample is input into the trained convolutional neural network, and a bolt loosening category is output accordingly. The bolt loosening category corresponding to the video sample is finally determined according to the "election method", that is, the category with the most categories classified in the 200 time-frequency images extracted from each video sample is regarded as the true classification of this video sample. For example, 150 of the 200 time-frequency image samples extracted from a certain video sample are predicted to be bolt loosening category 0, and the remaining 50 are predicted to be other categories, then the video sample is classified as bolt loosening category 0.

[0142] This application uses 31 video samples to verify the accuracy of the convolutional neural network trained for the three experiments in Table 2. The verification results are shown in Table 5. FIG. 5B to FIG. 5D The confusion matrices of the predicted classifications of the video samples in the three experiments are shown respectively.

[0143] Table 5 Prediction accuracy of video samples in three experiments

[0144]

[0145] From Table 5 and FIG. 5B to FIG. 5D It can be seen that the prediction accuracy of Experiment 3 is significantly higher than that of Experiments 1 and 2. Therefore, using vibration displacement signals of pixels at multiple regional locations is more conducive to improving the performance of the convolutional neural network.

[0146] As shown in Table 6, the present application also compares four convolutional neural network training methods in the prior art.

[0147] Table 6 Comparison of training methods for neural networks to detect bolt loosening

[0148]

[0149] From the comparison in Table 6, it can be seen that the prior art requires not only enough sensors to be used but also enough training samples to be collected for each bolt loosening state when training the neural network. However, the training method provided in this application only requires one camera (i.e., the sensor shown in Table 6) and one video sample for each bolt loosening state to train a convolutional neural network with an accuracy of 100%.

[0150] In summary, the training method of the convolutional neural network for detecting bolt loosening provided in the present application only uses one measured video sample in each bolt loose state to obtain a convolutional neural network with reliable accuracy. Among them, one video sample can provide thousands of vibration displacement signals for the training of the convolutional neural network. Sufficient training data can greatly improve the prediction accuracy of the convolutional neural network, thereby solving the problem of shortage of engineering measured data in the training of the convolutional neural network. It can also be seen from the experiment that the training method provided in the present application can not only detect the presence of bolt loosening damage, but also identify different degrees of bolt loosening.

[0151] like Figure 6 As shown, the present application also provides a method for detecting the loose state of a bolt, which uses a convolutional neural network trained by the training method provided in the above embodiment to detect the loose state of the bolt, and specifically includes the following steps:

[0152] Step 10: Acquire a vibration image sequence of the target structure, wherein the vibration image sequence includes multiple frames of vibration images arranged in time series.

[0153] Step 10 can refer to the description of the above step 100, which will not be repeated here. The difference is that the object to be detected in step 10 is a target structure, which has a bolt connection structure, and the looseness state of the bolts on the target structure is unclear.

[0154] Step 20: extract vibration displacement signals of a preset number of pixel points in the target area of ​​the vibration image sequence.

[0155] Step 30: Convert the extracted vibration displacement signal of each pixel point into a time-frequency image.

[0156] The above steps 20 and 30 can refer to the description of the above steps 200 and 300, which will not be repeated here. It should be noted that the target area and the preset number of pixels determined in the method for detecting the loose state of the bolt can be the same as or different from the target area and the preset number of pixels used in the training method of the convolutional neural network, and this application does not limit this.

[0157] Step 40: Input the time-frequency images one by one into a convolutional neural network, and output a predicted bolt loosening state category corresponding to each time-frequency image.

[0158] The convolutional neural network used in step 40 is a convolutional neural network trained using the training method for a convolutional neural network for detecting bolt loosening provided in the above embodiment, that is, a convolutional neural network trained to convergence.

[0159] The above embodiments have verified through test data that the accuracy of the convolutional neural network trained in the present application is high. Therefore, the trained convolutional neural network can output reliable classification results.

[0160] Step 50: Determine the bolt loosening state category of the target structure, wherein the bolt loosening state category of the target structure is the bolt loosening state category with the largest number among all the output predicted bolt loosening state categories.

[0161] First of all, it should be noted that after step 40, the predicted bolt loose state category corresponding to each input time-frequency image can be obtained, that is, based on the vibration displacement signal of a preset number of pixel points in the target area, a lot of predicted bolt loose state categories can be obtained, and the bolt loose state of the target structure should be a certain one. Therefore, the present application determines the bolt loose state category of the target structure through an "election method".

[0162] Specifically, the number of each predicted bolt loosening state category output by step 40 can be counted respectively; then, the category with the largest number of categories is used as the bolt loosening state category of the target structure. For example, the total number of predicted bolt loosening state categories output by step 40 is 200, and the types of predicted bolt loosening state categories include 5 types, namely category "0", category "1", category "2", category "3", and category "4". Among them, the number of output category "0" is 180, and the total number of other categories is 20, then the bolt loosening state category of the target structure is determined to be "0".

[0163] In summary, the method for detecting the loose state of a bolt provided in the embodiment of the present application has the following advantages:

[0164] (1) High spatial resolution, each pixel can be used as a sensor, so a large amount of signal data can be obtained at the same time; (2) It does not change the dynamic characteristics of the structure because this method does not require contact with the object being measured; (3) This method is simple to set up and has low cost. The equipment required is not expensive and does not require additional marking points; (4) This method is less affected by ambient temperature and other conditions; (5) Long-distance measurement can be achieved; (6) Compared with acceleration signals, displacement response signals can directly reflect the overall stiffness of the structure, which is of great help in structural evaluation.

[0165] like Figure 7 As shown, the present application also provides a convolutional neural network training system for detecting the loose state of bolts, including:

[0166] A first acquisition module 110 is used to acquire a vibration image sequence of the test structure under different bolt loosening state tests, wherein the vibration image sequence includes multiple frames of vibration images arranged in time series;

[0167] A first extraction module 120 is used to extract vibration displacement signals of a preset number of pixel points in a target area in the vibration image sequence;

[0168] A first conversion module 130, used to convert the extracted vibration displacement signal of each pixel point into a time-frequency image;

[0169] A training sample construction module 140 is used to construct a training sample, wherein the training sample includes all bolt loosening state categories tested on the test structure, and a time-frequency image corresponding to each bolt loosening state category;

[0170] The training module 150 is used to take the time-frequency image in the training sample as input and the predicted bolt loosening state category as output, and train the convolutional neural network to be trained until convergence.

[0171] like Figure 8 As shown, the present application also provides a system for detecting a loose state of a bolt, comprising:

[0172] A second acquisition module 210 is used to acquire a vibration image sequence of the target structure, wherein the vibration image sequence includes a plurality of frames of vibration images arranged in a time series;

[0173] A second extraction module 220 is used to extract vibration displacement signals of a preset number of pixel points within the target area of ​​the vibration image sequence;

[0174] The second conversion module 230 is used to convert the extracted vibration displacement signal of each pixel point into a time-frequency image;

[0175] A convolutional neural network calculation module 240, used to input the time-frequency images into a convolutional neural network one by one, and output a predicted bolt loosening state category corresponding to each time-frequency image, wherein the convolutional neural network is a convolutional neural network trained by the training method for a convolutional neural network for detecting bolt loosening according to claim 1;

[0176] The determination module 250 is used to determine the bolt loosening state category of the target structure, wherein the bolt loosening state category of the target structure is the bolt loosening state category with the largest proportion among all the output predicted bolt loosening state categories.

[0177] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0178] The present application is described in detail above in conjunction with specific implementation methods and exemplary examples, but these descriptions cannot be understood as limiting the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, a variety of equivalent replacements, modifications or improvements can be made to the technical solution of the present application and its implementation methods, all of which fall within the scope of the present application. The scope of protection of the present application shall be subject to the attached claims.

[0179] In a specific implementation, the embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, and when the program is executed, the program may include some or all of the steps in each embodiment of the convolutional neural network training method for detecting bolt looseness and / or the method for detecting the loose state of a bolt provided in the present application. The storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0180] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution in the embodiments of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application or some parts of the embodiments.

[0181] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A method for training a convolutional neural network for detecting bolt loosening, characterized in that: include: Acquire a vibration image sequence of the test structure under different bolt looseness state tests, wherein the vibration image sequence includes multiple frames of vibration images arranged in time series; Extracting vibration displacement signals of a preset number of pixel points in the target area of ​​the vibration image sequence; Convert the extracted vibration displacement signal of each pixel point into a time-frequency image; Constructing a training sample, wherein the training sample includes all bolt loosening state categories tested on the test structure, and a time-frequency image corresponding to each bolt loosening state category; The time-frequency image in the training sample is used as input, the predicted bolt loosening state category is used as output, and the convolutional neural network to be trained is trained until convergence.

2. The training method according to claim 1, characterized in that: The method for extracting the vibration displacement signal of the first pixel point in the vibration image sequence comprises: Extracting grayscale information of a first pixel in each frame of the vibration image, where the first pixel is any pixel in a target area of ​​the vibration image; Filtering the extracted grayscale information of each of the first pixels; Calculating the phase of the first pixel in each frame of the vibration image according to the filtered grayscale information; respectively calculating the phase difference between the first pixel point in each vibration image and the first frame vibration image; A vibration displacement signal of the first pixel is generated according to the phase difference of the first pixel in each frame of the vibration image.

3. The training method according to claim 1, characterized in that: The step of converting the extracted vibration displacement signal of each pixel point into a time-frequency image comprises: Through continuous wavelet transform, the time-frequency image corresponding to the vibration displacement signal of each pixel is drawn.

4. The training method according to claim 1, characterized in that: If the target area includes a sub-area, a time-frequency image in the training sample is used as input, and the predicted bolt loosening state category is used as output, and the convolutional neural network to be trained is trained until convergence.

5. The training method according to claim 1, characterized in that: If the target area includes multiple sub-areas, a time-frequency image is selected from each sub-area to obtain a time-frequency image sequence; The time-frequency image sequence is used as input, the predicted bolt loosening state category is used as output, and the convolutional neural network to be trained is trained until convergence.

6. The training method according to claim 1, characterized in that: Taking the time-frequency image in the training sample as input and the predicted bolt loosening state category as output, training the convolutional neural network to be trained until convergence, including: Inputting the time-frequency image in the training sample into the convolutional neural network to be trained; After being processed by the convolutional neural network to be trained, outputting a predicted bolt looseness state category; Calculating a loss value according to the predicted bolt loosening state category and the bolt loosening state category in the training sample; Determine whether the loss value converges, if so, the training ends; or, if not, continue training until the loss value converges.

7. The training method according to claim 1, characterized in that: The convolutional neural network to be trained includes an input layer, a plurality of groups of convolutional layers and pooling layers alternately stacked, a fully connected layer, a classifier, and an output layer; Input the time-frequency image in the training sample through the input layer; Through the multiple groups of alternately stacked convolutional layers and pooling layers, convolution calculation and downsampling processing are performed on the time-frequency image input to the convolutional neural network to be trained, and a first feature map is output; Through the fully connected layer, the first feature map is fully connected to output a second feature map; Classifying the second feature map by the classifier; Through the output layer, the predicted bolt looseness state category is output.

8. A convolutional neural network training system for detecting bolt looseness, characterized in that: include: A first acquisition module is used to acquire a vibration image sequence of the test structure under different bolt looseness state tests, wherein the vibration image sequence includes multiple frames of vibration images arranged in time series; A first extraction module, used to extract vibration displacement signals of a preset number of pixel points within a target area in the vibration image sequence; A first conversion module, used for converting the extracted vibration displacement signal of each pixel point into a time-frequency image; A training sample construction module, used to construct a training sample, wherein the training sample includes all bolt loosening state categories tested on the test structure, and a time-frequency image corresponding to each bolt loosening state category; The training module is used to take the time-frequency image in the training sample as input and the predicted bolt loosening state category as output, and train the convolutional neural network to be trained until convergence.

9. A method for detecting the loose state of a bolt, characterized in that: include: Acquire a vibration image sequence of the target structure, wherein the vibration image sequence includes a plurality of frames of vibration images arranged in a time series; Extracting vibration displacement signals of a preset number of pixel points in the target area of ​​the vibration image sequence; Convert the extracted vibration displacement signal of each pixel point into a time-frequency image; Inputting the time-frequency images one by one into a convolutional neural network, wherein the convolutional neural network is a convolutional neural network trained by the training method for a convolutional neural network for detecting bolt loosening according to claim 1; Output the predicted bolt loosening state category corresponding to each time-frequency image; The bolt loosening state category of the target structure is determined, wherein the bolt loosening state category of the target structure is the bolt loosening state category with the largest proportion among all the output predicted bolt loosening state categories.

10. A system for detecting the loose state of a bolt, characterized in that: include: A second acquisition module is used to acquire a vibration image sequence of the target structure, wherein the vibration image sequence includes a plurality of frames of vibration images arranged in a time series; A second extraction module is used to extract vibration displacement signals of a preset number of pixel points within a target area in the vibration image sequence; A second conversion module, used for converting the extracted vibration displacement signal of each pixel point into a time-frequency image; A convolutional neural network calculation module, used to input the time-frequency images into a convolutional neural network one by one, and output a predicted bolt loosening state category corresponding to each time-frequency image, wherein the convolutional neural network is a convolutional neural network trained by the training method of the convolutional neural network for detecting bolt loosening according to claim 1; The determination module is used to determine the bolt loosening state category of the target structure, wherein the bolt loosening state category of the target structure is the bolt loosening state category with the largest proportion among all the output predicted bolt loosening state categories.

Images