Method for detecting loose bolts in dense bolted structures, terminal device and storage medium

Abstract

The application is suitable for the technical field of image processing, and provides a dense bolt loosening video detection method, a terminal device and a storage medium. The method comprises the following steps: collecting a video image of a bolt; based on the video image of the bolt, the video image is segmented according to continuous video sequences of the same node plate region to obtain a segmented video image; a bolt shadow area feature is extracted from the segmented video image; based on the bolt shadow area feature, adjacent frame image bolt shadow correlation operations are sequentially performed on image sequences of the same node plate region to obtain a target shadow feature time sequence of a single bolt; and based on the target shadow feature time sequence of the single bolt, a loosening bolt is determined. The application can realize accurate detection of the loosening state of dense bolts of a large-span steel bridge.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to a video detection method, terminal equipment and storage medium for loose bolts. Background Technology

[0002] High-strength bolt connections, as an important method of steel bridge connection, have been widely used in bridge engineering due to their advantages of high connection strength and simple construction. However, with the long-term service of bridges, such as the bolt groups in densely bolted areas of long-span steel bridges, [the following text appears to be incomplete and requires further context: "...such as..."] Figure 1 As shown in the solid line box, the bridge is subjected to constant vibration and impact during operation. These negative conditions can easily cause bolts to fatigue, loosen, and fall off, seriously threatening bridge safety. Therefore, effectively assessing the loosening status of bolts is an important aspect of ensuring the safe passage of steel bridges.

[0003] Compared to bolt detachment, bolt loosening is more subtle. After a bolt loosens, its visual characteristics don't change significantly, making it difficult to identify the degree of loosening manually. Furthermore, when dealing with densely packed bolts on steel bridges, workers need to inspect a large number of bolts, requiring them to perform extremely difficult climbing operations. Sensor detection typically involves deploying sensors near the bolts to collect and identify abnormal bolt loosening signals. Sensor detection can obtain relatively accurate loosening detection results in localized areas. However, when inspecting steel bridge bolts, the large number of bolts in the gusset plate area, and the fact that bolt loosening often occurs randomly in one or two locations, makes targeted sensor deployment impossible, leading to excessively long deployment times. In addition, the need for manual sensor installation and deployment significantly increases the difficulty of sensor setup when inspecting bolts located high up on the steel bridge.

[0004] In recent years, deep learning technology has developed rapidly in various fields, especially in machine vision, where its applications are widespread. Compared to methods such as manual inspection and sensor inspection, using machine vision to assess bolt looseness can significantly reduce deployment time and difficulty. Machine vision inspection mainly involves capturing images of the bolt area with a camera, then extracting features from the captured images or videos to assess bolt looseness. Existing technologies include using Faster-RCNN networks to locate bolt area images and extracting five key nodes of the bolt through cascaded pyramid networks, using geometric imaging theory to calculate the length of exposed bolts; or using attention-based multi-task learning networks to achieve accurate bolt looseness detection; or establishing image datasets of normal bolts and loose bolts and using Faster-RCNN networks to detect loose bolts; or using one-dimensional CNNs to automatically extract and learn impedance feature changes in the bolt area to assess bolt looseness. However, for the problem of identifying bolt loosening due to its significant concealment characteristics, the aforementioned methods, which only process single-frame image information, cannot obtain sufficiently discriminative and effective bolt loosening features due to the large number of bolts, high bolt density, and variable shooting angles in densely bolted areas of long-span steel bridges. This results in poor bolt loosening identification performance. In summary, there is currently no effective detection method for the dense bolt loosening status of long-span steel bridges. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this application provides a video detection method, terminal equipment and storage medium for loosening of dense bolts, which can realize the accurate detection of the loosening state of dense bolts in long-span steel bridges.

[0006] This application is achieved through the following technical solution:

[0007] In a first aspect, embodiments of this application provide a video detection method for loosening of dense bolts, including:

[0008] Capture video images of the bolts;

[0009] Based on the bolt-based video images, the video images are segmented according to the continuous video sequence of the same node plate area to obtain the segmented video images;

[0010] Extract bolt shadow area features from the segmented video images;

[0011] Based on the bolt shadow area features, the bolt shadow association operation of adjacent frame images is performed sequentially on the image sequence of the same node plate region to obtain the target shadow feature time series of a single bolt.

[0012] Loose bolts are identified based on the time series of target shadow features of a single bolt.

[0013] In one possible implementation of the first aspect, the video images based on the bolts are segmented according to a continuous video sequence of the same node plate region to obtain segmented video images, including:

[0014] Based on video images of bolts, obtain feature descriptions of the same node plate;

[0015] Based on the feature description of the same node board, the spatiotemporal relationship of the same node board region in consecutive frame images is established.

[0016] Based on the spatiotemporal correlation of the same node board region in consecutive frame images, a continuous video sequence of the same node board region is obtained;

[0017] Segmented video images are obtained from continuous video sequences based on the same node board region.

[0018] In one possible implementation of the first aspect, the characteristic description of the same node plate includes the same node plate area type, the coordinate position of the same node plate area, and the number of bolts in the same node plate area;

[0019] Based on the feature description of the same node board, the spatiotemporal correlation of the same node board region in consecutive frame images is established, including:

[0020] Based on the node plate region type of two consecutive frame images, the distance between the node plate regions of two consecutive frame images, and the number of bolts in the node plate regions of two consecutive frame images, a similarity determination result sequence is determined; the distance between the node plate regions of two consecutive frame images is determined by the coordinate positions of two adjacent node plate regions.

[0021] Extract the minimum value from the similarity determination result sequence;

[0022] If the minimum value in the similarity determination result sequence is not greater than the preset threshold, then the two node board regions corresponding to the minimum value in the similarity determination result sequence establish a spatiotemporal association of the same node board region in consecutive frame images.

[0023] In one possible implementation of the first aspect, the determination function F(X) of the similarity determination result sequence i,j ,X i+1,k The expression for ) is:

[0024]

[0025] in, A function to determine the type of two adjacent node plate regions; The weighted calculation function for the distance between two adjacent node plate regions; This is a function to determine the number of bolts in two adjacent node plate regions. Let j be the type of the node board region identified in the i-th frame image. Let J be the coordinates of the j-th node plate region identified in the i-th frame image; The number of bolts identified in the j-th node plate region in the i-th frame image; Let k be the type of the k-th node board region identified in the (i+1)-th frame image. The coordinates of the kth node plate region identified in the (i+1)th frame image; The number of bolts in the region of the kth node plate identified in the (i+1)th frame image.

[0026] In one possible implementation of the first aspect, a continuous video sequence of the same node board region is obtained based on the spatiotemporal correlation of the same node board region in consecutive frame images, including:

[0027] If the same node board region establishes a spatiotemporal association in a consecutive first preset number of frame images, then it is determined that the same node board region appears in the video images after the consecutive first preset number of frame images, and the frame number of the first frame image in the consecutive first preset number of frame images is determined as the starting frame number.

[0028] After determining the starting frame number, if the same node board region cannot establish a spatiotemporal association in the second consecutive preset number of frame images, it is determined that the same node board region disappears in the video images after the second consecutive preset number of frame images, and the frame number of the last frame image in the second consecutive preset number of frame images is determined as the termination frame number; the continuous video sequence of the same node board region is the starting frame number to the termination frame number.

[0029] In one possible implementation of the first aspect, extracting bolt shadow area features from the segmented video image includes:

[0030] The segmented video images are input into the improved Unet network structure; the improved Unet network structure includes a multi-scale concatenated enhancement module; the multi-scale concatenated enhancement module is used to extract spatial information from the low-level feature maps of the segmented video images input into the improved Unet network structure;

[0031] Improved Unet network structure outputs bolt shadow area characteristics.

[0032] In one possible implementation of the first aspect, based on the bolt shadow area features, the bolt shadow association operation is performed sequentially on adjacent frame images of the same node plate region to obtain a time series of target shadow features for a single bolt, including:

[0033] For two consecutive frame images A and B in the same node board region, the spectrum of the two consecutive frame images is calculated using Fast Fourier Transform; image A is used as the template image and image B is used as the test image.

[0034] Based on the spectra of two consecutive frames, the cross-power spectrum of the two consecutive frames is obtained by multiplication.

[0035] Calculate the inverse Fourier transform of the cross-power spectrum of two consecutive frames, and move the zero-frequency component in the inverse Fourier transform of the cross-power spectrum of the two consecutive frames to the center of the result.

[0036] Based on the result center, the content correlation coefficient matrix of two consecutive frame images is obtained, and the coordinates of the position of the maximum value in the matrix are extracted.

[0037] Translate the center of image B to the position coordinates and associate the bolt shadows of two consecutive frames, image A and image B, in the same node plate region;

[0038] Following the above process of bolt shadow association for two consecutive frame images A and B of the same node plate region, the bolt shadow association operation of adjacent frame images is performed sequentially on the image sequence of the same node plate region to obtain the target shadow feature time series of all individual bolts in the same node plate region.

[0039] In one possible implementation of the first aspect, determining a loose bolt based on a time series of target shadow features of a single bolt includes:

[0040] Based on the target shadow feature time series of a single bolt, the target shadow feature time series of each bolt is normalized to obtain the normalized value of the target shadow feature time series of all bolts in the same node plate area.

[0041] K-means classification was performed on the normalized time series values ​​of the target shadow features of all bolts within the same node plate area to obtain the classification results;

[0042] Based on the classification results, the loose bolts were identified.

[0043] Secondly, embodiments of this application provide a terminal device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the dense bolt loosening video detection method as described in the first aspect.

[0044] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the dense bolt loosening video detection method as described in the first aspect.

[0045] The advantages of the embodiments in this application compared with related technologies are:

[0046] In this embodiment, the video images of bolts are precisely segmented through a continuous video sequence of the same node plate area. This allows for focusing on the continuous video sequence of the same node plate area, avoiding unnecessary image processing and improving detection efficiency. Furthermore, using the bolt shadow as a visual feature of bolt loosening can accurately reflect the relative positional relationship between the bolt and the background. By sequentially performing bolt shadow association operations on adjacent frames of the image sequence of the same node plate area, spatiotemporal correlation calculations are performed between bolt loosening features in the video clips, thereby obtaining richer bolt loosening-related features. This helps to accurately determine the state changes of bolts and achieve precise detection of the dense bolt loosening state of large-span steel bridges.

[0047] The beneficial effects of the second and third aspects mentioned above are the same as those of the beneficial effects of the dense bolt loosening video detection system in the first aspect, and will not be repeated here.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a diagram of a bolt group for a long-span steel bridge according to an embodiment of this application;

[0051] Figure 2 This is a schematic flowchart of a video detection method for loose bolts provided in an embodiment of this application;

[0052] Figure 3 This is a schematic flowchart of a video image segmentation process provided in an embodiment of this application;

[0053] Figure 4 This is a schematic diagram illustrating the determination of the start frame number and the end frame number according to an embodiment of this application;

[0054] Figure 5 This is a schematic diagram showing the shadow cast on the bolt cluster area according to an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of a normal bolt provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of a loose bolt provided in an embodiment of this application;

[0057] Figure 8 This is an improved Unet network structure diagram provided in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram illustrating the correlation principle of bolt shadow features in adjacent frame images according to an embodiment of this application;

[0059] Figure 10 This is a schematic diagram illustrating the correlation principle of bolt shadow features in sequentially adjacent frame images provided in an embodiment of this application;

[0060] Figure 11 This is a typical time series of target shadow features of a bolt provided in an embodiment of this application;

[0061] Figure 12 This is a schematic diagram of the time series of shadow features of different bolts provided in an embodiment of this application;

[0062] Figure 13 This is a schematic diagram of the bolt k-means classification results provided in an embodiment of this application;

[0063] Figure 14 This is a schematic diagram of the structure of a video detection device for loose bolts provided in one embodiment of this application;

[0064] Figure 15 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0065] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0066] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0067] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0068] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0069] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0071] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0072] Figure 2 This is a schematic diagram of the structure of a video detection system for loose bolts provided in an embodiment of this application, with reference to... Figure 2 The following is a detailed description of the video detection method for loosening of dense bolts:

[0073] Step 101: Acquire video images of the bolts.

[0074] Considering that the dense bolts of long-span steel bridges are located at a high position, video images of the dense bolts can be collected by drones.

[0075] Step 102: Based on the video image of the bolt, segment the video image according to the continuous video sequence of the same node plate area to obtain the segmented video image.

[0076] Considering that there may be a large number of image sequences that do not contain bolt targets in the video images of dense bolts on a long-span steel bridge captured by drones, and that most of the bolts on the outer side of the steel bridge are distributed in the node plate area, it is necessary to extract the bolt video segments first.

[0077] See Figure 3 Step 102 includes:

[0078] Step 1021: Based on the video images of the bolts, obtain the feature description of the same node plate.

[0079] The characteristic description of the same gusset plate includes the same gusset plate area type, the coordinate position of the same gusset plate area, and the number of bolts in the same gusset plate area.

[0080] Let the feature of the j-th node plate region identified in a certain frame image i be . For node board region type, Let be the coordinates of the node plate region. Assume that within the j-th node plate region, detection is possible. The j-th node plate region is characterized by a number of bolts.

[0081] In this embodiment, a node plate region localization model and a single bolt target localization model can be established using a YOLO-V5 network. The node plate region type and coordinate position are obtained through the established node plate region localization model, while the number of bolts within the node plate region is obtained through the established single bolt target localization model. Using these two localization models to obtain the node plate region type, coordinate position, and number of bolts within the node plate region, a determination function for the same node plate region within two consecutive frames is proposed. This completes the determination of the same node plate region in the video image, ultimately achieving the segmentation of continuous video sequences containing the same node plate region.

[0082] Step 1022: Based on the feature description of the same node board, establish the spatiotemporal correlation of the same node board region in consecutive frame images.

[0083] To fully utilize the spatiotemporal redundancy information of the same target in video, it is necessary to determine the node plate region X of two consecutive frames. i,j With X i+1,k Whether they are the same object, step 1022 includes:

[0084] Based on the node plate region type of two consecutive frame images, the distance between the node plate regions of the two consecutive frame images, and the number of bolts in the node plate regions of the two consecutive frame images, a similarity determination result sequence is determined; the distance between the node plate regions of two consecutive frame images is determined by the coordinate positions of two adjacent node plate regions.

[0085] The determination function F(X) of the similarity determination result sequence i,j ,X i+1,k The expression for ) is:

[0086]

[0087]

[0088]

[0089]

[0090] in, A function to determine the type of two adjacent node plate regions; The weighted calculation function is the distance between two adjacent node regions; dist(·,·) is the Euclidean distance calculation function between the centers of two adjacent node regions; and ln(·) is the natural logarithm calculation function. This is a function to determine the number of bolts in two adjacent node plate regions. Let j be the type of the node board region identified in the i-th frame image. Let J be the coordinates of the j-th node plate region identified in the i-th frame image; The number of bolts identified in the j-th node plate region in the i-th frame image; Let k be the type of the k-th node board region identified in the (i+1)-th frame image. The coordinates of the kth node plate region identified in the (i+1)th frame image; The number of bolts in the region of the kth node plate identified in the (i+1)th frame image.

[0091] Suppose that m node regions are detected in the i-th frame, and a certain node region is identified as X in the (i+1)-th frame. i+1,k Using formula (1), X can be obtained. i+1,k The similarity determination result sequence of all node plate region targets in the previous frame {F(X i,1 ,X i+1,k ),F(X i,2 ,X i+1,k ),...,F(X i,m ,X i+1,k )}.

[0092] According to formula (5), the minimum value in the similarity determination result sequence is extracted.

[0093] min{F(X i,1 ,X i+1,k ),F(X i,2 ,X i+1,k ),...,F(X i,m ,X i+1,k (5)

[0094] If the minimum value in the similarity determination result sequence is not greater than a preset threshold, then the two node plate regions corresponding to the minimum value in the similarity determination result sequence establish a spatiotemporal association of the same node plate region in consecutive frame images. For example, assuming the preset threshold is set to 5, F(X) i,2 ,X i+1,k If ) is the minimum value in the similarity determination result sequence, then the second node board region identified in the i-th frame image and the k-th node board region identified in the (i+1)-th frame image are the same node board region, and they are spatiotemporally correlated in consecutive frame images.

[0095] Step 1023: Based on the spatiotemporal correlation of the same node board region in consecutive frame images, obtain a continuous video sequence of the same node board region.

[0096] Step 1023 includes:

[0097] If the same node board region establishes a spatiotemporal association in a consecutive first preset number of frame images, then it is determined that the same node board region appears in the video images following the first preset number of frame images, and the frame number of the first frame image in the first preset number of frame images is determined as the starting frame number. Figure 4 As shown, the first preset quantity can be set to 4, and the second preset quantity can be set to 10.

[0098] After determining the starting frame number, if the same node board region cannot establish a spatiotemporal association in a consecutive second preset number of frames, then it is determined that the same node board region disappears in the video images following the second preset number of frames, and the frame number of the first frame in the second preset number of frames is determined as the ending frame number; the continuous video sequence of the same node board region is from the starting frame number to the ending frame number, such as... Figure 4 In the example, the three node board areas are arranged from the starting frame number circled in the circle to the ending frame number boxed in the box.

[0099] Step 1024: Based on the continuous video sequence of the same node board region, obtain the segmented video image.

[0100] Images that establish spatiotemporal correlations with the same node board target are combined to form video images with the same node board region, and the video images are segmented.

[0101] Step 103: Extract bolt shadow area features from the segmented video images.

[0102] like Figure 5 As shown, under preset lighting conditions, the bolt will cast a shadow. The shadow area of ​​a loose bolt will be larger than that of a normal bolt, such as... Figure 6 The normal bolt shown, and Figure 7 The loose bolt is shown. Using this lighting condition, the area of ​​the bolt's shadow is used as an input feature value to assess bolt looseness.

[0103] Considering that the Unet network's skip connection stage cannot effectively extract feature information of irregularly shaped and scaled shadow regions, resulting in inaccurate shadow area calculation, this invention proposes an improved Unet network structure, such as... Figure 8 As shown, the improved Unet network structure includes a Resnet50 network, which mainly performs Convolution (Conv), Regularization (BN), and ReLU activation function calculations on the input blocks (Blocks 1-4). A Multi-Scale Serial Enhancement (MSSE) module is added in series during the Unet skip connection stage, and then the signal reaches the Decoder Block, which finally extracts the shadow region feature information of the bolts in the segmented video image.

[0104] Step 103 includes:

[0105] The segmented video image is input into the improved Unet network structure; the improved Unet network structure includes a multi-scale concatenated enhancement module; the multi-scale concatenated enhancement module is used to extract spatial information from the low-level feature maps of the segmented video image input into the improved Unet network structure.

[0106] For example, in order to more effectively mine low-level feature map information of bolt shadow regions in Unet network skip connections, this embodiment proposes to use a parallel fusion structure of four convolutional kernels (sizes of 1×1, 3×3, 7×7, and 11×11, respectively) to effectively perceive the boundary details and overall contour features of bolt shadow regions. Using large convolutional kernels can expand the effective receptive field and obtain more global information, which is conducive to capturing more complete shadow information, as shown in formula (6):

[0107]

[0108] Among them, l n' L represents the effective receptive field of the n'th layer. n'-1 k represents the effective receptive field of the (n'-1)th layer. n' s represents the kernel size of the n'th layer. n'-1Let represent the convolution stride of the (n'-1)th layer. From the above equation, it can be seen that the effective receptive field improvement brought by large kernel convolution is more significant than increasing the number of network layers.

[0109] However, the bolt shadow feature information extracted by the above multi-kernel convolutional structure may have a certain degree of redundancy, which increases the computational burden on the calculation of bolt shadow area. In order to ensure the computational efficiency of the network structure and retain the spatial information of bolt shadows at different scales, a multi-diffraction dilated convolutional structure is connected in series after the multi-kernel convolutional structure. The multi-diffraction dilated convolutional structure is composed of dilated convolutional layers with dilation rates of 1, 3, 7, and 11 combined in parallel. The specific calculation process of the result y of the multi-diffraction dilated convolutional structure is shown in formula (7):

[0110] y = Conv((y1,y2,y3,y4) (7)

[0111] Where y1, y2, y3, and y4 are the results of dilated convolutions with different dilation rates applied to the input feature map, and Conv is the convolution operation.

[0112] For ease of description, in this embodiment, the cascaded structure of the multi-kernel convolutional structure followed by the dilated multi-rate dilated convolutional structure is named the Multi-Scale Cascaded Enhancement (MSSE) module, and it is added to the skip connections of the Unet network. The improved Unet network structure outputs bolt shadow area features, which can better extract spatial information from low-level feature maps, such as... Figure 5 As shown, bolt0.89, bolt0.90, etc. represent the accuracy of identifying and extracting bolt shadow area features, indicating that the improved Unet network structure in this embodiment has high accuracy in extracting bolt shadow area features.

[0113] Step 104: Based on the bolt shadow area features, perform bolt shadow association operations on adjacent frame images of the same node plate region sequentially to obtain the target shadow feature time series of a single bolt.

[0114] Within a single frame, it's impossible to guarantee a significant difference between the shadow area of ​​a loose bolt and that of a normal bolt. Furthermore, in video images, the scale change of the bolt target between adjacent frames is relatively small, exhibiting a certain degree of spatiotemporal correlation. Therefore, this embodiment uses an improved template-matching multi-frame image feature association method to obtain a time series of shadow features for a single bolt target, such as... Figure 9 As shown.

[0115] According to the principle of signal convolution, the template matching process is equivalent to convolving two adjacent images in the spatial domain. The calculation process is equivalent to multiplying the spectra of two adjacent images in the frequency domain. By using the phase correlation method, the content alignment operation of adjacent frame images is realized, and the position association of the same bolt target is completed.

[0116] Step 104 includes:

[0117] For two consecutive frame images, A and B, located in the same node board region, the spectra of the two consecutive frame images are calculated using Fast Fourier Transform (FTT). Image A is used as the template image, and image B is used as the test image. The spectrum of image A is denoted as F(A), and the spectrum of image B is denoted as F(B).

[0118] Based on the spectra of two consecutive frames, the cross-power spectrum H(w) of the two consecutive frames is obtained by multiplication.

[0119] H(w)=F(A)F′(B) (8)

[0120] Where F(A) represents the Fourier transform result of image A, F′(B) represents the conjugate value of the Fourier transform result F(B) of image B, and w is the frequency.

[0121] Calculate the inverse Fourier transform h(u,v) of the cross-power spectrum H(w) of two consecutive frames, and shift the zero-frequency component in the inverse Fourier transform of the cross-power spectrum of the two consecutive frames to the center of the result. U is the spatial frequency along the horizontal direction of the image; v is the spatial frequency along the vertical direction of the image.

[0122] For example, the zero-frequency component in h(u,v) is moved to the result center h(u′,v′). The expression for the result center h(u′,v′) is:

[0123]

[0124] Wherein, IFFT(H(w)) is the inverse Fourier transform of the cross-power spectrum of two consecutive frames; M is the width of h(u,v), and N is the height of the calculated result h(u,v).

[0125] Based on the result center h(u′,v′), the content correlation coefficient matrix C of two consecutive frame images is obtained, and the coordinates (a,b) of the position of the maximum value in the matrix are extracted.

[0126] The content correlation coefficient matrix C and the position coordinates (a, b) of two consecutive frames are expressed as follows:

[0127]

[0128] The content correlation coefficient matrix C between two adjacent images can be obtained by subsequently extracting the real part of h(u′,v′), and Re(h(u′,v′)) represents the extraction of the real part of h(u′,v′).

[0129] Translate the center of image B to position coordinates (a, b) and associate the bolt shadows of two consecutive frames, image A and image B, in the same node plate region.

[0130] For example, in such Figure 9 In the image of the i-th frame (equivalent to image A), there are 3 node plates. The node plate regions identified in the i-th frame are labeled j=0, j=1, and j=2, which are of 3 types. In the i+1-th frame (equivalent to image B), the node plate regions k=0 and k=1 are associated with the node plate regions j=0 and j=1. Then, for the same node plate region j=0 and k=0, the bolt shadow association operation of adjacent frame images is performed.

[0131] Following the above process of associating bolt shadows in two consecutive frames, A and B, for the same node plate region, the bolt shadow association operation is performed pairwise on the image sequence for the same node plate region, such as... Figure 10 As shown, the time series of target shadow features of a single bolt in all frame images of the same node plate region is obtained, such as... Figure 11 As shown in the figure. Where n is the number of frames of the image in the same node board region.

[0132] In this embodiment, considering that the template matching process is equivalent to convolution of two adjacent images in the spatial domain, and the calculation process is equivalent to multiplying the spectra of two adjacent images in the frequency domain, a method using phase correlation in the spectrum is proposed to accelerate the template matching process. Furthermore, phase correlation calculation is performed only on the bolt shadow segmentation results of two adjacent images, which further improves the calculation efficiency.

[0133] Step 105: Identify loose bolts based on the target shadow feature time series of a single bolt.

[0134] Within the same node plate area of ​​a long-span steel bridge, there are certain differences in the shadowed areas of loose bolts and normal bolts, but this difference exhibits considerable randomness. For example, in some image sequences, such as... Figure 6 In the area marked by the dashed rectangle, the shadow areas of loose bolts and normal bolts are similar, affecting the identification of whether a bolt has become loose. Therefore, this embodiment normalizes the time series of target shadow features for each bolt within each node region of each frame image and performs K-means classification analysis.

[0135] Step 105 includes:

[0136] Based on the target shadow feature time series of a single bolt, the target shadow feature time series of each bolt is normalized, that is, the time series of bolt shadow area within the same node plate region is normalized to obtain the normalized value of the target shadow feature time series of all bolts within the same node plate region, such as... Figure 12 As shown.

[0137] For example, in order to reduce the interference of bolt shadow area changes caused by changes in drone shooting distance, this embodiment normalizes the shadow area of ​​each bolt in each node plate area of ​​each frame image using formula (11) to obtain the normalized shadow area value of each bolt.

[0138]

[0139] in, x represents the number of bolts within region j of the same node plate. t y represents the shaded area of ​​the t-th bolt within the node plate region. t This represents the normalized area value of the t-th bolt.

[0140] K-means classification is performed on the normalized time series values ​​of the target shadow features of all bolts within the same node plate area to obtain the classification results; based on the classification results, the loose bolts are identified.

[0141] For example, K-means classification is achieved by performing mean normalization on the time series of shadow features of each bolt within the same node plate region j, as shown in formula (12).

[0142]

[0143] In the above formula, t is the bolt number within the current node plate region j, n is the total number of frames in the continuous video sequence containing the node plate region j, and y t (i) represents the normalized value of the shadow area of ​​the bolt in the i-th frame image, Y. t This represents the final characteristic value of the t-th bolt.

[0144] Finally, K-means clustering analysis was performed on the final feature values ​​of each bolt within the same node plate area. In this invention, there are only two categories: normal bolts and loose bolts, therefore the number of categories K is 2. Considering that bolt loosening in steel bridges is a low-probability event, the bolt with the fewer bolts in the two categories is considered a loose bolt, such as... Figure 13 As shown, the 7th, 14th, and 20th bolts are loose bolts.

[0145] The dense bolt loosening video detection method provided in this application embodiment accurately segments the video images of bolts through a continuous video sequence of the same node plate area, which enables focusing on the continuous video sequence of the same node plate area, avoids unnecessary image processing, and improves detection efficiency.

[0146] Building upon this foundation, bolt shadows are used as visual features indicating bolt loosening, and an improved Unet algorithm is proposed to calculate these features, accurately reflecting the relative positional relationship between the bolts and the background. Bolt shadow association operations are performed sequentially on adjacent frames of image sequences representing the same node plate region, yielding rich bolt loosening-related features. Furthermore, a signal phase correlation algorithm is used to perform spatiotemporal correlation calculations between bolt loosening features in video clips, aiding in the accurate determination of bolt state changes and enabling precise detection of dense bolt loosening in long-span steel bridges.

[0147] It should be understood that the sequence number of each step does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0148] See Figure 14 This embodiment provides a video detection device for loose bolts, which implements the video detection method for loose bolts as described in the above embodiment. The video detection device for loose bolts includes an acquisition module 201, an image segmentation module 202, a feature extraction module 203, a feature association module 204, and a recognition module 205.

[0149] Acquisition module 201 is used to acquire video images of the bolt;

[0150] Image segmentation module 202 is used to segment bolt-based video images according to a continuous video sequence of the same node plate region to obtain segmented video images;

[0151] Feature extraction module 203 is used to extract bolt shadow area features from the segmented video image;

[0152] Feature association module 204 is used to perform bolt shadow association operation on adjacent frame images of the same node plate region based on bolt shadow area features, to obtain target shadow feature time series of a single bolt.

[0153] The identification module 205 is used to identify loose bolts based on the time series of target shadow features of a single bolt.

[0154] For example, the image segmentation module 202 is specifically used for:

[0155] Based on video images of bolts, obtain feature descriptions of the same node plate;

[0156] Based on the feature description of the same node board, the spatiotemporal relationship of the same node board region in consecutive frame images is established.

[0157] Based on the spatiotemporal correlation of the same node board region in consecutive frame images, a continuous video sequence of the same node board region is obtained;

[0158] Segmented video images are obtained from continuous video sequences based on the same node board region.

[0159] For example, the characteristic description of the same node plate includes the same node plate area type, the coordinate position of the same node plate area, and the number of bolts in the same node plate area.

[0160] In the image segmentation module 202, based on the feature description of the same node board, the spatiotemporal association of the same node board region in consecutive frame images is established, including:

[0161] Based on the node plate region type of two consecutive frame images, the distance between the node plate regions of two consecutive frame images, and the number of bolts in the node plate regions of two consecutive frame images, a similarity determination result sequence is determined; the distance between the node plate regions of two consecutive frame images is determined by the coordinate positions of two adjacent node plate regions.

[0162] Extract the minimum value from the similarity determination result sequence;

[0163] If the minimum value in the similarity determination result sequence is not greater than the preset threshold, then the two node board regions corresponding to the minimum value in the similarity determination result sequence establish a spatiotemporal association of the same node board region in consecutive frame images.

[0164] For example, the determination function F(X) for the similarity determination result sequence i,j ,X i+1,k The expression for ) is:

[0165]

[0166] in, A function to determine the type of two adjacent node plate regions; The weighted calculation function for the distance between two adjacent node plate regions; This is a function to determine the number of bolts in two adjacent node plate regions. Let j be the type of the node board region identified in the i-th frame image. Let J be the coordinates of the j-th node plate region identified in the i-th frame image; The number of bolts identified in the j-th node plate region in the i-th frame image; Let k be the type of the k-th node board region identified in the (i+1)-th frame image. The coordinates of the kth node plate region identified in the (i+1)th frame image; The number of bolts in the region of the kth node plate identified in the (i+1)th frame image.

[0167] For example, based on the spatiotemporal correlation of the same node board region in consecutive frame images, a continuous video sequence of the same node board region is obtained, including:

[0168] If the same node board region establishes a spatiotemporal association in a consecutive first preset number of frame images, then it is determined that the same node board region appears in the video images after the consecutive first preset number of frame images, and the frame number of the first frame image in the consecutive first preset number of frame images is determined as the starting frame number.

[0169] After determining the starting frame number, if the same node board region cannot establish a spatiotemporal association in the second consecutive preset number of frame images, it is determined that the same node board region disappears in the video images after the second consecutive preset number of frame images, and the frame number of the last frame image in the second consecutive preset number of frame images is determined as the termination frame number; the continuous video sequence of the same node board region is the starting frame number to the termination frame number.

[0170] For example, feature extraction module 203 is specifically used for:

[0171] The segmented video images are input into the improved Unet network structure; the improved Unet network structure includes a multi-scale concatenated enhancement module; the multi-scale concatenated enhancement module is used to extract spatial information from the low-level feature maps of the segmented video images input into the improved Unet network structure;

[0172] Improved Unet network structure outputs bolt shadow area characteristics.

[0173] For example, feature association module 204 is specifically used for:

[0174] For two consecutive frame images A and B in the same node board region, the spectrum of the two consecutive frame images is calculated using Fast Fourier Transform; image A is used as the template image and image B is used as the test image.

[0175] Based on the spectra of two consecutive frames, the cross-power spectrum of the two consecutive frames is obtained by multiplication.

[0176] Calculate the inverse Fourier transform of the cross-power spectrum of two consecutive frames, and move the zero-frequency component in the inverse Fourier transform of the cross-power spectrum of the two consecutive frames to the center of the result.

[0177] Based on the result center, the content correlation coefficient matrix of two consecutive frame images is obtained, and the coordinates of the position of the maximum value in the matrix are extracted.

[0178] Translate the center of image B to the position coordinates and associate the bolt shadows of two consecutive frames, image A and image B, in the same node plate region;

[0179] Following the above process of bolt shadow association for two consecutive frame images A and B of the same node plate region, the bolt shadow association operation of adjacent frame images is performed sequentially on the image sequence of the same node plate region to obtain the target shadow feature time series of all individual bolts in the same node plate region.

[0180] For example, the identification module 205 is specifically used for:

[0181] Based on the target shadow feature time series of a single bolt, the target shadow feature time series of each bolt is normalized to obtain the normalized value of the target shadow feature time series of all bolts in the same node plate area.

[0182] K-means classification was performed on the normalized time series values ​​of the target shadow features of all bolts within the same node plate area to obtain the classification results;

[0183] Based on the classification results, the loose bolts were identified.

[0184] It should be noted that the information interaction and execution process between the above-mentioned devices are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0185] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0186] This application also provides a terminal device, see [link to relevant documentation] Figure 15 The terminal device 400 may include at least one processor 410 and a memory 420. The memory 420 stores a computer program 421 that can run on the at least one processor 410. When the processor 410 executes the computer program 421, it implements the steps in any of the above-described method embodiments, for example... Figure 2 Steps 101 to 105 in the illustrated embodiment, and for example Figure 15Modules 201 to 205 in the illustrated embodiment.

[0187] For example, computer program 421 may be divided into one or more modules / units, one or more of which are stored in memory 420 and executed by processor 410 to complete this application. One or more modules / units may be a series of computer program segments capable of performing a specific function, which describe the execution process of computer program 421 in terminal device 400.

[0188] Those skilled in the art will understand that Figure 15 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, buses, etc.

[0189] The processor 410 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0190] The memory 420 can be an internal storage unit of the terminal device or an external storage device, such as a plug-in hard drive, a smart media card (SMC), a secure digital card (SD), or a flash card. The memory 420 is used to store computer programs and other programs and data required by the terminal device. The memory 420 can also be used to temporarily store data that has been output or will be output.

[0191] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0192] The video detection method for loose bolts provided in this application can be applied to terminal devices such as computers, tablets, laptops, netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of terminal device.

[0193] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the various embodiments of the above-described dense bolt loosening video detection method.

[0194] This application provides a computer program product that, when run on a mobile terminal device, enables the mobile terminal device to implement the steps in the various embodiments of the above-described dense bolt loosening video detection method.

[0195] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0196] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0197] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0198] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0199] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0200] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A video detection method for loose bolts in dense clusters, characterized in that, include: Capture video images of the bolts; Based on the video image of the bolt, the video image is segmented according to a continuous video sequence of the same node plate area to obtain the segmented video image; Extract bolt shadow area features from the segmented video images; Based on the bolt shadow area characteristics, the bolt shadow association operation is performed sequentially on adjacent frame images of the same node plate region to obtain the target shadow feature time series of a single bolt. This includes: calculating the spectra of two consecutive frame images A and B for the same node plate region using Fast Fourier Transform; image A is used as the template image, and image B is used as the test image; based on the spectra of the two consecutive frame images, a multiplication operation is used to obtain the cross-power spectrum of the two consecutive frame images; the inverse Fourier transform of the cross-power spectrum of the two consecutive frame images is calculated, and the cross-power spectrum of the two consecutive frame images is... The zero-frequency component in the inverse Fourier transform of the spectrum is shifted to the result center; based on the result center, the content correlation coefficient matrix of two consecutive frame images is obtained, and the position coordinates of the maximum value in the matrix are extracted; the center of image B is shifted to the position coordinates, and the bolt shadows of two consecutive frame images A and B in the same node plate region are associated; following the above process of associating the bolt shadows of two consecutive frame images A and B in the same node plate region, the bolt shadow association operation of adjacent frame images is performed sequentially on the image sequence of the same node plate region to obtain the target shadow feature time series of all individual bolts in the same node plate region; Based on the time series of the target shadow features of the individual bolt, the loose bolt is identified; The results center The expression is: ； in, M is the inverse Fourier transform of the cross-power spectra of two consecutive frames; M is The width, N is the calculated result. high, u Spatial frequency along the horizontal direction of the image; v The spatial frequency is along the vertical direction of the image.

2. The video detection method for loose bolts as described in claim 1, characterized in that, The video image based on the bolt is segmented according to a continuous video sequence of the same node plate region to obtain segmented video images, including: Based on the video images of the bolts, obtain feature descriptions of the same node plate; Based on the feature description of the same node board, the spatiotemporal correlation of the same node board region in consecutive frame images is established. Based on the spatiotemporal correlation of the same node board region in consecutive frame images, a continuous video sequence of the same node board region is obtained; Based on the continuous video sequence of the same node board region, segmented video images are obtained.

3. The video detection method for loose bolts as described in claim 2, characterized in that, The characteristic description of the same node plate includes the same node plate area type, the coordinate position of the same node plate area, and the number of bolts in the same node plate area; The step of establishing the spatiotemporal correlation of the same node board region in consecutive frame images based on the feature description of the same node board includes: Based on the node plate region type of two consecutive frame images, the distance between the node plate regions of the two consecutive frame images, and the number of bolts in the node plate regions of the two consecutive frame images, a similarity determination result sequence is determined; the distance between the node plate regions of the two consecutive frame images is determined by the coordinate positions of two adjacent node plate regions. Extract the minimum value from the sequence of similarity determination results; If the minimum value in the similarity determination result sequence is not greater than a preset threshold, then the two node board regions corresponding to the minimum value in the similarity determination result sequence establish a spatiotemporal association of the same node board region in consecutive frame images.

4. The video detection method for loose bolts as described in claim 3, characterized in that, The determination function of the similarity determination result sequence The expression is: in, A function to determine the type of two adjacent node plate regions; The weighted calculation function for the distance between two adjacent node plate regions; This is a function to determine the number of bolts in two adjacent node plate regions. For the first i The first frame image identified j Each node board region type For the first i The first frame image identified j The coordinates of each node plate region; For the first i The first frame image identified j The number of bolts in each node plate area; For the first i The first one identified in the +1 frame image k Each node board region type For the first i The first one identified in the +1 frame image k The coordinates of each node plate region; For the first i The first one identified in the +1 frame image k The number of bolts in each node plate area.

5. The video detection method for loose bolts as described in claim 2, characterized in that, The method of obtaining a continuous video sequence of the same node board region based on the spatiotemporal correlation of the same node board region in consecutive frame images includes: If the same node board region establishes a spatiotemporal association in a consecutive first preset number of frame images, then it is determined that the same node board region appears in the video images after the consecutive first preset number of frame images, and the frame number of the first frame image in the consecutive first preset number of frame images is determined as the starting frame number. After determining the starting frame number, if the same node board region cannot establish a spatiotemporal association in a consecutive second preset number of frame images, it is determined that the same node board region disappears in the video images after the consecutive second preset number of frame images, and the frame number of the last frame image in the consecutive second preset number of frame images is determined as the termination frame number; the continuous video sequence of the same node board region is from the starting frame number to the termination frame number.

6. The video detection method for loose bolts as described in claim 1, characterized in that, Extracting bolt shadow area features from the segmented video images includes: The segmented video image is input into an improved Unet network structure; the improved Unet network structure includes a multi-scale concatenated enhancement module; the multi-scale concatenated enhancement module is used to extract spatial information from the low-level feature map of the segmented video image input into the improved Unet network structure; The improved Unet network structure outputs the bolt shadow area feature.

7. The video detection method for loose bolts as described in claim 1, characterized in that, The determination of loose bolts based on the target shadow feature time series of the single bolt includes: Based on the target shadow feature time series of the single bolt, the target shadow feature time series of each bolt is normalized to obtain the normalized value of the target shadow feature time series of all bolts in the same node plate area. K-means classification is performed on the normalized time series values ​​of the target shadow features of all bolts in the same node plate area to obtain the classification results; Based on the classification results, the loose bolts were identified.

8. A terminal device, comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the video detection method for loose dense bolts as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the video detection method for loose dense bolts as described in any one of claims 1 to 7.

Images