Bridge expansion joint damage detecting and positioning method based on multi-modal signal fusion

Abstract

The invention discloses a bridge expansion joint damage detection and positioning method based on multi-modal signal fusion, and the method comprises the following steps: S1, building a detection platform, and collecting the multi-source data of a vibration signal, a sound signal and a video signal; s2, the vibration signals, the sound signals and the video signals are preprocessed respectively, and respective features including time domain features, frequency domain features and time-frequency domain features are extracted; s3, carrying out sound-vibration fusion on the vibration signal and the sound signal, and carrying out audio-video fusion on the sound signal and the video signal to realize integration of multi-modal features; and S4, outputting a damage identification and positioning result, and providing decision support for maintenance and management of the bridge expansion joint. According to the method, through multi-modal fusion signal analysis, the health condition of the bridge expansion device is monitored in real time, and damage is found in time, so that potential safety risks are prevented, safe operation of the bridge is ensured, and a new technical means is provided for health monitoring and maintenance of the bridge.

Description

Technical Field

[0001] The present invention relates to the technical field of structural health monitoring and nondestructive testing of bridge expansion devices, and in particular to a method for detecting and locating damage of bridge expansion joints based on multimodal signal fusion. Background Art

[0002] In order to meet the requirements of bridge deck deformation, bridge expansion devices are usually installed at the two ends of the bridge beams, between the beam ends and the abutments, or at the hinges of the bridge. Its core function is to adjust the displacement and deformation caused by temperature fluctuations, vehicle loads, wind loads, etc., and to help release the generated stress. Due to the uncertainty of factors such as the service environment and the loads, the damage problem of bridge expansion devices is very prominent. Poor design, problems with the device itself, or failure to promptly and effectively repair damage during use will lead to a series of problems, reduce the safe service life of the bridge, and even threaten the overall safety of the bridge structure.

[0003] The existing bridge expansion device damage detection and location system mainly relies on manual inspection. According to the professional knowledge and experience of the inspectors, the expansion device turntable is evaluated by visual inspection, hammer tapping and other methods. Although this method is simple and effective, it is inherently subjective and therefore has certain limitations. As a more objective and accurate technical means, non-destructive testing technology has the functions of monitoring, evaluation, and early warning for the relevant parameters of structural changes. In recent decades, sound signals, vibration signals, and video signals have been increasingly used to detect damage or abnormal conditions of infrastructure in use. The multimodal signals generated by vehicle driving on the bridge expansion device contain information about the operating status of the bridge expansion device and have the potential to detect damage to the bridge expansion device.

[0004] With the development of deep learning technology, its application in the field of fault detection is becoming more and more extensive. Deep learning models can automatically learn and extract complex features in data through multi-layer neural network structures, which makes them perform well when processing nonlinear and high-dimensional data. These models can not only improve the accuracy of fault detection, but also shorten the time of fault diagnosis, providing strong technical support for industrial automation and intelligent manufacturing. Summary of the invention

[0005] Technical problem: The bridge expansion joint damage detection and location method based on multimodal signal fusion proposed in the present invention effectively reduces the subjective bias and labor intensity of manual inspection by collecting the sound signal, vibration signal and video signal generated when the vehicle passes through the bridge expansion device, and conducts in-depth analysis on them. This method can monitor the health status of the bridge expansion device in real time through multimodal fusion signal analysis, detect damage in time, thereby preventing potential safety risks and ensuring the safe operation of the bridge. The proposal of this method is an important supplement and improvement to the existing bridge expansion device damage detection technology, and provides a new technical means for bridge health monitoring and maintenance.

[0006] Technical solution: The present invention provides a bridge expansion joint damage detection and positioning method based on multimodal signal fusion, which comprises the following steps:

[0007] Step S1, building a detection platform to collect multi-source data of vibration signals, sound signals and video signals;

[0008] Step S2, preprocessing the vibration signal, the sound signal and the video signal respectively, and extracting their respective features, including time domain, frequency domain and time-frequency domain features;

[0009] Step S3, multi-modal fusion of the vibration signal, the sound signal and the video signal; by constructing a damage identification model, the fused multi-modal features are analyzed to achieve identification and location of the bridge expansion joint damage;

[0010] Step S4, output damage identification and location results to provide decision support for the maintenance and management of bridge expansion joints.

[0011] The detection platform of step S1 includes a vibration sensor array, a microphone array, a camera and a signal acquisition device; the vibration signal, the sound signal and the video signal are derived from the sound signal, the vibration signal and the video signal generated by the vehicle passing through the expansion joint, and the acquisition area is the lanes on both sides close to the expansion joint.

[0012] The feature extraction of the sound signal in step S2 comprises the following steps:

[0013] Step S211, pre-enhancement processing is performed on the collected sound signal;

[0014] S p (n) = S (n) - α S (n-1) (1)

[0015] S p (n) represents the pre-emphasized sound signal, α represents the pre-emphasis coefficient; S(n) represents the original sound signal, n represents the sampling point,

[0016] Step S212, framing and windowing the pre-enhanced sound signal;

[0017] The pre-enhanced sound signal is divided into short frames, and N sampling points are first grouped into an observation frame; N is usually set to 256;

[0018] In order to avoid too large changes between two adjacent frames, there is an overlapping area between two adjacent frames. This overlapping area contains N / 2 sampling points.

[0019] After the signal is framed, each frame is substituted into the window function for windowing:

[0020] S w (n) = S P (n)×w(n) (2)

[0021] S w (n) represents the windowed signal, and w(n) represents the windowing function;

[0022] Apply a Hamming window to reduce spectral leakage:

[0023]

[0024] Step S213, applying short-time Fourier transform (STFT) to calculate the power spectrum;

[0025]

[0026] X t (k) represents the frequency response of the signal at frequency k in the tth frame, P t (k) represents the power spectrum of the signal at frequency k in the tth frame;

[0027] Step S214, using the Mel filter bank to obtain the Mel power spectrum and Mel cepstral coefficients MFCC;

[0028] Each frequency value f corresponds to a Mel value Mel(f):

[0029]

[0030] The power spectrum is passed through a filter bank with M triangular filters. The frequency response of the triangular filter at frequency k is defined as:

[0031]

[0032] The logarithmic energy of each filter bank output is calculated as:

[0033]

[0034] The MFCC coefficients are obtained by discrete cosine transform DCT:

[0035]

[0036] L represents the MFCC coefficient order.

[0037] The step S2 of extracting the characteristic changes of the vibration signal comprises the following steps:

[0038] Step 221, perform discrete S transform on the vibration signal U(t) to obtain S transform coefficients:

[0039]

[0040] s U,m (t,f) represents the S transform coefficient, ψ m (t,f) represents the time-frequency window function, lΔt represents the discrete time point, and Δt represents the sampling interval;

[0041] Step 222, sinusoidal time-frequency window function for:

[0042]

[0043] w(f) represents the window shape function, a, b and c represent the parameters that control the window shape, and f s Indicates the signal sampling frequency;

[0044] Step 223, Hermite time-frequency window function for:

[0045]

[0046] Step 224, using the improved multi-window spectrum S transform method to calculate the time-frequency evolution spectrum matrix of the signal:

[0047] *, T represent the adjoint matrix and the transpose of the matrix respectively represents the video evolution spectrum matrix, and M(t,f) represents the number of windows that change with t and f:

[0048] Step 224, determine the optimal M(t,f) for each frequency point through an iterative process:

[0049] Use the first window to perform S-transform on all time and frequency points to generate an initial evolution spectrum estimate;

[0050] For each frequency f, the sum of the spectrum values ​​at all time points is used to estimate the total spectrum S sum (f)) Fit a quadratic function in the frequency interval [fW,f+W], where W represents the resolution bandwidth;

[0051] Calculate S from the fitted quadratic function sum (f) the first-order derivative y′ and the second-order derivative y″ with respect to f;

[0052] Use the derivative obtained from the fitting to calculate the optimal M. The specific calculation formula is:

[0053]

[0054] L(f) represents the number of time points at frequency f;

[0055] Smooth M using forward and backward filtering techniques, and ensure that the number of windows between consecutive frequencies does not vary by more than 1;

[0056] The above steps were repeated until the mean relative difference (MRD) in the number of windows between consecutive iterations was less than a predefined value of 10%.

[0057] Step 225, extracting features including energy distribution, frequency components, etc. from the evolution spectrum.

[0058] The feature change extraction of the video signal in step S2 comprises the following steps:

[0059] Step S231, pre-processing the video, adjusting the video frame rate to a speed suitable for processing, adjusting the video resolution to a size suitable for processing, and using a filter to remove image noise;

[0060] Step S232, performing frame extraction on the preprocessed video signal, and extracting key frames containing vehicle information and expansion joints from the video;

[0061] Step S233, extracting basic visual features of the expansion joint from the key frame, including its width, length, and color, and identifying whether the expansion joint has signs of damage, including cracks, deformation, or material shedding, and locating the damage location;

[0062] Step S234, performing vehicle recognition detection on the key frame, using background subtraction technology to identify the foreground vehicle, and further segmenting the vehicle by color, shape and texture features;

[0063] Step S235, analyzing the contact point between the vehicle and the expansion joint, and detecting the state change of the expansion joint after the vehicle passes through;

[0064] Step S235 , analyzing the behavior of the vehicle when passing through the expansion joint, including the speed, path, and any abnormal behavior, to identify vehicle behavior patterns that may cause damage to the expansion joint.

[0065] The step S3 of constructing a damage identification model for the vibration signal and the sound signal comprises the following steps:

[0066] Step S311, mapping the features of different modes to a common feature space, aligning the features of different modes to ensure that they correspond to the same expansion joint state in time;

[0067] Step S312: weight the features according to the contribution of the vibration signal and the sound signal to damage identification; add and fuse the sound feature graph F1 and the vibration feature graph F2 element by element according to the weights to form an acoustic-vibration feature graph F:

[0068]

[0069] represents element-by-element addition, and θ represents the signal feature weight;

[0070] Step S313: Use the acoustic vibration feature map to train a deep learning network to identify damage.

[0071] The step S3 of locating the damaged position of the bridge expansion joint according to the sound signal comprises the following steps:

[0072] According to the law of cosines:

[0073]

[0074] q i represents the microphone coordinates (u i ,v i ,w i ), Y represents the coordinates of the sound source to be located (u Y ,v Y ,w Y ), R i Indicates microphone q i Distance to the origin, R Y Indicates the distance from the sound source Y to the origin, d i Indicates the sound source Y to q0 and q i The distance difference:

[0075] d i =Δt·c (19)

[0076] Δt represents the distance from the sound source Y to q0 and q i The time difference is , c represents the speed of sound,

[0077] Formula (16) can be further obtained:

[0078]

[0079] Due to d i It is obtained by estimating the time delay. There will be a deviation ε between formula (16) and the actual value:

[0080]

[0081] When the deviation is minimum, the position of the sound source Y can be determined:

[0082]

[0083] Convert formula (18) into matrix form:

[0084]

[0085] in,

[0086] Simplifying further:

[0087] ε=Gμ-β (24)

[0088] Where G = [QD],

[0089] The least squares solution can be expressed as:

[0090] μ=(G T G) -1 G T β (25)

[0091] Define the projection matrix along G to D:

[0092]

[0093] The projection matrix along D to G is:

[0094]

[0095] According to the properties of the projection matrix:

[0096] G=P D [Q 0] (28)

[0097] Substitute into formula (21):

[0098] ε=P D QY-β (29)

[0099] Final sound source Y positioning result:

[0100] Y=(Q T P D Q) -1 Q T P D b (30).

[0101] The step S3 of locating the damaged position of the bridge expansion joint according to the audio and video signals comprises the following steps:

[0102] Step S321, combining visual and auditory features and enhancing key features through an attention mechanism;

[0103] Step S322, selecting semantically consistent segments for feature fusion;

[0104] Step S323, using the fused features to determine the specific location of the injury;

[0105] Step S324, judging the consistency between the audio and video fusion positioning result and the sound signal sound source positioning result, and correcting the single-mode positioning result if they are inconsistent;

[0106] Step S325, the consistent positioning results are integrated at the decision layer and the positioning results are output.

[0107] The step S4 outputs the damage identification and location results including:

[0108] Step S401, outputting expansion joint damage identification results according to the trained deep learning network;

[0109] Step S402, outputting the precise location where the damage occurs according to the positioning consistency judgment result;

[0110] Step S403, vehicle identification and monitoring are performed according to the characteristics of the video signal.

[0111] The damage maintenance and management includes performing preventive maintenance based on damage identification results and positioning results.

[0112] Beneficial effects: Through the above technical solution, the present invention can effectively detect and locate the damage of the bridge expansion device, improving the efficiency and accuracy of bridge maintenance. The method is particularly suitable for monitoring the structural damage of the expansion device on each lane, which helps to timely discover and repair potential safety hazards and ensure the safe operation of the bridge. BRIEF DESCRIPTION OF THE DRAWINGS

[0113] Figure 1 This is a flow chart of a bridge expansion joint damage detection and positioning system based on acoustic and vibration signal fusion proposed by the present invention;

[0114] Figure 2 A schematic diagram of acoustic-vibration signal feature fusion in a bridge expansion joint damage detection and positioning system based on acoustic-vibration signal fusion proposed by the present invention;

[0115] Figure 3 A schematic diagram of a method for locating the damage position of an expansion joint described in a bridge expansion joint damage detection and positioning system based on acoustic-vibration signal fusion proposed by the present invention;

[0116] Figure 4A schematic diagram of an audio and video signal fusion positioning method for a bridge expansion joint damage detection and positioning system based on acoustic and vibration signal fusion proposed by the present invention. DETAILED DESCRIPTION

[0117] A specific implementation of a bridge expansion joint damage detection and positioning method based on multimodal signal fusion of the present invention comprises the following steps:

[0118] Step S1, building a detection platform to collect multi-source data of vibration signals, sound signals and video signals;

[0119] Step S2, preprocessing the vibration signal, the sound signal and the video signal respectively, and extracting their respective features, including time domain, frequency domain and time-frequency domain features;

[0120] Step S3, multi-modal fusion of the vibration signal, the sound signal and the video signal; by constructing a damage identification model, the fused multi-modal features are analyzed to achieve identification and location of the bridge expansion joint damage;

[0121] Step S4, output damage identification and location results to provide decision support for the maintenance and management of bridge expansion joints.

[0122] The detection platform of step S1 includes a vibration sensor array, a microphone array, a camera and a signal acquisition device; the vibration signal, the sound signal and the video signal are derived from the sound signal, the vibration signal and the video signal generated by the vehicle passing through the expansion joint, and the acquisition area is the lanes on both sides close to the expansion joint.

[0123] The feature extraction of the sound signal in step S2 comprises the following steps:

[0124] Step S211, pre-enhancement processing is performed on the collected sound signal;

[0125] S p (n) = S (n) - α S (n-1) (1)

[0126] S p (n) represents the pre-enhanced sound signal, α represents the pre-enhanced coefficient, and α is selected as 0.97; S(n) represents the original sound signal, n represents the sampling point,

[0127] Step S212, framing and windowing the pre-enhanced sound signal;

[0128] The pre-enhanced sound signal is divided into short frames, and N sampling points are first grouped into an observation frame, where N is usually set to 256;

[0129] In order to avoid too large changes between two adjacent frames, there is an overlapping area between two adjacent frames. This overlapping area contains N / 2 sampling points.

[0130] After the signal is framed, each frame is substituted into the window function for windowing:

[0131] S w (n) = S P (n)×w(n) (2)

[0132] S w (n) represents the windowed signal, and w(n) represents the windowing function;

[0133] Apply a Hamming window to reduce spectral leakage:

[0134]

[0135] Step S213, applying short-time Fourier transform (STFT) to calculate the power spectrum;

[0136]

[0137] X t (k) represents the frequency response of the signal at frequency k in the tth frame, P t (k) represents the power spectrum of the signal at frequency k in the tth frame;

[0138] Step S214, using the Mel filter bank to obtain the Mel power spectrum and Mel cepstral coefficients MFCC;

[0139] Each frequency value f corresponds to a Mel value Mel(f):

[0140]

[0141] The power spectrum is passed through a filter bank with M triangular filters, where M is usually set to 40. The frequency response of the triangular filter at frequency k is defined as:

[0142]

[0143] The logarithmic energy of each filter bank output is calculated as:

[0144]

[0145] The MFCC coefficients are obtained by discrete cosine transform DCT:

[0146]

[0147] L represents the MFCC coefficient order.

[0148] The step S2 of extracting the characteristic changes of the vibration signal comprises the following steps:

[0149] Step 221, perform discrete S transform on the vibration signal U(t) to obtain S transform coefficients:

[0150]

[0151] s U,m (t,f) represents the S transform coefficient, ψ m (t,f) represents the time-frequency window function, and the sine time-frequency window function or the Hermite time-frequency window function is selected. lΔt represents the discrete time point, and Δt represents the sampling interval;

[0152] Step 222, sinusoidal time-frequency window function for:

[0153]

[0154] w(f) represents the window shape function, a, b and c represent the parameters that control the window shape, and f s Indicates the signal sampling frequency;

[0155] Step 223, Hermite time-frequency window function for:

[0156]

[0157] Step 224, using the improved multi-window spectrum S transform method to calculate the time-frequency evolution spectrum matrix of the signal:

[0158] *, T represent the adjoint matrix and the transpose of the matrix respectively represents the video evolution spectrum matrix, and M(t,f) represents the number of windows that change with t and f:

[0159] Step 224, determine the optimal M(t,f) for each frequency point through an iterative process:

[0160] Use the first window to perform S-transform on all time and frequency points to generate an initial evolution spectrum estimate;

[0161] For each frequency f, the sum of the spectrum values ​​at all time points is used to estimate the total spectrum S sum (f)) Fit a quadratic function in the frequency interval [fW,f+W], where W represents the resolution bandwidth;

[0162] Calculate S from the fitted quadratic function sum (f) the first-order derivative y′ and the second-order derivative y″ with respect to f;

[0163] Use the derivative obtained from the fitting to calculate the optimal M. The specific calculation formula is:

[0164]

[0165] L(f) represents the number of time points at frequency f;

[0166] Smooth M using forward and backward filtering techniques, and ensure that the number of windows between consecutive frequencies does not vary by more than 1;

[0167] The above steps were repeated until the mean relative difference (MRD) in the number of windows between consecutive iterations was less than a predefined value of 10%.

[0168] Step 225, extracting features including energy distribution, frequency components, etc. from the evolution spectrum.

[0169] The feature change extraction of the video signal in step S2 comprises the following steps:

[0170] Step S231, pre-processing the video, adjusting the video frame rate to a speed suitable for processing, adjusting the video resolution to a size suitable for processing, and using a filter to remove image noise;

[0171] Select 30fps for frame rate and 480p (640x480 pixels) for resolution;

[0172] Step S232, performing frame extraction on the preprocessed video signal, and extracting key frames containing vehicle information and expansion joints from the video;

[0173] Step S233, extracting basic visual features of the expansion joint from the key frame, including its width, length, and color, and identifying whether the expansion joint has signs of damage, including cracks, deformation, or material shedding, and locating the damage location;

[0174] Step S234, performing vehicle recognition detection on the key frame, using background subtraction technology to identify the foreground vehicle, and further segmenting the vehicle by color, shape and texture features;

[0175] Step S235, analyzing the contact point between the vehicle and the expansion joint, and detecting the state change of the expansion joint after the vehicle passes through;

[0176] Step S235 , analyzing the behavior of the vehicle when passing through the expansion joint, including the speed, path, and any abnormal behavior, to identify vehicle behavior patterns that may cause damage to the expansion joint.

[0177] The step S3 of constructing a damage identification model for the vibration signal and the sound signal comprises the following steps:

[0178] Step S311, mapping the features of different modes to a common feature space, aligning the features of different modes to ensure that they correspond to the same expansion joint state in time;

[0179] Step S312: weight the features according to the contribution of the vibration signal and the sound signal to damage identification; add and fuse the sound feature graph F1 and the vibration feature graph F2 element by element according to the weights to form an acoustic-vibration feature graph F:

[0180]

[0181] represents element-by-element addition, θ represents the signal feature weight, and θ is selected as 0.7;

[0182] Step S313: Use the acoustic vibration feature map to train a deep learning network to identify damage.

[0183] The step S3 of locating the damaged position of the bridge expansion joint according to the sound signal comprises the following steps:

[0184] According to the law of cosines:

[0185]

[0186] q i represents the microphone coordinates (u i ,v i ,w i ), Y represents the coordinates of the sound source to be located (u Y ,v Y ,w Y ), R i Indicates microphone q i Distance to the origin, R Y Indicates the distance from the sound source Y to the origin, d i Indicates the sound source Y to q0 and q i The distance difference:

[0187] d i =Δt·c (19)

[0188] Δt represents the distance from the sound source Y to q0 and q i The time difference is , c represents the speed of sound, and formula (16) can be further obtained:

[0189]

[0190] Due to d i It is obtained by estimating the time delay. There will be a deviation ε between formula (16) and the actual value:

[0191]

[0192] When the deviation is minimum, the position of the sound source Y can be determined:

[0193]

[0194] Convert formula (18) into matrix form:

[0195]

[0196] in, Simplifying further:

[0197] ε=Gμ-β (24)

[0198] Where G = [QD],

[0199] The least squares solution can be expressed as:

[0200] μ=(G T G) -1 G T β (25)

[0201] Define the projection matrix along G to D:

[0202]

[0203] The projection matrix along D to G is:

[0204]

[0205] According to the properties of the projection matrix:

[0206] G=P D [Q 0] (28)

[0207] Substitute into formula (21):

[0208] ε=P D QY-β (29)

[0209] Final sound source Y positioning result:

[0210] Y=(Q T P D Q) -1 Q T P D b (30).

[0211] The step S3 of locating the damaged position of the bridge expansion joint according to the audio and video signals comprises the following steps:

[0212] Step S321, combining visual and auditory features, and enhancing key features through spatial attention mechanism and self-attention mechanism;

[0213] Step S322, selecting semantically consistent segments for feature fusion;

[0214] Step S323, using the fused features to determine the specific location of the injury;

[0215] Step S324, judging the consistency between the audio and video fusion positioning result and the sound signal sound source positioning result, and correcting the single-mode positioning result if they are inconsistent;

[0216] Step S325, the consistent positioning results are integrated at the decision layer and the positioning results are output.

[0217] The step S4 outputs the damage identification and location results including:

[0218] Step S401, outputting expansion joint damage identification results according to the trained deep learning network;

[0219] Step S402, outputting the precise location where the damage occurs according to the positioning consistency judgment result;

[0220] Step S403, vehicle identification and monitoring are performed according to the characteristics of the video signal.

[0221] The damage maintenance and management includes performing preventive maintenance based on the damage identification results and positioning results. The method of the present invention is described in detail below with reference to the accompanying drawings:

[0222] like Figure 1 As shown in the figure, the detection platform is established to collect vibration, sound and video signals of expansion joints in different damage states. The platform collects vibration, sound and video signals. The preprocessing of vibration signals, sound signals and video signals includes noise removal and signal feature enhancement.

[0223] In the feature extraction stage, key features are extracted from the preprocessed signals. Feature extraction for vibration and sound signals focuses on identifying damage-indicating features, while feature extraction for video signals identifies video patterns and changes associated with damage.

[0224] In the feature fusion stage, the features of vibration, sound and video signals are integrated to build a comprehensive damage identification model. The establishment of the model includes damage identification and positioning consistency judgment to ensure the accuracy and reliability of damage status identification.

[0225] In the result output stage, the model outputs damage identification and location results, and simultaneously performs vehicle identification and monitoring. These results provide key information for damage monitoring and maintenance of expansion joints, ensuring the effectiveness of the detection platform.

[0226] like Figure 2As shown in the figure, after the sound and vibration signals are collected, the sound signal is first pre-enhanced to improve the recognition of important features. Subsequently, the signal is framed and windowed, and a window function is applied to each segment to reduce edge effects. Next, the short-time Fourier transform (STFT) is used to convert the signal from the time domain to the frequency domain and analyze its frequency components. The Mel filter processing converts the frequency domain signal into a Mel frequency spectrum to further refine the characteristics of the sound signal. The vibration signal is processed by the improved MTST method, which adaptively determines the optimal number of windows, extracts the key features in the vibration signal, and generates a time-frequency evolution diagram. Finally, the Mel power spectrum of the sound signal is combined with the time-frequency evolution diagram of the vibration signal to generate an acoustic-vibration fusion diagram.

[0227] like Figure 3 As shown, the sound source position of the damaged signal can be determined by the propagation time difference of the sound signal.

[0228] like Figure 4 As shown in the figure, the features of the sound signal and video frame are first extracted, and then the features are enhanced and the consistent segments are selected for fusion to form the audio and video fusion features. Next, the system will make a preliminary judgment on the positioning results to check whether the positioning results are consistent. If the positioning results are inconsistent, the process will turn to the correction of the single-mode positioning results to ensure the accuracy of the positioning. If the positioning results are consistent, the process will enter the decision-making layer fusion stage to further integrate and optimize the positioning results. Finally, the system outputs the fused and optimized positioning results.

[0229] The specific implementation operation method described above further explains the technical scheme and beneficial effects of the present invention in detail. It should be understood that the above is a specific implementation mode of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A bridge expansion joint damage detection and positioning method based on multimodal signal fusion, characterized in that: The method comprises the following steps: Step S1, building a detection platform to collect multi-source data of vibration signals, sound signals and video signals; Step S2, preprocessing the vibration signal, the sound signal and the video signal respectively, and extracting their respective features, including time domain, frequency domain and time-frequency domain features; Step S3, multi-modal fusion of the vibration signal, the sound signal and the video signal; by constructing a damage identification model, the fused multi-modal features are analyzed to achieve identification and location of the bridge expansion joint damage; Step S4, output damage identification and location results to provide decision support for the maintenance and management of bridge expansion joints.

2. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The detection platform of step S1 includes a vibration sensor array, a microphone array, a camera and a signal acquisition device; the vibration signal, the sound signal and the video signal are derived from the sound signal, the vibration signal and the video signal generated by the vehicle passing through the expansion joint, and the acquisition area is the lanes on both sides close to the expansion joint.

3. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The feature extraction of the sound signal in step S2 comprises the following steps: Step S211, pre-enhancement processing is performed on the collected sound signal; S p (n)=S(n)-αS(n-1) (1) S p (n) represents the pre-emphasized sound signal, α represents the pre-emphasis coefficient; S(n) represents the original sound signal, n represents the sampling point, Step S212, framing and windowing the pre-enhanced sound signal; The pre-enhanced sound signal is divided into short frames, and N sampling points are first grouped into an observation frame; In order to avoid too large changes between two adjacent frames, there is an overlapping area between two adjacent frames. This overlapping area contains N / 2 sampling points. After the signal is framed, each frame is substituted into the window function for windowing: S w (n)=S P (n)×w(n) (2) S w (n) represents the windowed signal, and w(n) represents the windowing function; Apply a Hamming window to reduce spectral leakage: Step S213, applying short-time Fourier transform (STFT) to calculate the power spectrum; X t (k) represents the frequency response of the signal at frequency k in the tth frame, P t (k) represents the power spectrum of the signal at frequency k in the tth frame; Step S214, using the Mel filter bank to obtain the Mel power spectrum and Mel cepstral coefficients MFCC; Each frequency value f corresponds to a Mel value Mel(f): The power spectrum is passed through a filter bank with M triangular filters. The frequency response of the triangular filter at frequency k is defined as: The logarithmic energy of each filter bank output is calculated as: The MFCC coefficients are obtained by discrete cosine transform DCT: L represents the MFCC coefficient order.

4. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The step S2 of extracting the characteristic changes of the vibration signal comprises the following steps: Step 221, perform discrete S transform on the vibration signal U(t) to obtain S transform coefficients: s U,m (t,f) represents the S transform coefficient, ψ m (t,f) represents the time-frequency window function, lΔt represents the discrete time point, and Δt represents the sampling interval; Step 222, sinusoidal time-frequency window function for: w(f) represents the window shape function, a, b and c represent the parameters that control the window shape, and f s Indicates the signal sampling frequency; Step 223, Hermite time-frequency window function for: Step 224, using the improved multi-window spectrum S transform method to calculate the time-frequency evolution spectrum matrix of the signal: *, T represent the adjoint matrix and the transpose of the matrix respectively represents the video evolution spectrum matrix, and M(t,f) represents the number of windows that change with t and f: Step 224, determine the optimal M(t,f) for each frequency point through an iterative process: Use the first window to perform S-transform on all time and frequency points to generate an initial evolution spectrum estimate; For each frequency f, the sum of the spectrum values ​​at all time points is used to estimate the total spectrum S sum (f)) Fit a quadratic function in the frequency interval [fW,f+W], where W represents the resolution bandwidth; Calculate S from the fitted quadratic function sum (f) the first-order derivative y′ and the second-order derivative y″ with respect to f; Use the derivative obtained from the fitting to calculate the optimal M. The specific calculation formula is: L(f) represents the number of time points at frequency f; Smooth M using forward and backward filtering techniques, and ensure that the number of windows between consecutive frequencies does not vary by more than 1; The above steps were repeated until the mean relative difference (MRD) in the number of windows between consecutive iterations was less than a predefined value of 10%. Step 225, extracting features including energy distribution, frequency components, etc. from the evolution spectrum.

5. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The feature change extraction of the video signal in step S2 comprises the following steps: Step S231, pre-processing the video, adjusting the video frame rate to a speed suitable for processing, adjusting the video resolution to a size suitable for processing, and using a filter to remove image noise; Step S232, performing frame extraction on the preprocessed video signal, and extracting key frames containing vehicle information and expansion joints from the video; Step S233, extracting basic visual features of the expansion joint from the key frame, including its width, length, and color, and identifying whether the expansion joint has signs of damage, including cracks, deformation, or material shedding, and locating the damage location; Step S234, performing vehicle recognition detection on the key frame, using background subtraction technology to identify the foreground vehicle, and further segmenting the vehicle by color, shape and texture features; Step S235, analyzing the contact point between the vehicle and the expansion joint, and detecting the state change of the expansion joint after the vehicle passes through; Step S235 , analyzing the behavior of the vehicle when passing through the expansion joint, including the speed, path, and any abnormal behavior, to identify vehicle behavior patterns that may cause damage to the expansion joint.

6. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The step S3 of constructing a damage identification model for the vibration signal and the sound signal comprises the following steps: Step S311, mapping the features of different modes to a common feature space, aligning the features of different modes to ensure that they correspond to the same expansion joint state in time; Step S312, weighting the features according to the contribution of the vibration signal and the sound signal to damage identification; The sound feature map F1 and the vibration feature map F2 are added element by element according to the weights to form a sound-vibration feature map F: represents element-by-element addition, and θ represents the signal feature weight; Step S313: Use the acoustic vibration feature map to train a deep learning network to identify damage.

7. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The step S3 of locating the damaged position of the bridge expansion joint according to the sound signal comprises the following steps: According to the law of cosines: q i represents the microphone coordinates (u i ,v i ,w i ), Y represents the coordinates of the sound source to be located (u Y ,v Y ,w Y ), R i Indicates microphone q i Distance to the origin, R Y Indicates the distance from the sound source Y to the origin, d i Indicates the sound source Y to q0 and q i The distance difference: d i =Δt·c (19) Δt represents the distance from the sound source Y to q0 and q i The time difference is , c represents the speed of sound, Formula (16) can be further obtained: Due to d i It is obtained by estimating the time delay. There will be a deviation ε between formula (16) and the actual value: When the deviation is minimum, the position of the sound source Y can be determined: Convert formula (18) into matrix form: in, Simplifying further: ε=Gμ-β (24) Where G = [QD], The least squares solution can be expressed as: μ=(G T G) -1 G T b (25) Define the projection matrix along G to D: The projection matrix along D to G is: According to the properties of the projection matrix: G=P D [Q 0] (28) Substitute into formula (21): e=P D QY-β (29) Final sound source Y positioning result: Y=(Q T P D Q) -1 Q T P D b (30)。 8. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The step S3 of locating the damaged position of the bridge expansion joint according to the audio and video signals comprises the following steps: Step S321, combining visual and auditory features and enhancing key features through an attention mechanism; Step S322, selecting semantically consistent segments for feature fusion; Step S323, using the fused features to determine the specific location of the injury; Step S324, judging the consistency between the audio and video fusion positioning result and the sound signal sound source positioning result, and correcting the single-mode positioning result if they are inconsistent; Step S325, the consistent positioning results are integrated at the decision layer and the positioning results are output.

9. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The step S4 outputs the damage identification and location results including: Step S401, outputting expansion joint damage identification results according to the trained deep learning network; Step S402, outputting the precise location where the damage occurs according to the positioning consistency judgment result; Step S403: Vehicle identification and monitoring are performed based on the video signal features.

10. The bridge expansion joint damage detection and positioning method based on multimodal signal fusion according to claim 1 is characterized in that: The damage maintenance and management includes performing preventive maintenance based on damage identification results and positioning results.

Images