Structural condition monitoring methods, devices, equipment and storage media

Abstract

This application relates to the field of structural health monitoring technology, and discloses a method, device, equipment, and storage medium for structural condition monitoring, including: acquiring vibration signals and image sequences of a target structure; extracting features from the vibration signals to obtain a modal parameter set, and extracting features from the image sequences to obtain an apparent deformation parameter set; performing temporal alignment and adaptive weighted fusion of the features in the modal parameter set and the apparent deformation parameter set to obtain fused features; inputting the fused features into a random forest convolutional neural network hybrid model, outputting the anomaly probability of the target structure, and determining the health status of the target structure based on the anomaly probability. By simultaneously acquiring vibration and visual data, and extracting corresponding features for fusion, a mapping relationship reflecting internal dynamic characteristics and external geometric deformation is established, overcoming the shortcomings of traditional single monitoring methods with incomplete dimensions, and realizing integrated, high-precision monitoring and early warning of internal structural damage and apparent deformation.

Description

Technical Field

[0001] This application relates to the field of structural health monitoring technology, and in particular to a method, apparatus, equipment and storage medium for structural condition monitoring. Background Technology

[0002] Structural health monitoring technologies mainly include modal analysis based on vibration signals and deformation recognition based on visual images. However, in existing technologies, vibration and visual data are processed independently, lacking an effective multimodal data fusion mechanism. This results in fragmented monitoring information, making it difficult to support a comprehensive and accurate assessment of structural safety status. In particular, it is difficult to overcome the inherent differences between the two in sampling rate, time series, and feature dimensions, making it impossible to establish an accurate correlation mapping between the internal dynamic characteristics of the structure and the external apparent deformation, leading to incomplete and inaccurate structural health status assessments. Summary of the Invention

[0003] The main objective of this application is to provide a structural condition monitoring method, device, equipment, and storage medium, which aims to solve the technical problem that existing structural health monitoring technologies have difficulty in effectively integrating multimodal data, resulting in weak correlation between internal damage and external deformation and limited assessment accuracy.

[0004] To achieve the above objectives, this application proposes a structural state monitoring method, the method comprising:

[0005] Acquire vibration signals and image sequences of the target structure;

[0006] Feature extraction is performed on the vibration signal to obtain a set of modal parameters, and feature extraction is performed on the image sequence to obtain a set of apparent deformation parameters;

[0007] The features in the modal parameter set and the apparent deformation parameter set are temporally aligned and adaptively weighted and fused to obtain fused features;

[0008] The fused features are input into a random forest convolutional neural network hybrid model, which outputs the anomaly probability of the target structure and determines the health status of the target structure based on the anomaly probability.

[0009] Optionally, the step of performing temporal alignment and adaptive weighted fusion of the features in the modal parameter set and the apparent deformation parameter set to obtain fused features includes:

[0010] The features in the modal parameter set and the apparent deformation parameter set are standardized to generate a standardized feature vector, which includes a vibration feature sequence and a visual feature sequence.

[0011] The vibration feature sequence and the visual feature sequence are time-synchronized based on the dynamic time warping algorithm to obtain synchronized vibration features and visual features.

[0012] Based on the signal-to-noise ratio of the corresponding channels of the vibration signal and the image sequence, the vibration features and visual features are adaptively weighted and stitched together to generate fused features.

[0013] Optionally, the sampling rate of the vibration feature sequence is greater than the sampling rate of the visual feature sequence;

[0014] The step of synchronizing the vibration feature sequence and the visual feature sequence based on the dynamic time warping algorithm to obtain synchronized vibration features and visual features includes:

[0015] Based on the dynamic time warping algorithm, the minimum cumulative distance path between the vibration feature sequence and the visual feature sequence is calculated;

[0016] The vibration feature sequence is non-uniformly resampled according to the minimum cumulative distance path to obtain vibration features corresponding to the visual features in the visual feature sequence.

[0017] Optionally, the step of adaptively weighting and stitching the vibration features and visual features based on the signal-to-noise ratio of the corresponding channels of the vibration signal and the image sequence to generate fused features includes:

[0018] Determine the first signal-to-noise ratio of the vibration channel corresponding to the vibration signal, and the second signal-to-noise ratio of the visual channel corresponding to the image sequence;

[0019] The first weighting coefficient of the vibration feature is calculated based on the first signal-to-noise ratio and the second signal-to-noise ratio, and the second weighting coefficient of the visual feature is calculated based on the first weighting coefficient.

[0020] The vibration features and visual features are adaptively weighted and stitched together based on the first weighting coefficient and the second weighting coefficient to generate fused features.

[0021] Optionally, the step of extracting features from the vibration signal to obtain a set of modal parameters includes:

[0022] The vibration signal is preprocessed using a filter to obtain a clean signal;

[0023] Perform a fast Fourier transform on the clean signal to obtain the frequency domain response of each measurement point in the target structure and the input force signal corresponding to the frequency domain response;

[0024] The frequency response function is determined based on the frequency domain response and the input force signal;

[0025] Based on the frequency response function, the high-precision modal parameters to be identified in the target structure are identified by an improved least-squares complex frequency domain algorithm, resulting in a modal parameter set, which includes modal frequencies, damping ratios, and mode shapes.

[0026] Optionally, the step of extracting features from the image sequence to obtain a set of apparent deformation parameters includes:

[0027] The image sequence is processed using a super-resolution reconstruction algorithm to obtain a high-resolution image sequence;

[0028] Extract and match robust feature points between consecutive frames in the image sequence;

[0029] The target structure is reconstructed based on the robust feature points to generate a corresponding high-precision 3D point cloud;

[0030] Based on the deformation parameters of the target structure calculated by the high-precision three-dimensional point cloud, an apparent deformation parameter set is obtained, which includes at least the tilt angle, settlement, and crack width.

[0031] Optionally, the random forest convolutional neural network hybrid model includes a first branch and a second branch;

[0032] The step of inputting the fused features into a random forest convolutional neural network hybrid model and outputting the anomaly probability of the target structure includes:

[0033] The fused features are input into a random forest convolutional neural network hybrid model. The vibration features in the fused features are processed through the first branch to obtain a first processing result, and the visual features in the fused features are processed through the second branch to obtain a second processing result.

[0034] By combining the first processing result and the second processing result, the anomaly probability of the target structure is obtained.

[0035] Furthermore, to achieve the above objectives, this application also proposes a structural condition monitoring device, which includes:

[0036] A multi-source sensing module is used to acquire vibration signals and image sequences of the target structure being monitored.

[0037] The feature extraction module is used to extract features from the vibration signal to obtain a modal parameter set, and to extract features from the image sequence to obtain an apparent deformation parameter set.

[0038] The data fusion module is used to perform temporal alignment and adaptive weighted fusion of the features in the modal parameter set and the apparent deformation parameter set to obtain fused features;

[0039] The intelligent early warning module is used to input the fused features into a random forest convolutional neural network hybrid model, output the abnormal probability of the target structure, and determine the health status of the target structure based on the abnormal probability.

[0040] In addition, to achieve the above objectives, this application also proposes a structural condition monitoring device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the structural condition monitoring method as described above.

[0041] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the structural state monitoring method described above.

[0042] This application simultaneously collects vibration and visual data, extracts corresponding features from each, and fuses them to establish a mapping relationship reflecting internal dynamic characteristics and external geometric deformation. This overcomes the shortcomings of traditional single-sensor methods, which lack comprehensive dimensions, and achieves integrated, high-precision monitoring and early warning of internal damage and apparent deformation in structures such as buildings and pipelines. The complementary fusion of vibration and visual data enables simultaneous perception of internal structural damage and apparent deformation. Vibration modal parameters (such as mode shapes) can sensitively reflect changes in internal mass distribution, while visual 3D reconstruction can accurately capture millimeter-level apparent deformation. The combination of the two not only expands the monitoring dimensions but also eliminates the uncertainties of single-sensor methods through data cross-validation. Attached Figure Description

[0043] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0044] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart illustrating the first embodiment of the structural state monitoring method of this application;

[0046] Figure 2 This is a schematic diagram of the data processing and feature extraction module of this application;

[0047] Figure 3This is a flowchart illustrating the second embodiment of the structural state monitoring method of this application;

[0048] Figure 4 This is a schematic diagram of the multimodal data fusion process in this application;

[0049] Figure 5 This is a flowchart illustrating the third embodiment of the structural state monitoring method of this application;

[0050] Figure 6 This is a schematic diagram of the architecture of the structural status monitoring system of this application;

[0051] Figure 7 This is a schematic diagram of the module structure of the structural status monitoring device according to an embodiment of this application;

[0052] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the structural state monitoring method in this application embodiment.

[0053] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0055] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0056] Structural health monitoring technology has currently formed two main directions: one is modal analysis based on vibration signals, which is widely used for internal condition diagnosis of structures such as pipelines and bridges. By deploying acceleration sensors, resonant frequencies and mode shapes are extracted, and AI algorithms are used to identify anomalies (such as pipeline blockage). However, it cannot directly obtain information on structural surface deformation. The second is deformation recognition based on visual images, which is suitable for monitoring building surface damage. It uses cameras to collect images and image processing algorithms to identify surface features such as tilting and cracks. However, it is not sensitive to changes in internal structure and is greatly affected by the environment.

[0057] However, these two technologies, when processed independently, have significant limitations in perception dimensions and application scenarios. While vibration monitoring can capture changes in the dynamic characteristics inside a structure, it is insensitive to external features such as cracks and tilting; visual monitoring directly reflects surface morphology but struggles to detect internal damage or is significantly affected by environmental disturbances. Existing technologies lack effective multimodal data fusion mechanisms, resulting in fragmented monitoring information that cannot support a comprehensive and accurate assessment of structural safety status. Furthermore, the inability to establish a correlation between the structure's internal dynamic characteristics and external apparent deformation leads to incomplete and inaccurate structural health status assessments.

[0058] Therefore, this application provides a structural state monitoring method based on the fusion of vibration and visual data. Through a multi-source temporal data adaptive alignment and feature-level fusion framework and a lightweight RF-CNN (Random Forest-Convolutional Neural Network) hybrid intelligent recognition algorithm, under the enhanced monitoring paradigm of "one sense, multiple recognitions," it achieves integrated, high-precision, and low-cost intelligent monitoring and early warning of internal damage and apparent deformation of structures such as buildings and pipelines.

[0059] It should be noted that the executing entity in this embodiment can be a computing service device with multi-source data acquisition, feature extraction, feature fusion, and program execution functions, such as a computer, or an electronic device capable of performing the above functions. The following description uses a structural state monitoring system as an example to illustrate this embodiment and the subsequent embodiments.

[0060] Based on this, embodiments of this application provide a structural state monitoring method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the structural state monitoring method of this application.

[0061] In this embodiment, the structural state monitoring method includes:

[0062] Step S10: Collect vibration signals and image sequences of the target structure.

[0063] It should be noted that the target structure refers to the physical structure or equipment that requires health monitoring, such as buildings, bridges, pipelines, storage tanks, towers, or industrial equipment. The integrity of the structure is directly related to safety and service life. Vibration signals refer to the sequence of physical quantities collected by sensors that reflect the dynamic acceleration, velocity, or displacement of the target structure under environmental or artificial excitation over time. These signals contain information about the structure's inherent dynamic characteristics, such as natural frequencies, damping ratios, and mode shapes. Image sequences refer to the collection of two-dimensional images of the target structure, continuously acquired by optical imaging equipment and arranged in chronological order. These images contain external visual information such as the structure's geometry, surface texture, color, and spatial location.

[0064] Specifically, as an optional implementation, a multi-source sensing module can be used to comprehensively acquire multi-physics data reflecting the structural state, such as vibration signals and visual image data. The multi-source sensing module consists of two highly coordinated sensing units: a vibration sensing unit and a visual sensing unit. The vibration sensing unit uses MEMS (Micro-Electro-Mechanical Systems) accelerometers or piezoelectric accelerometers, deployed at key dynamic response points of the structure (such as pipe support points, building load-bearing columns, etc.). It synchronously samples according to preset sampling conditions (such as sampling frequency ≥1kHz, accuracy ±0.1g) to acquire multi-channel vibration time-domain signals. The visual sensing unit includes a visible light camera (resolution ≥1080p) and a thermal infrared camera (thermal sensitivity ≤50mK) to acquire sequences of structural appearance images. The device can also be equipped with gimbal control to achieve multi-angle coverage, outputting timestamp-aligned image sequences containing rich appearance geometry and texture information. .

[0065] Step S20: Extract features from the vibration signal to obtain a modal parameter set, and extract features from the image sequence to obtain an apparent deformation parameter set.

[0066] It should be noted that the modal parameter set refers to a set of parameters calculated from vibration signals that characterize the inherent dynamic properties of a structure, including but not limited to the modal frequencies, modal damping ratios, and mode shapes of the structure. These parameters constitute the "dynamic fingerprint" of the structure, and their changes directly reflect changes in the internal mass, stiffness, or connection state of the structure. The apparent deformation parameter set refers to a set of parameters calculated from image sequences that quantifies changes in the external geometric shape of the structure, including but not limited to overall tilt angle, local settlement, crack width, and deflection. These parameters intuitively reflect the visible deformation of the structure under load or damage.

[0067] In one example, reference Figure 2 , Figure 2 This is a schematic diagram of the data processing and feature extraction module of this application. The data processing and feature extraction module is responsible for transforming the raw sensor data into high-value features characterizing the structural state. The module consists of two parallel pipelines: a vibration signal processing chain and a visual image processing chain, which extract the dynamic characteristic parameters and geometric deformation parameters of the structure, respectively, providing a foundation for subsequent fusion. The vibration signal processing chain takes the raw vibration acceleration time-domain signal acquired synchronously from multiple channels as input. It first performs preprocessing by filtering to remove high-frequency noise and low-frequency drift, then performs frequency domain transformation through fast Fourier transform, subsequently calculates the frequency response function and accurately identifies modal parameters such as resonance frequency, damping ratio, and mode shape, and finally obtains a standardized modal parameter set through standardization.

[0068] The visual image processing chain receives image sequences acquired by visible light / thermal infrared cameras, first enhances the images through super-resolution reconstruction algorithms, then extracts and matches robust feature points to reconstruct a high-precision 3D point cloud, and then calculates deformation parameters such as tilt angle, settlement, and crack width. These parameters are then standardized to form a standardized deformation parameter set. Finally, the two types of standardized feature vectors are integrated into a unified standardized feature vector, providing basic data for collaborative analysis in the subsequent multimodal data fusion module.

[0069] Furthermore, to obtain structural feature information more accurately and stably, and to provide highly sensitive and reliable quantitative indicators for detecting internal damage (such as stiffness loss and local defects), the step of extracting features from the vibration signal to obtain a modal parameter set may include:

[0070] The vibration signal is preprocessed using a filter to obtain a clean signal; a fast Fourier transform is performed on the clean signal to obtain the frequency domain response of each measuring point in the target structure and the input force signal corresponding to the frequency domain response; a frequency response function is determined based on the frequency domain response and the input force signal; based on the frequency response function, the high-precision modal parameters to be identified in the target structure are identified using an improved least-squares complex frequency domain algorithm to obtain a modal parameter set, which includes modal frequencies, damping ratios, and mode shapes.

[0071] It should be noted that "clean signal" refers to vibration time-domain data obtained after filtering, resulting in a significantly improved signal-to-noise ratio. "Input force signal" refers to the frequency domain representation of the synchronously acquired excitation force time-domain signal under known excitation (such as a hammer strike signal), obtained through a Fast Fourier Transform. The frequency response function describes the relationship between the output (structural response) and input (excitation force) of a linear system in the frequency domain. High-precision modal parameters refer to the structural dynamic characteristic parameters stably and accurately identified by this algorithm. Among these, modal frequency refers to the natural frequency of each order of vibration of the structure, damping ratio describes the rate of vibration energy dissipation, and mode shape describes the spatial distribution of the structure's vibration at a specific frequency.

[0072] In one example, a Butterworth bandpass filter is used in the preprocessing stage to remove high-frequency noise and low-frequency drift. An FFT (Fast Fourier Transform) is then performed on the filtered clean signal to obtain the frequency domain response at each measurement point. Simultaneously, if a known excitation is used, the input force signal acquired by the force sensor is also subjected to FFT to obtain the input force signal. .

[0073] Then, the resonant frequency is calculated using the FRF (Frequency Response Function). and mode shape The formula for calculating FRF is as follows:

[0074]

[0075] in: This is the frequency response function, with units of m / N; For response point The Fourier transform of is expressed in meters (m). As incentive points The Fourier transform of , with units of N; It is the excitation frequency, and the unit is rad / s; is the damping ratio of the nth mode; N is the total number of system modes; It is the undamped natural frequency of the nth mode, and its unit is rad / s; It is the nth mode.

[0076] Building upon this, an improved PolyMAX (Polyreference Maximum Likelihood) algorithm is applied for modal parameter identification. The core of this algorithm is solving a regularized least-squares optimization problem. The objective function is to minimize the error norm between the measured FRF and the model-predicted FRF, with a regularization term added. To prevent overfitting and improve numerical stability, modal frequencies are extracted stably and accurately. Damping ratio and mode shape { },as follows:

[0077]

[0078] in, Represents the set of modal parameters to be identified; Represents the k-th discrete frequency point; Representative at The frequency response function matrix obtained from the measurement, where each element of the matrix contains a frequency response function; Representative at The parameters are as follows The frequency response function predicted by the defined modal model; Represents the regularization coefficient; This represents the regularization function.

[0079] By solving the above optimization problem, the modal parameter set of the output structure is obtained. This reveals the inherent dynamic characteristics of the structure.

[0080] Furthermore, in order to transform two-dimensional image information into accurate three-dimensional geometric information through three-dimensional reconstruction technology, thereby directly calculating key safety indicators such as tilt angle, settlement, and crack width, making the results of visual monitoring more robust, the feature extraction of the image sequence to obtain the apparent deformation parameter set may include:

[0081] The image sequence is processed using a super-resolution reconstruction algorithm to obtain a high-resolution image sequence; robust feature points between consecutive frames in the image sequence are extracted and matched; the target structure is reconstructed based on the robust feature points to generate a corresponding high-precision three-dimensional point cloud; the deformation parameters of the target structure are calculated based on the high-precision three-dimensional point cloud to obtain an apparent deformation parameter set, which includes at least the tilt angle, settlement, and crack width.

[0082] It should be noted that super-resolution reconstruction algorithms are image processing techniques aimed at reconstructing a high-resolution image from one or more low-resolution images. For example, deep learning-based SRCNN models can be used to enhance the detail and clarity of an image. Robust feature points refer to local key points in an image sequence that remain stable and repeatable despite scale changes, rotations, and illumination variations. A high-precision 3D point cloud is a collection of numerous 3D spatial points (each containing XYZ coordinates), representing a digital representation of the surface geometry of a target structure. Deformation parameters are physical quantities derived from the 3D point cloud through geometric calculations, quantifying structural changes. The tilt angle typically refers to the angle between the main plane of the structure (e.g., a wall) and the vertical direction of gravity. Settlement refers to the vertical displacement of a specific point on the structure relative to a reference position. Crack width refers to the physical opening size of cracks on the structural surface.

[0083] In one example, a visual image processing chain can transform a two-dimensional image into a three-dimensional deformation, aiming to accurately capture the geometric changes of a structural surface, given an input sequence of images acquired by a visible light / thermal infrared camera. For each t=1,2,...,T, super-resolution reconstruction (such as the SRCNN algorithm) is used to improve image resolution and output a high-resolution image. ,as follows:

[0084]

[0085] in For low-resolution input, For network parameters, The algorithm used is SRCNN. Subsequently, the SIFT (Scale-Invariant Feature Transform) algorithm is run on the enhanced consecutive image frames to extract and match robust feature points between consecutive frames. Next, these robust feature points are input into an incremental SfM (Structure from Motion) pipeline, and the 3D structure of the scene is reconstructed by analyzing the image sequence, thus reconstructing a high-precision 3D point cloud of the monitored target. Based on this point cloud, the corresponding deformation parameters are calculated.

[0086] Regarding the overall tilt of the building, the tilt angle The normal vector of the main plane (such as the wall) can be calculated. With gravity vector The angle between the two angles is obtained as follows:

[0087]

[0088] in, This is the surface normal vector. This is the direction vector of gravity, usually (0,0,1).

[0089] By tracking specific benchmarks Elevation changes The settlement amount δ is obtained. For the crack width w, the crack region can be identified on the image using pixel width. Calculate the physical width using camera calibration parameters , where s represents the actual size for each pixel.

[0090] Finally, the apparent deformation parameter set of the target structure is output. The external geometric changes of the quantification structure.

[0091] Step S30: Perform temporal alignment and adaptive weighted fusion of the features in the modal parameter set and the apparent deformation parameter set to obtain fused features.

[0092] It should be noted that the temporal alignment and adaptive weighted fusion of features from the modal parameter set and the apparent deformation parameter set can be performed within the multimodal data fusion module. This aims to address the heterogeneity and asynchronicity between vibration and visual data, achieving information complementarity and synergistic enhancement, and providing unified, high-quality input features for subsequent state recognition. The fused feature refers to a joint feature vector integrating vibration and visual information generated after fusion processing.

[0093] Understandably, in this embodiment, feature fusion may include sub-steps such as standardization, temporal alignment, and weighted concatenation. Standardization aims to eliminate differences in units and numerical ranges between different features; temporal alignment aims to solve the problem of misalignment of feature sequences on the time axis caused by different sampling rates (e.g., 1kHz for vibration signals and 25fps for video); weighted concatenation assigns appropriate weights to different features based on the real-time reliability of each data source and then merges them to obtain fused features, and the weights of the corresponding features can be calculated based on the signal-to-noise ratio of each channel.

[0094] Alternatively, in signal-to-noise ratio (SNR) calculation, the SNR of the visual channel can be a composite quality score that integrates image sharpness, contrast, illumination uniformity, and the presence or absence of occlusion (rain, fog), rather than the traditional signal-to-noise power ratio.

[0095] Step S40: Input the fused features into the random forest convolutional neural network hybrid model, output the abnormal probability of the target structure, and determine the health status of the target structure based on the abnormal probability.

[0096] It should be noted that, in this embodiment, the random forest convolutional neural network hybrid model is a composite model architecture designed to process fused features. It contains two core branches: a random forest branch, which excels at processing structured numerical data such as vibration features; and a convolutional neural network branch, which excels at processing spatial or sequential data related to visual features. The anomaly probability is the predicted probability value output by the random forest convolutional neural network hybrid model, representing the likelihood of one or more anomalies (such as tilt exceeding limits, crack development, or internal damage) in the target structure. The health status is a state level determined based on the anomaly probability through preset decision rules (such as multi-level threshold comparison), and can be classified as "safe," "attention," "warning," or "alarm."

[0097] Optionally, before deploying the model, it is necessary to perform fully supervised training on the RF-CNN hybrid model using multimodal data under both historical normal and abnormal conditions. During training, methods such as cross-validation can be used to optimize model hyperparameters (e.g., tree depth in random forests, learning rate in CNNs) and branch fusion weights. The system can also support an online learning mechanism, fine-tuning the model based on new monitoring data and human feedback to adapt to changes in the environment or the model itself.

[0098] It is understood that in this embodiment, the intelligent early warning module can receive the anomaly probability analysis results from the multimodal data fusion module and transform them into actionable early warning information, visual reports, and decision suggestions. The core of the intelligent early warning module lies in its multi-level early warning mechanism, visual human-computer interaction, and closed-loop feedback management.

[0099] In one example, the intelligent early warning module adopts a multi-level early warning model based on a combination of fuzzy logic and probability thresholds to achieve an intelligent transformation from "anomaly perception" to "risk decision-making".

[0100] Input model output anomaly probability vector Each element represents the probability of an anomaly type (such as tilting, settlement, cracking, blockage) occurring. Decisions are made based on multi-level thresholds, as follows:

[0101] Safety: (For example The system is functioning normally; routine data recording and trend baseline updates are being performed.

[0102] Notice: (For example The system flags a minor anomaly, prompting operations and maintenance personnel to monitor trend changes.

[0103] Warning: (For example The system automatically generates an early warning work order, suggesting that on-site inspections be arranged.

[0104] Call the police: The system triggered the highest-level alarm, initiated the emergency response process, and automatically notified the relevant personnel.

[0105] The thresholds T mentioned above are not fixed, but are dynamically and adaptively adjusted according to the importance level of the structure, the statistical characteristics of historical data, and environmental factors (such as season and temperature) to reduce false alarms.

[0106] Meanwhile, the system displays 3D structural models, anomaly location markers, and historical trend charts via a web interface or mobile app. For example, color gradients can be used to display the stress distribution or deformation degree on the structural surface, and flashing icons can be used to accurately locate cracks or blockages.

[0107] Of course, the system will push alerts and implement closed-loop management based on the comparison results. It supports multiple alarm methods such as SMS, email, and API interface, and generates a unique alarm event ID. After on-site handling, maintenance personnel need to report the handling results through the APP (such as "false alarm confirmed" or "hardened and processed"), upload on-site photos, and record the complete "alarm-handling-feedback" pipeline to form a traceable closed-loop management file for optimizing the algorithm model.

[0108] In this embodiment, by simultaneously collecting vibration and visual data and extracting corresponding features for fusion, a mapping relationship reflecting internal dynamic characteristics and external geometric deformation is established. This overcomes the shortcomings of traditional single monitoring methods that lack comprehensive dimensions, and realizes integrated, high-precision monitoring and early warning of internal damage and apparent deformation of structures such as buildings and pipelines.

[0109] Reference Figure 3 , Figure 3 This is a flowchart illustrating the second embodiment of the structural state monitoring method of this application. Based on the first embodiment described above, a second embodiment of the structural state monitoring method of this application is proposed.

[0110] In the second embodiment, step S30 includes:

[0111] Step S301: Standardize the features in the modal parameter set and the apparent deformation parameter set to generate a standardized feature vector, which includes a vibration feature sequence and a visual feature sequence.

[0112] As can be understood, a standardized eigenvector refers to a unified numerical sequence formed by standardizing the original modal parameters (e.g., frequency in Hertz, mode shapes dimensionless but potentially very small) and apparent deformation parameters (e.g., tilt angle in degrees, settlement in millimeters), and then arranging them in order. The vibration feature sequence and the visual feature sequence are two components of the standardized eigenvector, corresponding to the standardized set of modal parameters and the set of apparent deformation parameters, respectively.

[0113] Specifically, in order to eliminate dimensional differences, for and All features are Z-score standardized to form a uniform standardized feature vector. This is to prepare for future integration.

[0114]

[0115] in, Represents the original feature value; Represents the mean; Represents standard deviation.

[0116] In one example, the standardization process employs the Z-Score standardization method. First, based on historical data or a sufficiently large training set, the mean μ and standard deviation σ of each feature parameter (e.g., first-order modal frequency, second-order damping ratio, overall tilt angle, maximum crack width w, etc.) are calculated. Then, for each feature value F extracted from real-time monitoring, the above formula is applied for calculation. All vibration-related parameters are standardized and arranged in a predetermined order to form a vibration feature sequence. Similarly, all visually relevant deformation parameters are standardized to form a visual feature sequence. These two sequences together constitute the standardized feature vector at the current monitoring time.

[0117] Step S302: The vibration feature sequence and the visual feature sequence are time-synchronized based on the dynamic time warping algorithm to obtain synchronized vibration features and visual features.

[0118] It is understandable that vibration signals have a high sampling rate (≥1 kHz), while video frame rates are low (25-30 fps), and there may be slight discrepancies in their initial acquisition times, leading to data asynchrony on the timeline. To address the timing asynchrony issue caused by the different sampling rates of vibration signals and image sequences, it is necessary to synchronize the vibration feature sequences and the visual feature sequences in time.

[0119] Furthermore, to ensure precise alignment of vibrational and visual features on the time axis and avoid temporal asynchrony, step S302 may include:

[0120] Based on the dynamic time warping algorithm, the minimum cumulative distance path between the vibration feature sequence and the visual feature sequence is calculated; the vibration feature sequence is then non-uniformly resampled according to the minimum cumulative distance path to obtain the vibration feature corresponding to the visual feature in the visual feature sequence.

[0121] It should be noted that the DTW (Dynamic Time Warping) algorithm is used to measure the similarity between two time series of different lengths or velocities. Through dynamic programming, it allows the sequences to be non-linearly "bent" and "stretched" along the time axis to find an optimal alignment path that minimizes the overall distance between the two sequences. The minimum cumulative distance path is the alignment mapping relationship obtained by the dynamic time warping algorithm, which minimizes the overall difference between the vibration feature sequence and the visual feature sequence. It is usually represented as a sequence of coordinate pairs, where each coordinate pair indicates which time point in the visual sequence corresponds to a time point in the vibration sequence.

[0122] It is understandable that non-uniform resampling refers to selectively extracting or generating new data points from the original vibration feature sequence, not according to a fixed time interval, but according to the correspondence specified by the minimum cumulative distance path, thereby forming a new sequence that is strictly aligned with the visual feature sequence in time. The vibration feature corresponding to the visual feature refers to the data point of the vibration feature obtained after the above resampling that precisely matches each visual feature point (i.e., each time frame) in the visual feature sequence in a temporal sense.

[0123] Specifically, in this embodiment, the DTW algorithm is used to calculate the minimum cumulative distance path between two sequences through dynamic programming. This achieves non-uniform resampling of high-frequency vibration data, precisely aligning it with visual features on the time axis. The formula is as follows:

[0124]

[0125] in, X and Y These represent vibrational feature sequences and visual feature sequences, respectively. and These are data points from the vibration feature sequence and the visual feature sequence, respectively. It is a point and The Euclidean distance between them. This path. This indicates the correspondence between data points in the two sequences. Based on the optimal path... Non-uniform resampling is performed on vibration data sequences with higher sampling rates to precisely align them with visual feature sequences at time points, laying the foundation for subsequent feature-level fusion.

[0126] Step S303: Based on the signal-to-noise ratio of the corresponding channels of the vibration signal and the image sequence, adaptively weightedly splice the vibration features and visual features to generate fused features.

[0127] It should be noted that the signal-to-noise ratio (SNR) of a channel is used to measure the reliability of the data source. For the vibration channel, the SNR quantifies the ratio of the power of the useful vibration signal to the background noise; for the visual channel, the SNR can be extended to a comprehensive image quality score, covering factors such as sharpness, contrast, illumination uniformity, and occlusion.

[0128] Understandably, the aligned vibration characteristics (such as modal frequencies) Mode shape Features (such as tilt angle θ and settlement δ) have different physical dimensions and numerical ranges from visual features. Directly concatenating them can cause the model to be biased towards features with larger numerical values. To enable the system to automatically prioritize more reliable data sources in complex environments, step S303 may include:

[0129] Determine the first signal-to-noise ratio (SNR) of the vibration channel corresponding to the vibration signal and the second SNR of the visual channel corresponding to the image sequence; calculate the first weighting coefficient of the vibration feature based on the first SNR and the second SNR, and calculate the second weighting coefficient of the visual feature using the first weighting coefficient; perform adaptive weighted splicing of the vibration feature and the visual feature based on the first weighting coefficient and the second weighting coefficient to generate a fused feature.

[0130] In one example, the fusion weights can be dynamically adjusted based on the real-time signal-to-noise ratio of the vibration and visual channels, enabling the system to automatically prioritize more reliable data sources in complex environments.

[0131] First, the standardized features are adaptively weighted and concatenated.

[0132]

[0133] Among them, the weighting coefficient and It is not a fixed value, but is dynamically adjusted based on the real-time signal-to-noise ratio of the vibration and visual channels:

[0134]

[0135] in, It is the first weighting coefficient of the vibration characteristics, a dimensionless value between 0 and 1. It represents the proportion of the vibration data's reliability at the current moment in the total reliability; It is the visual feature weight coefficient, which is also a dimensionless value between 0 and 1. It represents the reliability percentage of the visual data at the current moment.

[0136] The first signal-to-noise ratio (in dB) of the vibration channel is an indicator of the vibration signal quality, and its calculation formula is:

[0137]

[0138] in, Represents signal power. This represents the power of the noise signal. The higher the value, the purer the collected vibration signal, the less it is affected by environmental noise, and the higher the reliability.

[0139] The signal-to-noise ratio (SNR) of the visual channel is an indicator of image signal quality. In computer vision, this can refer to the image's SNR, but it can be more accurately extended to an image quality score. Its calculation is based on image processing algorithms that comprehensively consider image sharpness, contrast, illumination uniformity, and the presence or absence of occlusion (rain, snow, fog, haze, etc.). The higher the value, the better the image quality, and the more accurate the appearance features extracted from it.

[0140] The above strategies enable the system to adapt to different environments. For example, at night or on foggy days, the quality of visual images degrades (…). (Reduce), the system will automatically increase the weight of vibration characteristics. Conversely, in environments with high mechanical noise interference, visual features are relied upon more heavily. This significantly improves the robustness of the system.

[0141] In one example, reference Figure 4 , Figure 4This is a flowchart illustrating the multimodal data fusion process of this application. The process sequentially completes three core operations: data temporal alignment, feature fusion, and intelligent recognition. First, starting with the input feature sequence, the optimal path is calculated using the Dynamic Time Warping (DTW) algorithm, and then temporal alignment features are obtained through non-uniform resampling. Subsequently, using these temporal alignment features as input, the real-time signal-to-noise ratio (SNR) is calculated, and adaptive weighted concatenation is performed using a dynamic weight allocation strategy to generate a fused feature vector. Finally, the fused feature vector is received and processed by an RF-CNN hybrid model, where the CNN branch handles visual features and the RF branch handles vibration features. The processing results of the two branches are then fused by a decision layer, ultimately outputting the anomaly probability.

[0142] In this embodiment, standardization first eliminates the dimensional differences between different physical quantity characteristics, providing a fair benchmark for subsequent calculations. Temporal synchronization solves the temporal misalignment problem caused by different sampling rates, ensuring the accuracy of data association. By dynamically planning to calculate the minimum cumulative distance path between two sequences, non-uniform resampling of high-frequency vibration data is achieved, making it precisely aligned with visual features on the time axis and avoiding temporal asynchrony. Finally, adaptive weighting dynamically adjusts the contribution of the data source according to the signal-to-noise ratio, significantly improving the robustness and reliability of the entire fusion system in complex and variable real-world environments.

[0143] Reference Figure 5 , Figure 5 This is a flowchart illustrating the third embodiment of the structural state monitoring method of this application. Based on the above embodiments, a third embodiment of the structural state monitoring method of this application is proposed.

[0144] In the third embodiment, the random forest convolutional neural network hybrid model includes a first branch and a second branch, and step S40 includes:

[0145] Step S401: Input the fused features into the random forest convolutional neural network hybrid model, process the vibration features in the fused features through the first branch to obtain a first processing result, and process the visual features in the fused features through the second branch to obtain a second processing result.

[0146] It should be noted that the first branch here specifically refers to the branch in the hybrid model built on random forest, whose structure is an integration of multiple decision trees, and it excels at processing numerical features from vibration signals that have clear physical meaning and structured characteristics. The second branch here specifically refers to the branch in the hybrid model built on convolutional neural network, whose structure includes convolutional layers, pooling layers, etc., and it excels at processing visually related features with spatial or sequential patterns. The first and second processing results are intermediate representations output by the two branches after performing deep feature extraction and transformation on their respective input features, usually high-dimensional feature vectors or preliminary classification / regression scores, which form the basis for the final decision layer fusion.

[0147] Specifically, as an optional implementation, the system routes the vibration features of the fused features (such as standardized modal frequencies, mode shape components, etc.) to a random forest branch based on a predefined dimensional index, leveraging its excellent ability to process structured data and resist overfitting to obtain a first processing result. The system then routes the visual features of the fused features (such as standardized deformation parameter sequences, or reshaped feature maps with spatial dimensions) to a convolutional neural network branch, utilizing its powerful spatial feature extraction capabilities to capture patterns such as cracks and tilting, thus obtaining a second processing result.

[0148] Step S402: Combine the first processing result and the second processing result to obtain the anomaly probability of the target structure.

[0149] In one example, the outputs of the two branches are fused at the decision layer using a sigmoid function. Output the overall anomaly probability ,as follows:

[0150]

[0151] in, Let y be the probability of outputting y given input data x. In this model, y specifically refers to abnormal states, such as tilting, settlement, cracks, etc. The Sigmoid function is used to map the fused linear output to probability values, ensuring that the result is within the [0,1] interval, which facilitates decision-making by subsequent multi-level early warning modules. and These are learnable fusion weight coefficients used to balance the contributions of the CNN and RF branches. This indicates that the k-th decision tree is designed for vibration characteristics. The first processing result of the predicted output, Let be the weight of the k-th decision tree. The visual features output by the CNN branch The second processing result.

[0152] In one example, reference Figure 6 , Figure 6 This is a schematic diagram of the structural condition monitoring system of this application. First, the vibration sensor array and visual perception device of the multi-source sensing module collect raw vibration signals and raw image sequences, respectively. The data is then transmitted to the data processing and feature extraction module. This module generates standardized modal parameter sets and standardized deformation parameter sets through the vibration signal processing chain and visual image processing chain, respectively. Subsequently, the two parameter sets enter the multi-modal data fusion module, where data time-series alignment and feature-level fusion are performed sequentially to obtain a fused feature vector. This vector is then intelligently identified and outputs anomaly probabilities using a preset model. Finally, the anomaly probabilities flow into the intelligent early warning module, which completes the condition assessment and generates visualized early warning information. Furthermore, it generates and pushes early warning commands based on the condition assessment and adjusts the parameters of the preset model based on feedback from on-site processing results, thus achieving a closed-loop monitoring process from multi-source data acquisition, processing and fusion to intelligent early warning push.

[0153] In one example, in a scenario of detecting blockages in an industrial pipeline, the resin transport pipeline in a chemical plant is prone to polymer coagulation after long-term operation, leading to localized blockages. Vibration sensors (11 in total) can be installed every 5 meters along a pipeline approximately 50 meters long; an explosion-proof high-definition camera is installed above a critical section of the pipeline. The pipeline is excited by a hammer impact method, and the vibration response is collected. An improved PolyMAX algorithm is then used to accurately identify the modal frequencies of each order. and mode shape By monitoring minute displacements at pipe supports via video and reconstructing the pipe axis morphology using SfM technology, the system aligns and fuses the vibration-identified modal changes with the visually measured pipe axis deformation using DTW (Dynamic Transformation and Weaving) and inputs the data into an RF-CNN model. The system successfully identified a blockage 22.5 meters from the starting point and determined its severity to be moderate. This precise location and quantitative assessment of pipe blockages avoids preventative downtime, reduces unplanned downtime by 70%, improves production efficiency, and prevents the risk of pipe rupture caused by blockages.

[0154] In this embodiment, the CNN branch excels at extracting visual spatial features, while the RF branch is adept at processing structured vibration data. By fusing the outputs of the two branches at the decision layer, a comprehensive anomaly probability is output. This allows the final anomaly probability to incorporate deep evidence from both internal and external structures and different data formats, resulting in a more comprehensive and accurate identification result than any single model.

[0155] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the structural state monitoring method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0156] This application also provides a structural condition monitoring device; please refer to [reference needed]. Figure 7 The structural condition monitoring device includes:

[0157] The multi-source sensing module 10 is used to acquire vibration signals and image sequences of the target structure being monitored;

[0158] The feature extraction module 20 is used to extract features from the vibration signal to obtain a modal parameter set, and to extract features from the image sequence to obtain an apparent deformation parameter set.

[0159] Data fusion module 30 is used to perform temporal alignment and adaptive weighted fusion of features in the modal parameter set and the apparent deformation parameter set to obtain fused features;

[0160] The intelligent early warning module 40 is used to input the fused features into the random forest convolutional neural network hybrid model, output the abnormal probability of the target structure, and determine the health status of the target structure based on the abnormal probability.

[0161] The structural condition monitoring device provided in this application, employing the structural condition monitoring method described in the above embodiments, can solve the technical problem in existing structural health monitoring technologies where multimodal data is difficult to fuse effectively, resulting in weak correlation between internal damage and external deformation and limited assessment accuracy. Compared with the prior art, the beneficial effects of the structural condition monitoring device provided in this application are the same as those of the structural condition monitoring method provided in the above embodiments, and other technical features in the structural condition monitoring device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0162] This application provides a structural condition monitoring device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the structural condition monitoring method in Embodiment 1 above.

[0163] The following is for reference. Figure 8The diagram illustrates a structural state monitoring device suitable for implementing embodiments of this application. The structural state monitoring device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The structural condition monitoring device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0164] like Figure 8 As shown, the structural condition monitoring device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory 1002 or a program loaded from a storage device 1003 into a random access memory 1004. The random access memory 1004 also stores various programs and data required for the operation of the structural condition monitoring device. The processing unit 1001, the read-only memory 1002, and the random access memory 1004 are interconnected via a bus 1005. An input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the structural condition monitoring equipment to communicate wirelessly or wiredly with other devices to exchange data. Although the figure shows structural condition monitoring equipment with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0165] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0166] The structural condition monitoring device provided in this application, employing the structural condition monitoring method described in the above embodiments, can solve the technical problem in existing structural health monitoring technologies where multimodal data is difficult to effectively fuse, resulting in weak correlation between internal damage and external deformation and limited assessment accuracy. Compared with the prior art, the beneficial effects of the structural condition monitoring device provided in this application are the same as those of the structural condition monitoring method provided in the above embodiments, and other technical features of this structural condition monitoring device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0167] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0168] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0169] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the structural state monitoring method in the above embodiments.

[0170] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0171] The aforementioned computer-readable storage medium may be included in the structural condition monitoring device; or it may exist independently and not be assembled into the structural condition monitoring device.

[0172] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the structural condition monitoring device, cause the structural condition monitoring device to perform the structural condition monitoring method described above.

[0173] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0175] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0176] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described structural condition monitoring method. This solves the technical problem in existing structural health monitoring technologies where multimodal data is difficult to fuse effectively, leading to weak correlation between internal damage and external deformation and limited assessment accuracy. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the structural condition monitoring method provided in the above embodiments, and will not be repeated here.

[0177] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

Claims

1. A method for monitoring structural condition, characterized in that, The method includes: Acquire vibration signals and image sequences of the target structure; Feature extraction is performed on the vibration signal to obtain a modal parameter set, and feature extraction is performed on the image sequence to obtain an apparent deformation parameter set. The modal parameter set includes modal frequency, damping ratio and mode shape, and the apparent deformation parameter set includes at least tilt angle, settlement and crack width. The features in the modal parameter set and the apparent deformation parameter set are temporally aligned and adaptively weighted and fused to obtain fused features; The fused features are input into a random forest convolutional neural network hybrid model, which outputs the anomaly probability of the target structure and determines the health status of the target structure based on the anomaly probability. The random forest convolutional neural network hybrid model includes a first branch and a second branch. The step of extracting features from the vibration signal to obtain a modal parameter set includes: The vibration signal is preprocessed using a filter to obtain a clean signal; Perform a fast Fourier transform on the clean signal to obtain the frequency domain response of each measurement point in the target structure and the input force signal corresponding to the frequency domain response; The frequency response function is determined based on the frequency domain response and the input force signal; Based on the frequency response function, the high-precision modal parameters to be identified in the target structure are obtained by using an improved least-squares complex frequency domain algorithm, thus obtaining a set of modal parameters. The step of extracting features from the image sequence to obtain the apparent deformation parameter set includes: The image sequence is processed using a super-resolution reconstruction algorithm to obtain a high-resolution image sequence; Extract and match robust feature points between consecutive frames in the image sequence, wherein the robust feature points are stable and repeatable local key points of the image; The target structure is reconstructed based on the robust feature points to generate a corresponding high-precision 3D point cloud; Based on the deformation parameters of the target structure calculated using the high-precision three-dimensional point cloud, a set of apparent deformation parameters is obtained.

2. The structural condition monitoring method as described in claim 1, characterized in that, The step of performing temporal alignment and adaptive weighted fusion of features from the modal parameter set and the apparent deformation parameter set to obtain fused features includes: The features in the modal parameter set and the apparent deformation parameter set are standardized to generate a standardized feature vector, which includes a vibration feature sequence and a visual feature sequence. The vibration feature sequence and the visual feature sequence are time-synchronized based on the dynamic time warping algorithm to obtain synchronized vibration features and visual features. Based on the signal-to-noise ratio of the corresponding channels of the vibration signal and the image sequence, the vibration features and visual features are adaptively weighted and stitched together to generate fused features.

3. The structural condition monitoring method as described in claim 2, characterized in that, The sampling rate of the vibration feature sequence is greater than the sampling rate of the visual feature sequence; The step of synchronizing the vibration feature sequence and the visual feature sequence based on the dynamic time warping algorithm to obtain synchronized vibration features and visual features includes: Based on the dynamic time warping algorithm, the minimum cumulative distance path between the vibration feature sequence and the visual feature sequence is calculated; The vibration feature sequence is non-uniformly resampled according to the minimum cumulative distance path to obtain vibration features corresponding to the visual features in the visual feature sequence.

4. The structural condition monitoring method as described in claim 2, characterized in that, The step of adaptively weighting and stitching the vibration features and visual features based on the signal-to-noise ratio of the corresponding channels of the vibration signal and the image sequence to generate fused features includes: Determine the first signal-to-noise ratio of the vibration channel corresponding to the vibration signal, and the second signal-to-noise ratio of the visual channel corresponding to the image sequence; The first weighting coefficient of the vibration feature is calculated based on the first signal-to-noise ratio and the second signal-to-noise ratio, and the second weighting coefficient of the visual feature is calculated based on the first weighting coefficient. The vibration features and visual features are adaptively weighted and stitched together based on the first weighting coefficient and the second weighting coefficient to generate fused features.

5. The structural condition monitoring method as described in claim 1, characterized in that, The step of inputting the fused features into a random forest convolutional neural network hybrid model and outputting the anomaly probability of the target structure includes: The fused features are input into a random forest convolutional neural network hybrid model. The vibration features in the fused features are processed through the first branch to obtain a first processing result, and the visual features in the fused features are processed through the second branch to obtain a second processing result. By combining the first processing result and the second processing result, the anomaly probability of the target structure is obtained.

6. A structural condition monitoring device, characterized in that, The device includes: A multi-source sensing module is used to acquire vibration signals and image sequences of the target structure being monitored. The feature extraction module is used to extract features from the vibration signal to obtain a modal parameter set, and to extract features from the image sequence to obtain an apparent deformation parameter set. The modal parameter set includes modal frequency, damping ratio and mode shape, and the apparent deformation parameter set includes at least tilt angle, settlement and crack width. The data fusion module is used to perform temporal alignment and adaptive weighted fusion of features from the modal parameter set and the apparent deformation parameter set to obtain fused features; The intelligent early warning module is used to input the fused features into the random forest convolutional neural network hybrid model, output the abnormal probability of the target structure, and determine the health status of the target structure based on the abnormal probability. The random forest convolutional neural network hybrid model includes a first branch and a second branch. The feature extraction module is further configured to preprocess the vibration signal based on a filter to obtain a clean signal; perform a fast Fourier transform on the clean signal to obtain the frequency domain response of each measuring point in the target structure and the input force signal corresponding to the frequency domain response; determine the frequency response function based on the frequency domain response and the input force signal; and identify the high-precision modal parameters to be identified in the target structure based on the frequency response function using an improved least squares complex frequency domain algorithm to obtain a modal parameter set. The feature extraction module is further configured to process the image sequence based on a super-resolution reconstruction algorithm to obtain a high-resolution image sequence; extract and match robust feature points between consecutive frames in the image sequence, wherein the robust feature points are stable and repeatable local key points of the image; reconstruct the target structure based on the robust feature points to generate a corresponding high-precision three-dimensional point cloud; and obtain an apparent deformation parameter set based on the deformation parameters of the target structure calculated from the high-precision three-dimensional point cloud.

7. A structural condition monitoring device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the structural state monitoring method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the structural state monitoring method as described in any one of claims 1 to 5.

Images