Expansion joint monitoring device and method based on audio and video

Abstract

The invention relates to the technical field of bridge equipment, and discloses an expansion joint monitoring device and method based on audio and video, and the method comprises the following steps: 1, collecting the image information of a target point location and the audio information of a target scene, and forming a disease database; the target point location comprises an expansion joint on the bridge and a cover beam at the position; the target scene comprises a scene in which the vehicle travels through the expansion joint; step 2, based on the disease database, dividing disease grades of the expansion joint according to a preset rule; according to the disease grade, the monitoring frequency required by the expansion joint is determined; and step 3, monitoring the expansion joint according to the monitoring frequency, analyzing the disease development condition of the expansion joint according to the monitoring condition, adjusting the monitoring frequency when the disease development condition shows that the disease grade is upgraded, and outputting a maintenance suggestion. According to the invention, efficient, accurate and safe expansion joint monitoring can be realized, the bridge inspection and maintenance efficiency can be improved, and the safety risk can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of bridge equipment, and particularly relates to a monitoring device and method for expansion joints based on audio and video. Background Art

[0002] With the vigorous development of highway construction in China, the mileage of opened-to-traffic highways has been increasing year by year. During the construction of highways, bridges, as key transportation facilities, are also increasing in number. Expansion joints, as important connecting components of bridges, play a key role in ensuring the smooth passage of vehicles across the bridge deck and adapting to the deformation of the upper structure of the bridge. Once local damage occurs to the expansion device and is not discovered and repaired in time, it is extremely easy to deteriorate rapidly, ultimately leading to overall damage and failure. This will not only affect the driving comfort but also pose a serious threat to driving safety. For example, the steel beam of the modular expansion device on the Guangzhou-Zhuhai section of the Guangao Expressway fractured and was damaged, resulting in a serious accident where a car rolled over and fell, and tire blowouts caused by local damage to the expansion device occur frequently. Thus, it is very necessary to effectively monitor bridge expansion joints, which is directly related to road traffic safety and people's travel safety.

[0003] Currently, the diseases of expansion devices have attracted great attention from the maintenance department. However, there is a lack of innovation and effectiveness in the inspection methods. Daily inspections, regular inspections, and periodic inspections still mainly rely on manual visual inspection. This traditional inspection method, although simple to operate, has many limitations. On the one hand, the timeliness of inspection is low, and it is difficult to discover diseases in real time and quickly. On the other hand, for highway inspectors, the safety risk is relatively high, and there are significant safety hazards when inspecting sections with heavy traffic. At the technical level, internal diseases and external subtle diseases are not easily discovered. The internal structure of the expansion joint is relatively complex, and it is difficult to deeply understand its internal condition using traditional visual inspection methods. External subtle diseases are also easily overlooked due to their small size. At the same time, the development process of diseases is not clear. Due to the lack of effective monitoring means, it is difficult to accurately grasp the whole process of disease occurrence and development, which makes the maintenance department lack sufficient basis when formulating maintenance strategies. In addition, there is a lack of an early warning mechanism for expansion device diseases, and it is impossible to issue an alarm in advance before the disease occurs, resulting in the disease often being discovered when it is already relatively serious, increasing the maintenance cost and safety risk.

[0004] Therefore, how to overcome these difficulties and develop more efficient, accurate, and safe bridge expansion joint monitoring technologies and methods has become an urgent problem to be solved currently. Summary of the Invention

[0005] The present invention aims to provide a monitoring device and method for expansion joints based on audio and video, which can achieve efficient, accurate, and safe monitoring of expansion joints, contribute to improving the inspection and maintenance efficiency of bridges, and reducing safety risks.

[0006] To achieve the above object, the present invention provides the following basic solution.

[0007] Solution 1

[0008] A method for monitoring expansion joints based on audio and video, comprising the following steps:

[0009] Step 1, collect image information of the target point and audio information of the target scene, and form a disease database; the target point includes the expansion joint on the bridge and the capping beam at this position; the target scene includes the scene of a vehicle driving through the expansion joint;

[0010] Step 2, based on the disease database, divide the disease level of the expansion joint according to preset rules; and determine the required monitoring frequency of the expansion joint according to the disease level;

[0011] Step 3, monitor the expansion joint according to the monitoring frequency, analyze the development status of the disease of the expansion joint according to the monitoring situation, and when the disease development status shows an upgrade of the disease level, adjust the monitoring frequency and output maintenance suggestions.

[0012] Further, it further includes Step 4, generating a daily report of the expansion joint according to the monitoring situation.

[0013] Further, in Step 1, a high-definition camera is used to collect image information, and a microphone is used to collect audio information.

[0014] Further, in Step 1, before forming the disease database, the image information and audio information are also preprocessed; the preprocessing includes: distortion correction, multispectral fusion, and defect enhancement for the image information; noise suppression and feature extraction for the audio information.

[0015] Further, after the preprocessing, a first defect detection algorithm is also used to output the disease type and disease probability of the expansion joint corresponding to the preprocessed image information according to the preprocessed image information; the first defect detection algorithm includes an input layer, a dual-branch detection network, a multimodal fusion module, and an output layer; the dual-branch detection network includes a YOLOv8-seg network for detecting surface defects and a UNet++ network for detecting internal damage that are arranged in parallel; the YOLOv8-seg network is provided with an adaptive anchor box mechanism, and a CBAM module is inserted into the Backbone of the YOLOv8-seg network.

[0016] Further, after the preprocessing, a second defect detection algorithm is also used to output the disease type and disease probability of the expansion joint corresponding to the preprocessed audio information according to the preprocessed audio information; the second defect detection algorithm includes an input layer, a BiLSTM-Attention network, and an output layer.

[0017] Further, in step 1, the preprocessing further includes: performing spatio-temporal alignment on the image information and the audio information to form groups of information pairs; after the first defect detection algorithm and the second defect detection algorithm are completed, the detection results of the two are also verified; the verification includes: comparing the detection results of each information pair; if the matching degree of the detection results is higher than the threshold, the verification is passed; if the matching degree of the detection results is lower than the threshold, offline re-inspection is performed on the expansion joint corresponding to the information pair.

[0018] Further, in step 2, when dividing the disease levels, a Bayesian network is used for intelligent division.

[0019] Further, in step 3, when analyzing the disease development status of the expansion joint, an LSTM model based on physical information is used for analysis; the LSTM model based on physical information takes the current disease type, disease level and environmental parameters of the expansion joint in the monitoring situation as inputs, and takes the disease upgrade probability as the output.

[0020] Solution Two

[0021] An expansion joint monitoring device based on audio and video includes a high-definition camera, a microphone, and a monitoring control system that establishes communication and control connections with the high-definition camera and the microphone; a controller is provided in the monitoring control system; the method of an expansion joint monitoring based on audio and video as described in Solution One is written in the controller; the high-definition camera and the microphone are arranged at the bridge guardrails on both sides of the expansion joint or mounted on a mobile inspection vehicle, and the installation height of the high-definition camera satisfies that its field of view covers the entire expansion joint.

[0022] The working principle and advantages of the present invention are as follows:

[0023] An expansion joint monitoring device and method based on audio and video of the present invention can achieve efficient, accurate and safe expansion joint monitoring, which helps to improve the efficiency of bridge inspection and maintenance and reduce safety risks. The key points are:

[0024] First, the operation mode of this solution is simple. It can operate based on a high-definition camera and a microphone, and can be mounted on a daily inspection vehicle to achieve non-stop detection, which helps to protect the safety of inspection personnel and reduce the traffic impact on the operating highway. It can also be fixedly installed on the guardrail near the expansion joint to facilitate mastering the diseases inside and outside the expansion device and their development trends, so as to timely give disease warnings and take remedial measures.

[0025] Second, this solution combines image information and audio information to identify the diseases of expansion joints, with high accuracy in evaluating damage identification. Based on the image information, the macroscopic diseases on the surface of the expansion joints can be quickly captured. By combining the image information and audio information, the microscopic diseases and the diseases inside the structure can be further identified, which is adapted to the characteristics of the modular and comb-type expansion joint devices commonly used in highway bridges.

[0026] Third, this solution specially sets up the first defect detection algorithm and the second defect detection algorithm, which can improve the efficiency and accuracy of disease detection. Among them, the first defect detection algorithm integrates multiple network structures, which can accurately distinguish different disease types and comprehensively identify surface damage; the second defect detection algorithm can identify abnormal structural vibrations based on audio information, and then comprehensively identify internal damage. The two are fused and verified to accurately confirm the disease type and disease probability of the expansion joint. Description of the Drawings

[0027] Figure 1 It is a schematic diagram of the layout of the high-definition camera and the microphone in the first embodiment of a monitoring device and method for expansion joints based on audio and video of the present invention;

[0028] Figure 2 It is a general schematic diagram of the monitoring device in the first embodiment of a monitoring device and method for expansion joints based on audio and video of the present invention;

[0029] Figure 3 It is a schematic diagram of the method flow in the first embodiment of a monitoring device and method for expansion joints based on audio and video of the present invention.

[0030] The marks in the attached drawings of the specification include: high-definition camera 1, microphone 2, and monitoring control system 3. Detailed Embodiments

[0031] The following is a more detailed description through specific embodiments:

[0032] The embodiment is basically as shown in the attached Figure 3 shown: A monitoring method for expansion joints based on audio and video includes the following steps:

[0033] Step 1, collect the image information of the target point and the audio information of the target scene, and form a disease database; the target point includes the expansion joint on the bridge and the capping beam at this position; the target scene includes the scene of the vehicle passing through the expansion joint.

[0034] In this step, a high-definition camera is used to collect image information, and a pick-up is used to collect audio information. In this embodiment, the high-definition camera is a multi-spectral high-definition camera with a resolution higher than 4K and integrated with the near-infrared band, which can collect fine image information, penetrate surface dirt, and detect relevant structural information of the expansion joint (such as the structural aging information of rubber seals, etc.). The pick-up is a pick-up integrated with a MEMS microphone array to facilitate separating vehicle noise and structural vibration signals through time-frequency domain features. Preferably, an environmental sensor group can also be added to monitor temperature and humidity information at the expansion joint, etc., which helps to fully analyze the state of the expansion joint.

[0035] Specifically, before forming the disease database, the image information and audio information are also preprocessed; the preprocessing includes: distortion correction, multi-spectral fusion, and defect enhancement for the image information; in this embodiment, the distortion correction method uses an existing geometric correction algorithm (such as Zhang Zhengyou calibration method), the multi-spectral fusion is to fuse visible light and the near-infrared band to enhance specific structural features (such as the aging features of rubber parts), and the defect enhancement is processed using the existing CLANE algorithm to enhance the image information reflecting defect features such as crack width, rubber part deformation, section steel damage, and rust area of the anchorage area. Noise suppression and feature extraction for the audio information. In this embodiment, both the noise suppression and feature extraction are completed by adjusting the spectrogram.

[0036] After preprocessing, a first defect detection algorithm is also used to output the disease type and disease probability of the expansion joint corresponding to the preprocessed image information according to the preprocessed image information; the first defect detection algorithm includes an input layer, a double-branch detection network, a multi-modal fusion module, and an output layer; the double-branch detection network includes a YOLOv8-seg network for detecting surface defects and a UNet++ network for detecting internal damage, which are arranged in parallel; the YOLOv8-seg network is provided with an adaptive anchor box mechanism, and a CBAM module is inserted into the Backbone of the YOLOv8-seg network. Among them, the YOLOv8-seg network has high operating efficiency and can achieve real-time processing. The adaptive anchor box mechanism means that based on the size of the expansion joint components, the K-means++ algorithm is used to re-cluster the anchor boxes to distinguish minor damage, medium damage, and large-area damage. By adding a CBAM module, the feature expression ability of the network can be further enhanced, and the algorithm performance can be enhanced.

[0037] Preferably, a first self-built dataset is also used to strengthen the training of the first defect detection algorithm. The first self-built training set contains image information of target points of expansion joints with different disease types in ordinary scenarios and interference scenarios with interference items such as rain, fog, and oil stains, which helps to improve the accuracy of algorithm detection.

[0038] After preprocessing, a second defect detection algorithm is also used to output the disease type and disease probability of the expansion joint corresponding to the preprocessed audio information according to the preprocessed audio information; the second defect detection algorithm includes an input layer, a BiLSTM-Attention network, and an output layer. Among them, the BiLSTM-Attention network has a high classification accuracy in time-frequency features, can accurately analyze audio information, and extract temporal features.

[0039] Preferably, a second self-built dataset is also used to strengthen the training of the second defect detection algorithm. The second self-built training set contains audio information of the target scenarios of expansion joints with different disease types under normal scenarios and interference scenarios with interference items such as background noise (which can be imported from the highway noise database), which helps to improve the accuracy of algorithm detection.

[0040] Step 2: Based on the disease database, divide the disease level of the expansion joint according to a preset rule; and determine the monitoring frequency required for the expansion joint according to the disease level.

[0041] In this embodiment, when dividing the disease level, a Bayesian network is used for intelligent division based on a preset rule.

[0042] Among them, the preset rule is set based on the existing bridge detection specifications. Specifically, there are slight differences in the division rules for urban bridges and highway bridges. Specifically, according to the corresponding specifications, such as the "Code for Maintenance of Highway Bridges and Culverts" (JTG 5120—2021), the "Technical Code for Maintenance of Urban Bridges" (CJJ 99-2017), etc., a Bayesian network is used to qualitatively analyze and rate the existing diseases according to the corresponding disease description characteristics.

[0043] As described in the following tables (Table 1, Table 2, Table 3, and Table 4): Check whether the expansion joint is blocked and ineffective, whether there are defects and damages in the anchorage area, whether there is breakage, whether there are unevennesses, etc., and then qualitatively analyze and rate the existing diseases.

[0044] Table 1 Evaluation criteria for unevenness

[0045]

[0046]

[0047] Table 2 Evaluation criteria for anchorage area defects

[0048]

[0049] Table 3 Evaluation criteria for breakage

[0050]

[0051]

[0052] Table 4 Failure Assessment Criteria

[0053]

[0054] Step 3: Monitor the expansion joint according to the monitoring frequency, analyze the disease development status of the expansion joint based on the monitoring situation, adjust the monitoring frequency when the disease development status shows an upgrade in the disease level, and output maintenance suggestions.

[0055] When analyzing the disease development status of the expansion joint, an LSTM model based on physical information is used for analysis; the LSTM model based on physical information takes the current disease type, disease level, and environmental parameters of the expansion joint in the monitoring situation as inputs and the disease upgrade probability as the output.

[0056] Step 4: Generate a daily report of the expansion joint according to the monitoring situation.

[0057] In this embodiment, the content of the daily report of the expansion joint includes: basic information of the expansion joint (such as the location and type of the expansion joint), monitoring information (monitoring date and time, monitoring frequency), status and disease conditions of the expansion joint (such as apparent disease conditions, such as whether there are cracks and damages, describe their sizes, shapes, distribution ranges, etc.; status of the rubber sealing strip, whether there are damages, aging, etc., and the impact on the surrounding structure, such as whether there is water seepage on the capping beam; internal structure conditions, such as whether the steel section is damaged and whether the anchorage area is damaged; disease type; disease level; disease development status), maintenance suggestions (such as repair, reinforcement, adjustment of the monitoring frequency, etc.).

[0058] As attached Figure 1 and Figure 2 As shown, this embodiment also provides an expansion joint monitoring device based on audio and video, including a high-definition camera, a microphone, and a monitoring control system that establishes communication and control connections with the high-definition camera and the microphone; a controller is provided in the monitoring control system; the above-mentioned expansion joint monitoring method based on audio and video is written in the controller; the high-definition camera and the microphone are arranged at the bridge guardrails on both sides of the expansion joint or mounted on a mobile inspection vehicle, and the installation height of the high-definition camera satisfies that its field of view covers the entire expansion joint.

[0059] In specific applications, the installation positions of the high-definition camera and the microphone can be selected according to actual inspection or maintenance needs.

[0060] A kind of expansion joint monitoring device and method based on audio and video provided by this embodiment monitors the expansion joint by using a high-definition camera and a pickup, and combined with the set monitoring method, it can evaluate and monitor the bridge expansion joint device more conveniently and safely, and can form a disease database about the expansion joint device, which can provide more accurate suggestions for subsequent maintenance, and can better monitor the working state of the expansion joint device, reduce the impact of the diseases of the expansion joint on the whole bridge, so as to better monitor the safety of the bridge and further extend the service life of the bridge. This solution can realize efficient, accurate and safe expansion joint monitoring, help improve the inspection and maintenance efficiency of the bridge, and reduce safety risks.

[0061] Embodiment 2

[0062] A kind of expansion joint monitoring method based on audio and video, on the basis of Embodiment 1, in step 1, the preprocessing further includes: performing spatio-temporal alignment on the image information and the audio information to form groups of information pairs.

[0063] After the first defect detection algorithm and the second defect detection algorithm finish detection, the detection results of the two are also verified; the verification includes: comparing the detection results of each information pair; if the matching degree of the detection results (including the consistency of the detected disease type and disease probability) is higher than the threshold (in actual application, this threshold is set according to actual needs), then the verification is passed; if the matching degree of the detection results is lower than the threshold, then offline re-inspection is carried out on the expansion joint corresponding to this information pair.

[0064] A kind of expansion joint monitoring device and method based on audio and video provided by this embodiment has higher detection result accuracy compared with Embodiment 1.

[0065] The above are only the embodiments of the present invention. Specific structures and common knowledge such as characteristics well known in the art are not described in detail here. Those of ordinary skill in the art know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can know all the existing technologies in this field, and have the ability to apply the conventional experimental means before this date. Those of ordinary skill in the art can, under the inspiration given by this application, combine their own abilities to improve and implement this solution. Some typical well-known structures or well-known methods should not become an obstacle for those of ordinary skill in the art to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can still be made, and these should also be regarded as the protection scope of the present invention, and these will not affect the implementation effect of the present invention and the practicability of the patent.

Claims

1. A method for monitoring expansion joints based on audio and video, characterized in that: The following steps are involved: Step 1, collecting image information of target points and audio information of target scenes, and forming a disease database; the target points include expansion joints on bridges and cap beams at the locations; the target scenes include scenes of vehicles driving through expansion joints; Step 2: Based on the disease database, classify the disease level of the expansion joint according to the preset rules; and determine the monitoring frequency required for the expansion joint according to the disease level; Step 3, monitor the expansion joints at the monitoring frequency, and analyze the disease development of the expansion joints based on the monitoring situation. When the disease development situation shows that the disease level has been upgraded, adjust the monitoring frequency and output maintenance suggestions.

2. The method for monitoring expansion joints based on audio and video according to claim 1, characterized in that: It also includes step 4, generating a daily report on the expansion joint based on the monitoring situation.

3. The method for monitoring expansion joints based on audio and video according to claim 1, characterized in that: In step 1, a high-definition camera is used to collect image information, and a microphone is used to collect audio information.

4. The method for monitoring expansion joints based on audio and video according to claim 1, characterized in that: In step 1, before forming the disease database, the image information and audio information are preprocessed; the preprocessing includes: distortion correction, multi-spectral fusion and defect enhancement for the image information; and noise suppression and feature extraction for the audio information.

5. The method for monitoring expansion joints based on audio and video according to claim 4, characterized in that: After preprocessing, a first defect detection algorithm is also used to output the defect type and defect probability of the expansion joint corresponding to the image information according to the preprocessed image information; the first defect detection algorithm includes an input layer, a dual-branch detection network, a multimodal fusion module and an output layer; The dual-branch detection network includes a YOLOv8-seg network for detecting surface defects and a UNet++ network for detecting internal damage, which are arranged in parallel; an adaptive anchor frame mechanism is provided in the YOLOv8-seg network, and a CBAM module is inserted into the Backbone of the YOLOv8-seg network.

6. The method for monitoring expansion joints based on audio and video according to claim 5, characterized in that: After preprocessing, a second defect detection algorithm is used to output the defect type and defect probability of the expansion joint corresponding to the audio information according to the preprocessed audio information; the second defect detection algorithm includes an input layer, a BiLSTM-Attention network and an output layer.

7. The method for monitoring expansion joints based on audio and video according to claim 6, characterized in that: In step 1, the preprocessing also includes: aligning the image information and the audio information in time and space to form a group of information pairs; after the first defect detection algorithm and the second defect detection algorithm have completed the detection, the detection results of the two are also verified; the verification includes: comparing the detection results of each information pair; if the matching degree of the detection result is higher than the threshold, the verification is passed; if the matching degree of the detection result is lower than the threshold, the corresponding expansion joint of the information pair is re-inspected offline.

8. The method for monitoring expansion joints based on audio and video according to claim 1, characterized in that: In step 2, when classifying the disease levels, the Bayesian network is used for intelligent classification.

9. The method for monitoring expansion joints based on audio and video according to claim 1, characterized in that: In step 3, when analyzing the disease development status of the expansion joint, an LSTM model based on physical information is used for analysis; the LSTM model based on physical information takes the current disease type, disease level and environmental parameters of the expansion joint in the monitoring situation as input, and takes the probability of disease upgrade as output.

10. An expansion joint monitoring device based on audio and video, characterized in that: The invention comprises a high-definition camera, a microphone and a monitoring and control system which establishes communication and control connection with the high-definition camera and the microphone; the monitoring and control system is provided with a controller; the controller is written with an expansion joint monitoring method based on audio and video as described in any one of claims 1 to 9; the high-definition camera and the microphone are arranged at the bridge guardrails on both sides of the expansion joint or are carried on a mobile inspection vehicle, and the installation height of the high-definition camera satisfies that its field of view angle covers the complete expansion joint.

Images