Crack detection method and system based on digital twinning and multiple intelligent perception

Abstract

The invention discloses a crack detection method and system based on digital twinning and multiple intelligent perception, and the method comprises the steps: S100, installing a track, an X-shaped support frame, an IMU unit and an industrial camera, and collecting the image data of a detected bridge; s200, calling an industrial camera to realize training of crack detection; s300, performing comprehensive scanning and three-dimensional model modeling on the tested bridge to obtain initial point cloud data of the tested bridge; s400, setting an inspection path and speed, and dynamically detecting the bridge crack; s500, when a crack is detected, adjusting the position of the industrial camera for continuous detection; s600, performing micro motion enhancement by using phase information between adjacent images, and converting the acquired crack continuous image into a video; s700, splitting the video frame by frame into enhanced images, and performing three-dimensional model reconstruction on cracks; s800, obtaining crack data at each moment; and S900, detecting environmental data at the crack in real time, and carrying out early warning on bridge safety. And rapid and comprehensive data acquisition and crack monitoring are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of bridge detection, and more specifically, relates to a crack detection method and system based on digital twins and multiple intelligent perceptions. Background Art

[0002] The concrete structure of a bridge will crack under the influence of vehicle loads, environmental loads and erosion. The presence of cracks will affect the mechanical properties of the load-bearing structure and reduce its bearing capacity. When the cracks expand to a certain extent, they will affect the overall stability of the bridge and even cause structural failure, resulting in serious accidents such as collapse. Therefore, it is very necessary to detect cracks in important bridge structures, analyze the crack generation mechanism based on multiple factors, and predict the development trend to ensure the safety of bridge structures.

[0003] At present, crack detection mainly includes contact and non-contact methods. The contact method detects cracks by installing sensors such as piezoelectric ceramics and optical fibers. This type of method often requires advance installation during the construction period. Later additions will damage the structure, and the detection range is limited, and full coverage cannot be guaranteed. The non-contact method mainly relies on visual means. The commonly used two-dimensional method collects the surface image of the structure and uses computer vision algorithms for detection and measurement. This method can only obtain the two-dimensional morphological characteristics of the cracks, but cannot obtain the crack depth characteristics. Three-dimensional technologies such as stereo vision and lidar have solved the problem of crack depth acquisition to a certain extent. However, both two-dimensional and three-dimensional detection methods are static detection, and the detection results only represent the crack status at the current moment. The apparent cracks of the bridge will have tiny movements such as opening and closing, rubbing, etc. under the combined effects of various environments and vehicles. These tiny movements can characterize the damage degree and damage mechanism of the cracks to a certain extent, judge the movement form of the structural cracks, and assist in providing corresponding maintenance and repair decisions. However, these microscopic movements cannot be effectively obtained using conventional means and methods.

[0004] Therefore, a crack detection method and system based on digital twins and multiple intelligent perceptions takes the apparent cracks of bridges as the detection object, integrates multiple environmental perception data such as temperature and humidity with the three-dimensional micro-motion model of cracks, and conducts crack occurrence mechanism and development prediction analysis through digital twin analysis to realize the detection of apparent cracks in bridges. Summary of the invention

[0005] In response to the above defects or improvement needs of the prior art, the present invention provides a crack detection method and system based on digital twins and multiple intelligent perceptions. The crack detection model is used to quickly detect whether there are cracks on the surface. The digital image in the vertical direction of the crack plane is continuously collected at the appropriate position by adjusting the camera posture. The image phase information is used to enhance small movements. The enhanced dual-camera crack images are reconstructed frame by frame to obtain a crack-enhanced three-dimensional point cloud. A movable scanning and freely adjustable binocular vision hardware system is composed of two industrial cameras and a support frame that can be translated, telescoped and rotated. Reasonable path planning is performed through the structural three-dimensional model to achieve fast and comprehensive data collection and crack monitoring.

[0006] To achieve the above objectives, according to a first aspect of an embodiment of the present invention, a crack detection method based on digital twins and multiple intelligent perceptions is provided, comprising the following steps: S100, a track is laid around the bridge to be tested, an X-shaped support frame is installed on the track, and an IMU unit and an industrial camera are set at the end of the X-shaped support frame; S200, calling industrial cameras to train intelligent recognition models; S300, a mobile industrial camera, scans the entire outer surface of the bridge under test, builds a three-dimensional model of the bridge under test based on the scanned initial image information, and obtains its complete initial point cloud data; S400, based on the three-dimensional model of the structure under test, with the goal of covering the entire surface of the bridge under test, set the inspection path and speed to achieve dynamic detection; S500, when any industrial camera detects a crack, the position information of the two industrial cameras is obtained through the IMU unit, and the positions thereof are adjusted to continuously detect the crack position; S600, continuously collecting images of the cracks, converting the collected continuous images of the cracks into videos, and performing micro-motion enhancement using phase information between adjacent images; S700, decomposing the converted video frame by frame into enhanced images, reconstructing a three-dimensional model of the cracks of the measured bridge, and obtaining dynamic and continuous enhanced point cloud data thereof; S800, based on the dynamic and continuous enhanced point cloud data of the tested bridge, obtain crack data including the width, length, depth change law and change speed at each moment; S900, real-time detection of environmental data including humidity, temperature, load and pH value at the crack, and building a database in combination with crack parameters to predict the crack data at the next moment.

[0007] Further, in step S100, the X-shaped support frame includes a support rotation axis arranged on the track, and a telescopic rod arranged on the support rotation axis; The support rotation axis moves on the track, and the telescopic rods are used in pairs, cross-arranged on the support rotation axis, and rotate on the support rotation axis; Each of the telescopic rods is also provided at the end thereof with a rotating rod, which rotates around the end of the telescopic rod, and the IMU unit and the industrial camera are both provided at one end of the rotating rod away from the telescopic rod; The support rotation shaft drives the X-shaped support frame to move on the track, and the angle between the two telescopic rods is adjusted by rotating the support rotation shaft, and the telescopic rods are telescoped along their radial directions; The rotating rod drives the industrial camera to rotate to adjust its angle, and the two industrial cameras form a binocular vision system.

[0008] Furthermore, step S300 specifically includes the following steps: S310, establishing a spatial coordinate system based on the space where the measured bridge is located, and calibrating the initial position and posture information of the IMU unit in the spatial coordinate system; S320, realize multi-degree-of-freedom movement including translation, extension and rotation through the X-shaped support frame, so that the industrial camera scans the entire outer surface of the bridge to be measured to obtain its initial image information, and obtains the real-time position information of the industrial camera through the IMU unit; S330, combining the initial image information of the measured bridge and the real-time position information of the industrial camera, obtaining an overall shape image of the measured bridge, and performing three-dimensional modeling according to the overall shape image; S340. According to the three-dimensional model, initial point cloud data of the measured bridge in the spatial coordinate system is obtained.

[0009] Furthermore, step S400 specifically includes the following steps: S410, obtaining the size information of the bridge under test according to the initial point cloud data, determining the best installation position of the track in combination with the size and shape of the telescopic rod and the rotating rod, and installing the track; S420, installing a telescopic rod, a rotating rod, an IMU unit, and an industrial camera on the track through a bracket rotation axis, and recalibrating the initial positions of the two industrial cameras in the spatial coordinate system; S430, using the track as the inspection path, and driving the industrial camera to move along the track through the servo motor on the rotating shaft of the bracket; S440. Determine the inspection speed based on the shooting frequency of the industrial camera to ensure that the overlap rate of two adjacent photos is higher than 50%.

[0010] Furthermore, step S500 specifically includes the following steps: S510: When any one of the industrial cameras detects a crack, position information of the two industrial cameras and position information of the crack are obtained; S520, analyzing the positional relationship between the two industrial cameras, and determining the target detection positions of the two industrial cameras with the position of the crack as the center; S530, the two industrial cameras are moved by the support rotation axis and the telescopic rod so that both reach the target detection position; S540, adjust the angles of the two industrial cameras by rotating the rod so that they are aimed at the cracks and continuously detect them; S550, two industrial cameras each collect a crack image once, and use SFM technology to perform three-dimensional reconstruction on the part within the overlapping detection range of the two industrial cameras to obtain its point cloud data; S560, performing point cloud plane fitting by least square method to obtain the angle and distance between the plane where the crack is located and the industrial camera; S570. Adjust the positions and angles of the two industrial cameras again so that their imaging planes are perpendicular to the plane where the cracks are located, to ensure that the three-dimensional shape of the cracks is correctly and completely constructed when three-dimensional reconstruction is performed based on the collected image data.

[0011] Further, in step S550, the target detection positions of the two industrial cameras are ensured to be within the maximum overlap area of ​​the detection ranges of the two industrial cameras to the maximum extent that the cracks are detected; In step S570, it is ensured that the distance between the imaging planes of the two industrial cameras and the crack plane is adjusted to ensure that a crack with a width of 0.1 mm has no less than 2 pixels in the image.

[0012] Furthermore, step S600 specifically includes the following steps: S610, after determining and adjusting the postures of the two industrial cameras, conduct short-term detection of the cracks and continuously collect crack images; S620, converting the continuous crack images collected by each industrial camera into videos respectively, performing micro-motion enhancement using phase information between adjacent images, obtaining the microscopic motion state of the crack through micro-motion enhancement, and visualizing the micro-motion that is invisible to the naked eye; S630, performing motion enhancement based on image phase information to amplify motion features while avoiding enhancing noise; S640, motion enhancement based on image phase information is performed by performing spatial domain decomposition, temporal domain filtering, phase denoising, motion amplification, and then video reconstruction to obtain a motion enhanced video.

[0013] Furthermore, step S700 specifically includes the following steps: S710, obtaining position information of two industrial cameras, and decomposing the converted video into enhanced images frame by frame; S720, remove image noise and reduce noise points through technologies including Gaussian smoothing and median filtering, improve contrast and highlight crack features through adaptive histogram equalization; S730, performing image correction through steps including camera calibration, calculating reprojection error, applying reprojection, and correcting image distortion; S740, then calculating the disparity map by a semi-global matching method, obtaining the disparity map of the enhanced image and then performing three-dimensional reconstruction; S750, performing three-dimensional reconstruction on the enhanced images frame by frame to obtain a continuous enhanced three-dimensional point cloud of the microscopic motion of the crack, and performing key point selection and feature point matching on the enhanced point cloud between two adjacent frames of enhanced images through three-dimensional feature point matching; S760, removing the spatial points that cannot be matched, completing point cloud downsampling and point cloud standardization processing, so that the number of points in the point cloud between the previous and next frames is consistent, and each point can obtain a corresponding matching point in the next frame; S770. Obtain the dynamic change and spatial movement process of each point, present the spatial change of each point in the time series, and intuitively show the spatial movement form of the crack.

[0014] Furthermore, step S900 specifically includes: S910, combining crack data and environmental data to build a database, and using deep learning to build a time series feature analysis model for a specific bridge structure; S920, using environmental data, time and bridge structure as model inputs and crack data as model outputs, constructing a crack generation cause analysis and development trend prediction model based on time series through a recurrent neural network, and obtaining a crack microscopic motion state model based on multiple environmental perceptions through training; S930. Monitor the movement microscopic state of the crack by acquiring the current environmental data in real time. When the crack data output by the model exceeds the safety threshold, perform continuous detection and timely warning.

[0015] According to a second aspect of an embodiment of the present invention, a crack detection system based on digital twins and multiple intelligent perceptions is provided, comprising: The installation module is used to lay out a track around the bridge to be tested, install an X-shaped support frame on the track, and set an IMU unit and an industrial camera at the end of the X-shaped support frame. The industrial camera is used to continuously collect the surface image data of the bridge to be tested, and the IMU unit records the position and posture information of the industrial camera; The training module is used to call the industrial camera to perform real-time crack detection on the bridge under test. When cracks are detected, they are marked with rectangular frames to achieve crack detection training; The modeling module is used to move the industrial camera so that its scanning range covers the entire outer surface of the bridge to be tested, and to perform a comprehensive scan on it. A three-dimensional model of the bridge to be tested is built based on the scanned initial image information to obtain its complete initial point cloud data; The planning module is used to set the inspection path and speed based on the three-dimensional model of the structure to be tested, with the goal of covering the entire surface of the bridge to be tested, and to achieve dynamic detection of bridge cracks at a fixed frequency; An adjustment module is used to obtain the position information of the two industrial cameras through the IMU unit when any industrial camera detects a crack, and adjust the positions of the two industrial cameras to continuously detect the crack position; The conversion module is used to continuously collect images of cracks, use the phase information between adjacent images to perform micro-motion enhancement, and convert the continuous crack images collected by two industrial cameras into videos; The enhancement module is used to decompose the converted video frame by frame into enhanced images, and combine the posture information of the two industrial cameras to reconstruct the three-dimensional model of the cracks of the measured bridge to obtain its dynamic and continuous enhanced point cloud data; The data acquisition module is used to obtain crack data including width, length, depth change rules and change speed at each moment based on the dynamic and continuous enhanced point cloud data of the tested bridge; The early warning module is used to detect environmental data including humidity, temperature, load and pH value at the crack in real time, and build a database in combination with crack parameters. Based on the current environmental data, it predicts the crack data at the next moment and issues an early warning for bridge safety.

[0016] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art: 1. The crack detection method of the present invention uses a crack detection model to quickly detect whether there are cracks on the surface, adjusts the camera posture to continuously collect digital images in the vertical direction of the crack plane at an appropriate position, uses image phase information to enhance tiny movements, and performs three-dimensional reconstruction of the enhanced dual-camera crack images frame by frame to obtain crack enhanced three-dimensional point clouds. A movable scanning and freely adjustable binocular vision hardware system is composed of two industrial cameras and a support frame that can be translated, retracted, and rotated, and reasonable path planning is performed through a structural three-dimensional model to achieve rapid and comprehensive data collection and crack monitoring.

[0017] 2. The crack detection method of the present invention performs key point selection and feature point matching on the enhanced point cloud of each frame to achieve point cloud downsampling and construct a standardized point cloud so that the number of points in the point cloud between the previous and next frames remains consistent. Each point can obtain a corresponding matching point in the next frame, and the spatial coordinate change process of each point is obtained. The dynamically enhanced standardized crack three-dimensional point cloud can realize the visualization of the crack micro-motion and make an intuitive judgment on the movement form of the crack.

[0018] 3. The crack detection method of the present invention integrates the environmental perception data obtained by multiple sensors with the crack morphology data and three-dimensional point cloud data to construct a time series crack generation cause analysis and development trend prediction model, thereby realizing crack monitoring and development trend prediction and early warning through multiple environmental perceptions.

[0019] 4. The crack detection method of the present invention integrates the local and the whole, the internal structure and the external environment, and builds a digital twin platform for multi-sensing crack monitoring of bridge structures with an overall component model, a local crack micro-motion model, an environmental model and a crack development prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of the steps of a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 2 A schematic diagram of a process of a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the position of an X-shaped support frame of a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 4 This is a schematic diagram of specific steps of step S300 in a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 5 This is a schematic diagram of specific steps of step S400 in a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 6 This is a schematic diagram of specific steps of step S500 in a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 7 This is a schematic diagram of specific steps of step S600 in a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Figure 8 This is a schematic diagram of specific steps of step S700 in a crack detection method based on digital twins and multiple intelligent perceptions according to an embodiment of the present invention; Fig. 9This is a schematic diagram of specific steps of step S900 in a crack detection method based on digital twins and multiple intelligent perceptions in an embodiment of the present invention.

[0021] In all the drawings, the same reference numerals represent the same technical features, specifically: 1-track, 2-bracket rotation axis, 3-telescopic rod, 4-rotating rod, 5-IMU unit, 6-industrial camera. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0023] Example 1 like Figure 1 , 2 As shown, an embodiment of the present invention provides a crack detection method based on digital twins and multiple intelligent perceptions, which specifically includes the following steps: S100, a track 1 is laid around the bridge to be tested, an X-shaped support frame is installed on the track 1, and an IMU unit 5 and an industrial camera 6 are arranged at the end of the X-shaped support frame, the industrial camera 6 is used to continuously collect the surface image data of the bridge to be tested, and the IMU unit 5 records the position information of the industrial camera 6; S200, input a crack image database, mark the cracks, train a crack detection model through an artificial neural network, obtain a recognition model, and call an industrial camera 6 to perform crack detection; S300, moving the industrial camera 6 so that its scanning range covers the entire outer surface of the bridge to be measured, performing a comprehensive scan on the bridge, and building a three-dimensional model of the bridge to be measured based on the scanned initial image information to obtain its complete initial point cloud data; S400, based on the three-dimensional model of the structure to be tested, with the goal of covering the entire surface of the bridge to be tested, setting the inspection path and speed, and realizing dynamic detection of bridge cracks at a fixed frequency; S500, during the detection process, when any industrial camera 6 detects a crack, the position information of the two industrial cameras 6 is obtained through the IMU unit 5, and the positions of the two industrial cameras 6 are adjusted to continuously detect the crack position; S600, continuously collecting images of the crack, converting the continuous images of the crack collected by the two industrial cameras 6 into videos, and performing micro-motion enhancement using phase information between adjacent images; S700, decomposing the converted video frame by frame into enhanced images, combining the position information of the two industrial cameras 6, reconstructing a three-dimensional model of the cracks of the measured bridge, and obtaining its dynamic and continuous enhanced point cloud data; S800, based on the dynamic and continuous enhanced point cloud data of the tested bridge, obtain crack data including the width, length, depth change law and change speed at each moment; S900, real-time detection of environmental data including humidity, temperature, load and pH value at the crack, and build a database based on the crack parameters. According to the current environmental data, the crack data at the next moment is predicted to provide early warning for bridge safety.

[0024] like Figure 3 As shown, in step S100, the X-shaped support frame includes a support rotation axis 2 arranged on the track 1, and a telescopic rod 3 arranged on the support rotation axis 2. The support rotation axis 2 moves on the track 1, and the telescopic rods 3 are used in pairs, cross-arranged on the support rotation axis 2, and rotate on the support rotation axis 2. A rotating rod 4 is also provided at the end of each telescopic rod 3, and the rotating rod 4 rotates around the end of the telescopic rod 3. The IMU unit 5 and the industrial camera 6 are both arranged at one end of the rotating rod 4 away from the telescopic rod 3. The support rotation axis 2 drives the X-shaped support frame to move on the track 1, and the angle between the two telescopic rods 3 is adjusted by rotating the support rotation axis 2, and the telescopic rod 3 is extended and retracted along its radial direction. The rotating rod 4 drives the industrial camera 6 to rotate to adjust its angle. The two industrial cameras 6 form a binocular vision system.

[0025] In step S200, each industrial camera 6 performs detection separately to improve the detection speed and coverage. The two industrial cameras 6 form a binocular vision system, and the real-time detection range coverage overlap rate of the two is maintained at no less than 30% to ensure the subsequent modeling accuracy.

[0026] like Figure 4 As shown, step S300 specifically includes the following steps: S310, establishing a spatial coordinate system based on the space where the measured bridge is located, and calibrating the initial position and posture information of the IMU unit 5 in the spatial coordinate system; S320, realizing multi-degree-of-freedom movement including translation, extension and rotation through an X-shaped support frame, so that the industrial camera 6 scans the entire outer surface of the bridge to be measured to obtain its initial image information, and obtains the real-time position and posture information of the industrial camera 6 through the IMU unit 5; S330, combining the initial image information of the measured bridge and the real-time position information of the industrial camera 6, obtaining an overall shape image of the measured bridge, and performing three-dimensional modeling according to the overall shape image; S340. According to the three-dimensional model, initial point cloud data of the measured bridge in the spatial coordinate system is obtained.

[0027] In step S310, the initial position information of the IMU unit 5 is the same as the initial position information of the industrial camera 6. The initial position information of the industrial camera 6 is , the initial posture information is Through the static calibration procedure, it is ensured that the output of the IMU unit 5 is zero without external interference, that is, ,in are the acceleration outputs of the IMU unit 5 in the x, y, and z directions respectively. The initial pose information of the industrial camera 6 is calibrated using the least squares method, specifically: , in, is the location information of the industrial camera, is the measurement error of the IMU unit, is the rotation matrix corresponding to the initial posture information.

[0028] In step S320, considering the composite transformation of translation and rotation, the transformed point is represented by homogeneous coordinates. for: , in, is the transformation matrix, is the point before transformation; The transformation matrix Specifically: , in, is the rotation matrix, is the translation vector.

[0029] Use B-spline curve or Bezier curve for path planning to ensure smooth path and avoid sudden changes. Specifically: , in, is a cubic B-spline curve, is the cubic B-spline basis function, For the control point.

[0030] In step S330, the Gauss-Newton iteration method is used to optimize the image stitching, and the optimization target is: , in, is the transformation parameter between the two images, , There are two images, is the transformation matrix, For the control point.

[0031] The Poisson reconstruction algorithm is used for 3D modeling, and the reconstructed surface is obtained by solving the Poisson equation: , in, is the normal vector, is the implicit function to be found, is the gradient operator.

[0032] In step S340, the iterative closest point algorithm is used to perform point cloud registration, and the objective function of each iteration is: , in, is the source point cloud, The closest point in the source point cloud to the target point cloud.

[0033] Use bilateral filter to denoise point cloud, specifically: , in, is the point after filtering, and are distance weight and color weight respectively, is the normalization factor, is the neighborhood point of the source point cloud.

[0034] like Figure 5 As shown, step S400 specifically includes the following steps: S410, obtaining the size information of the bridge under test according to the initial point cloud data, determining the best installation position of the track 1 in combination with the size and shape of the telescopic rod 3 and the rotating rod 4, and installing the track 1; S420, installing the telescopic rod 3, the rotating rod 4, the IMU unit 5 and the industrial camera 6 on the track 1 through the bracket rotating shaft 2, and recalibrating the initial positions of the two industrial cameras 6 in the space coordinate system; S430, using track 1 as the inspection path, and driving the industrial camera 6 to move along track 1 through the servo motor on the bracket rotating shaft 2; S440, determining the inspection speed according to the shooting frequency of the industrial camera 6, and ensuring that the overlap rate of two adjacent photos is higher than 50%.

[0035] In step S410, the optimal installation position of the track 1 is to minimize the total length of the track 1 while ensuring that the industrial camera 6 can scan the entire surface of the bridge to be tested, so as to reduce the scanning time of the industrial camera 6 and ensure that it can quickly detect cracks.

[0036] like Figure 6 As shown, step S500 specifically includes the following steps: S510, any one of the industrial cameras 6 detects a crack, and obtains the position information of the two industrial cameras 6 and the position information of the crack; S520, analyzing the positional relationship between the two industrial cameras 6, and determining the target detection positions of the two industrial cameras 6 with the position of the crack as the center; S530, the bracket rotating shaft 2 and the telescopic rod 3 move in coordination so that both industrial cameras 6 reach the target detection position; S540, adjusting the angles of the two industrial cameras 6 by rotating the rod 4 so that they are aimed at the cracks and continuously detecting them; S550, the two industrial cameras 6 collect crack images once each, and perform three-dimensional reconstruction on the part in the overlapping area of ​​the detection range of the two industrial cameras 6 through the SFM technology to obtain its point cloud data; S560, performing point cloud plane fitting by least square method to obtain the angle and distance between the plane where the crack is located and the industrial camera 6; S570, adjusting the positions and angles of the two industrial cameras 6 again so that their imaging planes are perpendicular to the plane where the cracks are located, to ensure that when three-dimensional reconstruction is performed based on the collected image data, the three-dimensional shape of the cracks is correctly and completely constructed.

[0037] In step S550 , the target detection positions of the two industrial cameras 6 are ensured to be within the maximum overlapping area of ​​the detection ranges of the two industrial cameras 6 to satisfy the maximum degree of crack detection.

[0038] In step S570, it is ensured that the distance between the imaging planes of the two industrial cameras 6 and the crack plane is adjusted to ensure that a crack with a width of 0.1 mm has no less than 2 pixels in the image.

[0039] like Figure 7 As shown, step S600 specifically includes the following steps: S610, after determining and adjusting the postures of the two industrial cameras 6, short-term detection of the cracks is performed to continuously collect crack images; S620, converting the continuous crack images collected by each industrial camera 6 into videos, performing micro-motion enhancement using phase information between adjacent images, obtaining the microscopic motion state of the crack through micro-motion enhancement, and visualizing the micro-motion that is invisible to the naked eye; S630, performing motion enhancement based on image phase information to amplify motion features while avoiding enhancing noise; S640, motion enhancement based on image phase information is performed by performing spatial domain decomposition, temporal domain filtering, phase denoising, motion amplification, and then video reconstruction to obtain a motion enhanced video.

[0040] In step S700, the dynamic three-dimensional point cloud is used to quickly judge the movement forms of the crack, such as opening and closing, dislocation, and horizontal rubbing. On the other hand, the quantitative analysis of the time series spatial position change of a certain point or a certain point can realize the quantitative analysis of the change of the crack in the three-dimensional space and time, and judge the change law and change speed of the width, length, and depth of the crack at a certain moment.

[0041] like Figure 8 As shown, step S700 specifically includes the following steps: S710, obtaining the position information of the two industrial cameras 6, and decomposing the converted videos into enhanced images frame by frame; S720, remove image noise and reduce noise points through technologies including Gaussian smoothing and median filtering, improve contrast and highlight crack features through adaptive histogram equalization; S730, performing image correction through steps including camera calibration, calculating reprojection error, applying reprojection, and correcting image distortion; S740, then calculating the disparity map by a semi-global matching method, obtaining the disparity map of the enhanced image and then performing three-dimensional reconstruction; S750, performing three-dimensional reconstruction on the enhanced images frame by frame to obtain a continuous enhanced three-dimensional point cloud of the microscopic motion of the crack, and performing key point selection and feature point matching on the enhanced point cloud between two adjacent frames of enhanced images through three-dimensional feature point matching; S760, removing the spatial points that cannot be matched, completing point cloud downsampling and point cloud standardization processing, so that the number of points in the point cloud between the previous and next frames is consistent, and each point can obtain a corresponding matching point in the next frame; S770. Obtain the dynamic change and spatial movement process of each point, present the spatial change of each point in the time series, and intuitively show the spatial movement form of the crack.

[0042] In step S720, a Gaussian filter is used to smooth the image to reduce noise. for: , in, is the convolution kernel of Gaussian filter, is the original image, is the convolution operation.

[0043] In step S740, the three-dimensional coordinates of each pixel are calculated by this method, and together they form a three-dimensional point cloud in the maximum overlapping area of ​​the detection range of the two industrial cameras 6. , specifically: , in, is the cost function.

[0044] Disparity Map The optimization is performed by minimizing the global energy function: , in, Pixel Neighborhood of is the image domain, is the weight of the smoothing term, is the smoothing term.

[0045] In the three-dimensional reconstruction based on the disparity map and camera parameters, the three-dimensional point cloud is: , in, and is the focal length, and are the principal point coordinates, , is the baseline distance.

[0046] Use SIFT or SURF feature point matching algorithm to find the corresponding points between adjacent frames. The matching point pairs are ,in: , in, For the The descriptor of feature points, is the descriptor of the candidate point.

[0047] like Fig. 9As shown, step S900 specifically includes: S910, combining crack data and environmental data to build a database, and using deep learning to build a time series feature analysis model for a specific bridge structure; S920, using environmental data, time and bridge structure as model inputs and crack data as model outputs, constructing a crack generation cause analysis and development trend prediction model based on time series through a recurrent neural network, and obtaining a crack microscopic motion state model based on multiple environmental perceptions through training; S930. Monitor the movement microscopic state of the crack by acquiring the current environmental data in real time. When the crack data output by the model exceeds the safety threshold, perform continuous detection and timely warning.

[0048] After step S900, it also includes: determining the position of the monitored cracks in the bridge structure based on the structural three-dimensional model obtained by the preliminary scan and the spatial position of the camera during crack monitoring, constructing the relative position relationship between the local crack enhancement point cloud model and the bridge component, fixing the crack enhancement point cloud in the three-dimensional point cloud model of the bridge structure, and realizing the dynamic monitoring and generation, development and evolution trend model of the cracks in the bridge structure. In addition, a digital twin platform for multi-sensing crack monitoring of bridge structures is built, including the overall component model, the local crack micro-motion model, the environmental model, and the crack development prediction model.

[0049] Example 2 The embodiment of the present invention provides a crack detection system based on digital twin and multiple intelligent perceptions, including: An installation module is used to lay out a track 1 around the bridge to be tested, install an X-shaped support frame on the track 1, and set an IMU unit 5 and an industrial camera 6 at the end of the X-shaped support frame, use the industrial camera 6 to continuously collect the surface image data of the bridge to be tested, and the IMU unit 5 records the position and posture information of the industrial camera 6; A training module is used to call the industrial camera 6 to perform real-time crack detection on the bridge under test, and annotate cracks with rectangular frames when cracks are detected, so as to realize crack detection training; A modeling module is used to move the industrial camera 6 so that its scanning range covers the entire outer surface of the bridge to be measured, and to perform a comprehensive scan on the bridge, and to perform a three-dimensional modeling of the bridge to be measured based on the scanned initial image information, and to obtain its complete initial point cloud data; The planning module is used to set the inspection path and speed based on the three-dimensional model of the structure to be tested, with the goal of covering the entire surface of the bridge to be tested, and to achieve dynamic detection of bridge cracks at a fixed frequency; An adjustment module, used for obtaining the position information of the two industrial cameras 6 through the IMU unit 5 when any one of the industrial cameras 6 detects a crack, and adjusting the positions of the two industrial cameras 6 to continuously detect the crack position; A conversion module, used for continuously collecting images of cracks, using phase information between adjacent images to perform micro-motion enhancement, and converting continuous images of cracks collected by two industrial cameras 6 into videos; An enhancement module is used to decompose the converted video into enhanced images frame by frame, and to reconstruct a three-dimensional model of the cracks of the measured bridge in combination with the position information of the two industrial cameras 6 to obtain its dynamic and continuous enhanced point cloud data; The data acquisition module is used to obtain crack data including width, length, depth change rules and change speed at each moment based on the dynamic and continuous enhanced point cloud data of the tested bridge; The early warning module is used to detect environmental data including humidity, temperature, load and pH value at the crack in real time, and build a database in combination with crack parameters. Based on the current environmental data, it predicts the crack data at the next moment and issues an early warning for bridge safety.

[0050] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A crack detection method based on digital twins and multiple intelligent perceptions, characterized in that: The following steps are involved: S100, laying out a track (1) around the bridge to be tested, installing an X-shaped support frame on the track (1), and setting an IMU unit (5) and an industrial camera (6) at the end of the X-shaped support frame; S200, calling the industrial camera (6) to train the intelligent recognition model; S300, a mobile industrial camera (6) is used to scan the entire outer surface of the bridge under test, and a three-dimensional model is built for the bridge under test based on the scanned initial image information to obtain its complete initial point cloud data; S400, based on the three-dimensional model of the structure under test, with the goal of covering the entire surface of the bridge under test, set the inspection path and speed to achieve dynamic detection; S500, when any one of the industrial cameras (6) detects a crack, the position information of the two industrial cameras (6) is obtained through the IMU unit (5), and their positions are adjusted to continuously detect the crack position; S600, continuously collecting images of the cracks, converting the collected continuous images of the cracks into videos, and performing micro-motion enhancement using phase information between adjacent images; S700, decomposing the converted video frame by frame into enhanced images, reconstructing a three-dimensional model of the cracks of the measured bridge, and obtaining dynamic and continuous enhanced point cloud data thereof; S800, based on the dynamic and continuous enhanced point cloud data of the tested bridge, obtain crack data including the width, length, depth change law and change speed at each moment; S900, real-time detection of environmental data including humidity, temperature, load and pH value at the crack, and building a database in combination with crack parameters to predict the crack data at the next moment.

2. A crack detection method based on digital twin and multiple intelligent perception according to claim 1, characterized in that: In step S100, the X-shaped support frame comprises a support rotation axis (2) arranged on the track (1), and a telescopic rod (3) arranged on the support rotation axis (2); The support rotation axis (2) moves on the track (1), and the telescopic rods (3) are used in pairs, cross-arranged on the support rotation axis (2), and rotate on the support rotation axis (2); A rotating rod (4) is also provided at the end of each telescopic rod (3), and the rotating rod (4) rotates around the end of the telescopic rod (3); the IMU unit (5) and the industrial camera (6) are both provided at an end of the rotating rod (4) away from the telescopic rod (3); The support rotating shaft (2) drives the X-shaped support frame to move on the track (1), and the angle between the two telescopic rods (3) is adjusted by rotating the support rotating shaft (2), and the telescopic rods (3) are telescoped along their radial direction; The rotating rod (4) drives the industrial camera (6) to rotate to adjust its angle, and the two industrial cameras (6) form a binocular vision system.

3. A crack detection method based on digital twin and multiple intelligent perception according to claim 2, characterized in that: In step S300, the following steps are specifically included: S310, establishing a spatial coordinate system based on the space where the measured bridge is located, and calibrating the initial position and posture information of the IMU unit (5) in the spatial coordinate system; S320, achieving multi-degree-of-freedom movement including translation, extension and rotation through an X-shaped support frame, so that the industrial camera (6) scans the entire outer surface of the bridge to be measured to obtain its initial image information, and obtains real-time position and posture information of the industrial camera (6) through the IMU unit (5); S330, combining the initial image information of the bridge under test and the real-time position information of the industrial camera (6), obtaining an overall shape image of the bridge under test, and performing three-dimensional modeling based on the overall shape image; S340. According to the three-dimensional model, initial point cloud data of the measured bridge in the spatial coordinate system is obtained.

4. A crack detection method based on digital twin and multiple intelligent perception according to claim 2 or 3, characterized in that: In step S400, the following steps are specifically included: S410, obtaining the size information of the bridge under test according to the initial point cloud data, determining the best installation position of the track (1) in combination with the size and shape of the telescopic rod (3) and the rotating rod (4), and installing the track (1); S420, installing the telescopic rod (3), the rotating rod (4), the IMU unit (5) and the industrial camera (6) on the track (1) through the bracket rotating shaft (2), and recalibrating the initial positions of the two industrial cameras (6) in the spatial coordinate system; S430, using the track (1) as an inspection path, and driving the industrial camera (6) to move along the track (1) via a servo motor on the support rotating shaft (2); S440, determining the inspection speed based on the shooting frequency of the industrial camera (6), and ensuring that the overlap rate of two adjacent photos is greater than 50%.

5. A crack detection method based on digital twin and multiple intelligent perception according to claim 2 or 3, characterized in that: In step S500, the following steps are specifically included: S510, any one of the industrial cameras (6) detects a crack, and obtains position information of the two industrial cameras (6) and position information of the crack; S520, analyzing the positional relationship between the two industrial cameras (6), and determining the target detection positions of the two industrial cameras (6) with the position of the crack as the center; S530, by means of the coordinated movement of the support rotating shaft (2) and the telescopic rod (3), the two industrial cameras (6) are both brought to the target detection position; S540, adjusting the angles of the two industrial cameras (6) by rotating the rod (4) so ​​that the two industrial cameras (6) are exactly aimed at the cracks, and continuous detection is carried out thereon; S550, the two industrial cameras (6) each collect a crack image once, and three-dimensionally reconstruct the portion within the overlapping detection range of the two industrial cameras (6) using the SFM technology to obtain point cloud data thereof; S560, performing point cloud plane fitting by least square method to obtain the angle and distance between the plane where the crack is located and the industrial camera (6); S570, adjusting the positions and angles of the two industrial cameras (6) again so that their imaging planes are perpendicular to the plane where the crack is located, to ensure that when three-dimensional reconstruction is performed based on the collected image data, the three-dimensional shape of the crack is correctly and completely constructed.

6. A crack detection method based on digital twin and multiple intelligent perception according to claim 5, characterized in that: In step S550, the target detection positions of the two industrial cameras (6) are ensured to be within the maximum overlap area of ​​the detection ranges of the two industrial cameras (6) to the maximum extent that the cracks are detected; In step S570, it is ensured that the distance between the imaging planes of the two industrial cameras (6) and the crack plane is adjusted to ensure that a crack with a width of 0.1 mm has no less than 2 pixels in the image.

7. A crack detection method based on digital twin and multiple intelligent perception according to claim 2 or 3, characterized in that: In step S600, the following steps are specifically included: S610, after determining and adjusting the postures of the two industrial cameras (6), short-term detection of the cracks is performed to continuously collect crack images; S620, converting the continuous crack images collected by each industrial camera (6) into videos, performing micro-motion enhancement using phase information between adjacent images, obtaining the microscopic motion state of the crack through micro-motion enhancement, and visualizing the micro-motion that is invisible to the naked eye; S630, performing motion enhancement based on image phase information to amplify motion features while avoiding enhancing noise; S640, motion enhancement based on image phase information is performed by performing spatial domain decomposition, temporal domain filtering, phase denoising, motion amplification, and then video reconstruction to obtain a motion enhanced video.

8. A crack detection method based on digital twin and multiple intelligent perception according to claim 2 or 3, characterized in that: In step S700, the following steps are specifically included: S710, obtaining the position information of the two industrial cameras (6), and decomposing the converted video into enhanced images frame by frame; S720, remove image noise and reduce noise points through technologies including Gaussian smoothing and median filtering, improve contrast and highlight crack features through adaptive histogram equalization; S730, performing image correction through steps including camera calibration, calculating reprojection error, applying reprojection, and correcting image distortion; S740, then calculating the disparity map by a semi-global matching method, obtaining the disparity map of the enhanced image and then performing three-dimensional reconstruction; S750, performing three-dimensional reconstruction on the enhanced images frame by frame to obtain a continuous enhanced three-dimensional point cloud of the microscopic motion of the crack, and performing key point selection and feature point matching on the enhanced point cloud between two adjacent frames of enhanced images through three-dimensional feature point matching; S760, removing the spatial points that cannot be matched, completing point cloud downsampling and point cloud standardization processing, so that the number of points in the point cloud between the previous and next frames is consistent, and each point can obtain a corresponding matching point in the next frame; S770. Obtain the dynamic change and spatial movement process of each point, present the spatial change of each point in the time series, and intuitively show the spatial movement form of the crack.

9. A crack detection method based on digital twin and multiple intelligent perception according to claim 2 or 3, characterized in that: Step S900 specifically includes: S910, combining crack data and environmental data to build a database, and using deep learning to build a time series feature analysis model for a specific bridge structure; S920, using environmental data, time and bridge structure as model inputs and crack data as model outputs, constructing a crack generation cause analysis and development trend prediction model based on time series through a recurrent neural network, and obtaining a crack microscopic motion state model based on multiple environmental perceptions through training; S930. Monitor the movement microscopic state of the crack by acquiring the current environmental data in real time. When the crack data output by the model exceeds the safety threshold, perform continuous detection and timely warning.

10. A crack detection system based on digital twins and multiple intelligent perceptions, used to implement a crack detection method based on digital twins and multiple intelligent perceptions as described in any one of claims 1 to 9, characterized in that: include: An installation module is used to lay out a track (1) around the bridge to be tested, install an X-shaped support frame on the track (1), and set an IMU unit (5) and an industrial camera (6) at the end of the X-shaped support frame, use the industrial camera (6) to continuously collect surface image data of the bridge to be tested, and use the IMU unit (5) to record the position and posture information of the industrial camera (6); A training module is used to call an industrial camera (6) to perform real-time crack detection on the bridge under test, and annotate cracks with rectangular frames when cracks are detected, thereby realizing crack detection training; A modeling module is used to move the industrial camera (6) so that its scanning range covers the entire outer surface of the bridge to be measured, and to perform a comprehensive scan on the bridge, and to perform a three-dimensional modeling of the bridge to be measured based on the scanned initial image information, and to obtain its complete initial point cloud data; The planning module is used to set the inspection path and speed based on the three-dimensional model of the structure to be tested, with the goal of covering the entire surface of the bridge to be tested, and to achieve dynamic detection of bridge cracks at a fixed frequency; An adjustment module, used for obtaining the position information of the two industrial cameras (6) through the IMU unit (5) when any one of the industrial cameras (6) detects a crack, and adjusting the positions of the two industrial cameras (6) to continuously detect the position of the crack; A conversion module, used for continuously collecting images of cracks, using phase information between adjacent images to perform micro-motion enhancement, and converting continuous images of cracks collected by two industrial cameras (6) into videos; An enhancement module is used to decompose the converted video frame by frame into enhanced images, and to reconstruct a three-dimensional model of the cracks of the bridge under test by combining the position information of the two industrial cameras (6) to obtain its dynamic and continuous enhanced point cloud data; The data acquisition module is used to obtain crack data including width, length, depth change rules and change speed at each moment based on the dynamic and continuous enhanced point cloud data of the tested bridge; The early warning module is used to detect environmental data including humidity, temperature, load and pH value at the crack in real time, and build a database in combination with crack parameters. Based on the current environmental data, it predicts the crack data at the next moment and issues an early warning for bridge safety.

Images