Modality fused bridge crack 4d point cloud micro-motion enhancement method and system

Abstract

The application discloses a kind of modal fusion's bridge crack 4D point cloud microcosmic motion enhancement method and system, method includes: the acquisition and pre-processing of visible light image and 4D point cloud data are completed;Crack two-dimensional image is decomposed using direction controllable pyramid, 2D displacement field, confidence chart and main frequency information are obtained;Based on space-time Transformer network, point cloud sequence in sliding time window is input, after space coding, time self-attention calculation, regression head decoding, 4D point cloud crack microcosmic motion feature is extracted;According to camera calibration parameter, image two-dimensional displacement is inverted into three-dimensional pseudo displacement, combined with image confidence and perspective reliability to construct weight, weighted fusion point cloud original displacement and image inversion displacement, obtain corrected microcosmic displacement;Adjust point cloud and image perspective consistent, combined with crack candidate area positioning crack, dynamically amplify its displacement, reconstruct point cloud sequence, realize bridge crack microcosmic motion three-dimensional visual presentation.Provide support for bridge structure damage assessment.

Description

Technical Field

[0001] This invention belongs to the field of bridge structural health monitoring and intelligent sensing technology, and more specifically, relates to a modal fusion method and system for enhancing the microscopic motion of 4D point cloud of bridge cracks. Background Technology

[0002] Bridge concrete and steel structures are prone to cracking under vehicle loads, environmental loads, and corrosion. The presence of cracks affects the mechanical properties of the load-bearing structure and reduces its load-bearing capacity. When cracks extend to a certain extent, they can affect the overall stability of the bridge and even lead to structural failure, causing serious accidents such as collapse. Therefore, crack monitoring of critical bridge structures, analysis of crack formation mechanisms based on multiple factors, and prediction of crack development trends are essential to ensuring the safety of bridge structures.

[0003] Early-stage cracks in bridge structures are typically very fine, mostly ranging from 0.1 to 1 mm in width, making it difficult to identify their movement using conventional methods. However, surface cracks in bridges exhibit minute movements, such as opening and closing, and rubbing, under the combined influence of various environmental factors and vehicles. These minute movements can characterize the degree and mechanism of crack damage to a certain extent. By determining the movement patterns of structural cracks, data can be provided to support maintenance and repair decisions. Traditional monitoring methods, such as displacement gauges and strain gauges, are costly to deploy and have limited coverage, only sensing one-dimensional spatial movement. Video micro-motion magnification technology can achieve two-dimensional visualization of bridge vibration modes, offering high signal-to-noise ratio and phase sensitivity. However, its results are limited to a two-dimensional plane, lacking depth information and a three-dimensional spatial correspondence, failing to effectively reflect the three-dimensional movement characteristics of cracks and hindering comprehensive support for bridge structural crack damage assessment and degradation trend analysis. Three-dimensional technologies such as stereo vision and lidar have solved the problem of crack depth acquisition to some extent. However, inherent measurement noise exists at the 0.1–1 mm level, while the micro-opening and closing displacements of bridge cracks are often on the same order of magnitude as the noise.

[0004] Therefore, taking the apparent cracks of bridges as the monitoring object, a 4D point cloud micro-motion visualization method based on image-point cloud modal fusion was invented. The method uses the Transformer model to perceive the local micro-motion characteristics in the temporal 4D data of crack point cloud, performs crack identification and extraction through two-dimensional images, and realizes the three-dimensional motion visualization of cracks through image-point cloud modal data fusion by geometric projection constraints, thus filling the gap in the three-dimensional detection and motion analysis technology of bridge cracks. Summary of the Invention

[0005] This invention aims to simultaneously acquire and preprocess visible light images and 4D point cloud data, combine directional controllable pyramids and spatiotemporal Transformer networks to extract image and point cloud features, then correct displacement through cross-modal fusion, finally locate the crack region and dynamically amplify the displacement and reconstruct the point cloud sequence, and present the sub-millimeter-level micro-motion of the crack through 3D visualization, providing support for bridge structural damage assessment.

[0006] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention provides a method and system for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion, comprising:

[0007] S1. Complete the acquisition and preprocessing of visible light images and 4D point cloud data; at the same time, use a microcontroller to synchronously control the lidar and industrial camera to achieve time alignment.

[0008] S2. Use an orientable pyramid to perform multi-scale decomposition on the two-dimensional image of the crack to obtain the 2D displacement field, confidence map and main frequency information of the crack pixels;

[0009] S3. Based on the spatiotemporal Transformer network, the point cloud sequence within the sliding time window is selected as input. First, the geometric features of the point cloud are extracted through spatial encoding. Then, the cross-frame dynamic correlation is calculated through temporal self-attention. Finally, the original microscopic displacement vector of the point cloud is decoded by the regression head to complete the extraction of microscopic motion features of cracks in 4D point cloud.

[0010] S4. Based on the camera calibration parameters, the two-dimensional displacement of the image is inverted into a three-dimensional pseudo displacement. Combining the image confidence and the viewpoint reliability, a fusion weight is constructed. The original displacement of the point cloud and the inverted displacement of the image are weighted and fused to obtain the corrected micro-displacement.

[0011] S5. Adjust the point cloud viewpoint and the image acquisition viewpoint to be consistent, locate the crack region by combining the crack motion enhancement candidate region, and dynamically amplify the displacement of the region only along the local normal; then reconstruct the motion-amplified point cloud sequence based on the initial reference point cloud, and present the crack micro motion through the 3D visualization platform to complete the adaptive motion amplification and visualization of the crack point cloud.

[0012] Furthermore, the specific process of extracting the geometric features of the point cloud through spatial encoding in S3 is as follows:

[0013] The point cloud data in the model input is represented as follows:

[0014] ,

[0015] in, This indicates that the time frame represents the time frame. Centered on, the time window range is A set of continuous point cloud sequences, The parameter representing the span of the time window; Indicates the first The set of point cloud coordinates of the frames, totaling One point;

[0016] The Transformer is used to extract local and global geometric features from the point cloud in each frame; for the first frame... Frame point cloud The spatial encoder outputs each point Spatial feature vectors:

[0017] ,

[0018] in, For point In frame Spatial characteristics, This represents a spatial encoder function that models the spatial relationships between points based on a self-attention mechanism.

[0019] Furthermore, the specific process of calculating cross-frame dynamic correlation through temporal self-attention in S3 is as follows:

[0020] The same spatial point A self-attention mechanism for feature inputs in different time frames along the temporal dimension, spatial points. The features at different time frames are represented as follows:

[0021] ,

[0022] in, Representing a spatial point With the center frame As a benchmark, time window The set of spatial feature vectors for each time frame, each It is a point Geometric features extracted by the spatial encoder in the corresponding time frame; Pick arrive Time frames between;

[0023] Construct the time series feature matrix:

[0024] ,

[0025] in, It is a spatial point In the time window The time series feature matrix within;

[0026] Calculate cross-frame dynamic associations using temporal self-attention:

[0027] ,

[0028] in, , , , , , For learnable parameter matrix, The output of the model is the point cloud micro-motion features after fusing the temporal context. These are the query matrix, key matrix, and value matrix. For feature dimensions.

[0029] Furthermore, the specific process of inverting the two-dimensional displacement of the image into a three-dimensional pseudo-displacement in S4 is as follows:

[0030] Image displacement It is two-dimensional observation data, which is inverted into three-dimensional spatial pseudo-displacement based on the projected Jacobian matrix. ,satisfy:

[0031] ,

[0032] in, The projected Jacobian matrix is ​​used to construct the optimization problem using prior constraints, and then solved using regularized least squares. .

[0033] Furthermore, the specific method for constructing the fusion weights in S4 is as follows:

[0034] Extracted from images , representing pixels exist Motion vector at time step, dominant frequency information Reflecting the main vibration frequencies in this region, a confidence map was obtained through analysis of local signal-to-noise ratio, texture intensity, and phase stability. ;

[0035] First, the temporal displacement of the 2D image is mapped to the 4D point cloud space. The camera intrinsic parameter matrix obtained through camera calibration is K, and the extrinsic parameter matrix is... Points in a point cloud The projection in the image is:

[0036] ,

[0037] in, For perspective projection functions, These are the coordinates of three-dimensional points in a point cloud. It is a rotation matrix. It is a translation vector. It is a point The corresponding two-dimensional pixel coordinates in the image;

[0038] ,

[0039] ,

[0040] in, pixels in an image Confidence level at the point, For point Surface normal, The direction of the camera's line of sight to the point. It is a point Surface normal With the camera at the designated time line of sight The angle between them The larger the value, the more accurate the viewing angle, and the more reliable the image information. It is a fusion weight that combines image confidence and viewpoint reliability.

[0041] Furthermore, the method for calculating the micro-displacement in S4 is as follows:

[0042] Using fusion weights to analyze the micro displacement vectors of point clouds and Perform fusion correction:

[0043] ,

[0044] in, For point cloud microscopic motion displacement based on 2D image fusion correction, It is the original microscopic displacement vector of the point cloud, decoded and output by the spatiotemporal Transformer network. It is a 3D pseudo-displacement vector derived from the image inversion, obtained by inverting the 2D displacement of the image through camera calibration parameters. It is the index of a single point in the point cloud, identifying the specific point; It is a time frame, which identifies the time point corresponding to the displacement.

[0045] Furthermore, the specific process of dynamically amplifying the displacement of the region along the local normal in S4 is as follows:

[0046] To avoid overall deformation distortion, the displacement component is amplified only along the local normal direction within the active region of the crack. The amplified displacement is defined as follows:

[0047] ,

[0048] in, It is a magnified displacement, representing a point in space. In time frame The displacement component magnified along the local normal at the lower edge. displacement after fusion The processing function is used to amplify the displacement components along the local normal. The dynamic amplification factor satisfies:

[0049] ,

[0050] in, Based on magnification factor, Based on signal-to-noise ratio and curvature Adjustments should be made to prevent excessive amplification of high-noise or sharp areas from causing self-crossing. This indicates that the point is located in the crack area.

[0051] Furthermore, the reconstruction of the motion-amplified point cloud sequence based on the initial reference point cloud in S4 specifically includes:

[0052] ,

[0053] in, It is a time frame after motion magnification Lower point The point cloud coordinates represent the position of the point cloud after micro-motion enhancement; It is a point The initial reference point cloud coordinates are used as the basis for displacement calculation.

[0054] As a second aspect of the present invention, a 4D point cloud microscopic motion enhancement system for bridge cracks based on modal fusion is also provided, comprising:

[0055] The data acquisition and preprocessing synchronization unit is used to complete the acquisition and preprocessing of visible light images and 4D point cloud data; at the same time, the microcontroller performs synchronous control of the lidar and industrial camera to achieve time alignment.

[0056] The image feature extraction unit is used to perform multi-scale decomposition of the crack two-dimensional image using an orientation-controllable pyramid to obtain the 2D displacement field, confidence map and main frequency information of the crack pixels;

[0057] The point cloud motion feature extraction unit is used to select the point cloud sequence within the sliding time window as input based on the spatiotemporal Transformer network. First, it extracts the geometric features of the point cloud through spatial encoding, then calculates the cross-frame dynamic correlation through temporal self-attention, and finally decodes and outputs the original microscopic displacement vector of the point cloud by the regression head to complete the extraction of microscopic motion features of cracks in 4D point cloud.

[0058] The cross-modal displacement correction unit is used to invert the two-dimensional displacement of the image into a three-dimensional pseudo displacement based on the camera calibration parameters. It combines the image confidence and the viewpoint reliability to construct a fusion weight, and performs weighted fusion of the original displacement of the point cloud and the inverted displacement of the image to obtain the corrected micro displacement.

[0059] The point cloud magnification and visualization unit is used to adjust the point cloud viewpoint to be consistent with the image acquisition viewpoint, locate the crack region by combining the crack motion enhancement candidate region, and dynamically magnify the displacement of the region only along the local normal; then, based on the initial reference point cloud, the motion magnification point cloud sequence is reconstructed, and the micro motion of the crack is presented through the 3D visualization platform, thus completing the adaptive motion magnification and visualization of the crack point cloud.

[0060] As a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor of any step of the modal fusion method for enhancing the microscopic motion of 4D point cloud of bridge cracks.

[0061] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0062] 1. This invention discloses a modal fusion method for enhancing the microscopic motion of 4D point clouds of bridge cracks. It simultaneously acquires visible light images and 4D point cloud data, achieves temporal alignment through microcontroller synchronization, then performs Mask R-CNN segmentation on the images to obtain crack candidate regions. Preprocessing of the point cloud includes denoising, downsampling, registration, and interpolation, laying the foundation for subsequent multi-source data fusion and feature extraction. This process ensures the spatiotemporal consistency of the image and point cloud data, effectively eliminates background interference and point cloud noise, and preserves the detailed features of the cracks, providing reliable data support for subsequent analysis of the microscopic motion of the cracks.

[0063] 2. This invention provides a modal fusion-based method for enhancing the microscopic motion of 4D point clouds in bridge cracks. It utilizes a direction-controllable pyramid to decompose the 2D crack image at multiple scales, obtaining 2D displacement fields, confidence maps, and dominant frequency information. Based on a spatiotemporal Transformer network, it takes a point cloud sequence within a sliding time window as input, extracts geometric features of the point cloud through spatial encoding, models cross-frame dynamic association through temporal self-attention, and decodes and outputs the original microscopic displacement vector of the point cloud using a regression head. The former achieves accurate extraction of the 2D dynamic features of cracks in the image, while the latter captures the microscopic motion features of the point cloud in 3D space, providing multimodal feature data for crack motion analysis from both 2D and 3D dimensions.

[0064] 3. This invention provides a modal fusion method for enhancing the microscopic motion of 4D point clouds of bridge cracks. It involves inverting two-dimensional displacement of an image into a three-dimensional pseudo-displacement based on camera calibration parameters. A fusion weight is constructed by combining image confidence and viewpoint reliability. The original point cloud displacement and the image-inverted displacement are then weighted and fused to obtain a high-precision microscopic displacement. The crack region is then located, and the displacement is dynamically amplified only along the local normal. The amplified point cloud sequence is reconstructed and presented through 3D visualization. This process achieves multimodal displacement fusion and correction, improving the accuracy of microscopic displacement. Simultaneously, through adaptive motion amplification, sub-millimeter-level microscopic motion of the crack is clearly presented, ultimately providing intuitive and accurate visualization results for bridge structural damage assessment. Attached Figure Description

[0065] Figure 1 This is a flowchart of a method for enhancing the microscopic motion of 4D point cloud of bridge cracks using modal fusion, according to an embodiment of the present invention.

[0066] Figure 2 This is a block diagram illustrating the method implementation of an embodiment of the present invention;

[0067] Figure 3 This is a visible light image of bridge crack damage according to an embodiment of the present invention;

[0068] Figure 4 This is a 3D point cloud of bridge crack damage according to an embodiment of the present invention;

[0069] Figure 5 This is a schematic diagram of the three-dimensional curvature of bridge crack damage, as shown in the embodiment of the present invention.

[0070] Figure 6 This is a system unit diagram of an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0072] Example 1

[0073] Please refer to Figure 1 This embodiment 1 provides a method and system for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion, including:

[0074] S1. Complete the acquisition and preprocessing of visible light images and 4D point cloud data; at the same time, use a microcontroller to synchronously control the lidar and industrial camera to achieve time alignment.

[0075] S2. Use an orientable pyramid to perform multi-scale decomposition on the two-dimensional image of the crack to obtain the 2D displacement field, confidence map and main frequency information of the crack pixels;

[0076] S3. Based on the spatiotemporal Transformer network, the point cloud sequence within the sliding time window is selected as input. First, the geometric features of the point cloud are extracted through spatial encoding. Then, the cross-frame dynamic correlation is calculated through temporal self-attention. Finally, the original microscopic displacement vector of the point cloud is decoded by the regression head to complete the extraction of microscopic motion features of cracks in 4D point cloud.

[0077] S4. Based on the camera calibration parameters, the two-dimensional displacement of the image is inverted into a three-dimensional pseudo displacement. Combining the image confidence and the viewpoint reliability, a fusion weight is constructed. The original displacement of the point cloud and the inverted displacement of the image are weighted and fused to obtain the corrected micro-displacement.

[0078] S5. Adjust the point cloud viewpoint and the image acquisition viewpoint to be consistent, locate the crack region by combining the crack motion enhancement candidate region, and dynamically amplify the displacement of the region only along the local normal; then reconstruct the motion-amplified point cloud sequence based on the initial reference point cloud, and present the crack micro motion through the 3D visualization platform to complete the adaptive motion amplification and visualization of the crack point cloud.

[0079] Please refer to Figure 2 The method in this embodiment 1 first detects and extracts bridge cracks using visible light image sensing, removes background interference, and extracts information such as two-dimensional phase, dominant frequency, and sub-pixel displacement of the cracks based on a direction-controllable pyramid. Secondly, it synchronously acquires continuous point clouds of the cracks in the time dimension using a high-precision lidar and a visible light camera, performing preprocessing such as denoising and downsampling rigid body registration. Joint calibration is performed based on the intrinsic and extrinsic parameters of the visible light camera and lidar, followed by spatiotemporal calibration. A spatiotemporal Transformer network is constructed to extract the microscopic motion features of the 4D point cloud, fuses the two-dimensional image, and enhances the microscopic motion of the crack point cloud based on frequency, phase, geometric projection, and spatial orientation constraints, achieving visualization of the crack's three-dimensional motion and supporting bridge structural damage diagnosis and assessment.

[0080] Furthermore, this embodiment 1 provides a detailed explanation of the above steps.

[0081] (1) Data acquisition and preprocessing synchronization

[0082] See image acquisition and preprocessing; please refer to Figure 3 Industrial cameras were used to continuously acquire optically detectable images of cracks in bridge steel and concrete structures. Based on Shannon sampling theorem, the acquisition frame rate was greater than 20Hz, forming a crack image sequence containing temporal information. Mask R-CNN was then used to perform high-precision pixel-level segmentation of the target crack region, generating candidate regions for crack motion enhancement.

[0083] 4D point cloud data acquisition and preprocessing; please refer to Figure 4 A high-precision lidar sensor was selected to scan the bridge cracks. The scanning frequency was consistent with that of the industrial camera to acquire crack point cloud data containing time information. Gaussian filtering was used to denoise the point cloud data, removing drift points and isolated points caused by the instrument, surrounding environment, and the characteristics of the scanned target itself. Gaussian filtering helps to identify gaps and endpoints, and while filtering, it also preserves the original data structure relatively well. Please refer to [reference needed]. Figure 5 Curvature downsampling is performed on the point cloud. Since the area containing cracks has significant curvature, curvature downsampling can largely preserve the point cloud's detailed description of the cracks. Time-series point clouds are registered based on Iterative Closest Point (ICP). All point cloud frames are first rigidly aligned to the reference frame to reduce overall rigid body drift, preserve local deformation, and avoid interference from overall motion on the micro-motion enhancement results. Interpolation processing is then performed on the registered point clouds to ensure spatial consistency and guarantee that the number of points in each frame is equal.

[0084] Image and point cloud fusion calibration and spatiotemporal alignment are performed. An STM32 microcontroller is used for external trigger signal control to synchronously control the LiDAR and industrial camera, ensuring consistent sampling frequency and time, thus achieving time alignment. Displacement amplitude calibration is performed based on LiDAR-camera joint calibration.

[0085] (2) Image feature extraction

[0086] In the scenario of bridge crack detection, a direction-controllable pyramid is used to analyze two-dimensional crack images. Multi-scale decomposition needs to be integrated with subsequent point cloud processing procedures. During decomposition, the image is broken down into different scales. ( ) and direction ( Subband components These sub-bands correspond to the texture and structural features of cracks at different spatial scales, complementing the geometric details preserved in the point cloud preprocessing, and satisfying... This ensures that the original image information can be completely reconstructed after decomposition.

[0087] For adjacent frame subbands and Calculate the phase difference ,in for The sub-band phase at time. After eliminating winding through phase unwinding, combined with scale... corresponding spatial frequency The sub-band displacement is obtained. This displacement reflects the local movement of the crack in the two-dimensional plane, providing a basis for subsequent fusion with the three-dimensional displacement of the point cloud.

[0088] The 2D displacement field is obtained by fusing the displacements of each sub-band. Here, the sub-band component amplitude is used as the weight because high-energy sub-bands correspond to significant crack regions, forming a spatial correlation with the crack regions of interest in point cloud feature extraction. The generated displacement field will be used for subsequent camera parameter inversion, transforming it into a 3D pseudo-displacement, and participating in the fusion and correction of point cloud displacement.

[0089] Through displacement field gradient Construct a confidence graph Regions with small gradients correspond to reliable displacements, and their confidence values, along with the reliability of the point cloud perspective, constitute the fusion weights, thereby improving the accuracy of cross-modal data fusion. Indicates the function in Rate of change of direction Indicates the function in Rate of change of direction. Time-frequency analysis of the displacement field is performed to extract the dominant frequency. ( (For Fourier transform), this frequency corresponds to the main period of crack movement, which echoes the dynamic correlation analysis in the point cloud time series features in the time dimension, providing a frequency basis for subsequent dynamic amplification of the crack region.

[0090] (3) Point cloud motion feature extraction

[0091] Microscopic motion feature extraction of cracks in 4D point clouds based on Transformer architecture and spatiotemporal attention mechanism: This network is used to extract the microscopic motion features of each point in the time dimension from the point cloud sequence within a sliding time window, and outputs the displacement vector of each point between consecutive frames.

[0092] The point cloud data in the model input is represented as follows:

[0093] ,

[0094] in, This indicates that the time frame represents the time frame. Centered on, the time window range is A set of continuous point cloud sequences, The parameter representing the span of the time window; Indicates the first The set of point cloud coordinates of the frames, totaling One point;

[0095] Construct a spatial encoder; use Transformer to extract local and global geometric features from the point cloud of each frame; for the first frame... Frame point cloud The spatial encoder outputs each point Spatial feature vectors:

[0096] ,

[0097] in, For point In frame Spatial characteristics, This represents a spatial encoder function that models the spatial relationships between points based on a self-attention mechanism.

[0098] Construct a temporal attention module; group the same spatial points A self-attention mechanism for feature inputs in different time frames along the temporal dimension, spatial points. The features at different time frames are represented as follows:

[0099] ,

[0100] in, Representing a spatial point With the center frame As a benchmark, time window The set of spatial feature vectors for each time frame, each It is a point Geometric features extracted by the spatial encoder in the corresponding time frame; Pick arrive Time frames between;

[0101] Construct the time series feature matrix:

[0102] ,

[0103] in, It is a spatial point In the time window The time series feature matrix within;

[0104] Calculate cross-frame dynamic associations using temporal self-attention:

[0105] ,

[0106] in, , , , , , For learnable parameter matrix, The output of the model is the point cloud micro-motion features after fusing the temporal context. These are the query matrix, key matrix, and value matrix. For feature dimensions.

[0107] Finally, the micro-displacement is decoded using the temporal features corresponding to the center frame t as the output:

[0108] ,

[0109] in, For point At any moment The micro displacement vector, These are the parameters for the transfactor regression head.

[0110] (4) Transmodal displacement correction

[0111] Extracted from images , representing pixels exist Motion vector at time step, dominant frequency information To reflect the main vibration frequencies in this region, the temporal displacement of the two-dimensional image is first mapped to a 4D point cloud space. The camera intrinsic parameter matrix obtained through camera calibration is K, and the extrinsic parameter matrix is... Points in a point cloud The projection in the image is:

[0112] ,

[0113] in, For perspective projection functions, These are the coordinates of three-dimensional points in a point cloud. It is a rotation matrix. It is a translation vector. It is a point The corresponding two-dimensional pixel coordinates in the image;

[0114] Image displacement It is two-dimensional observation data, which is inverted into three-dimensional spatial pseudo-displacement based on the projected Jacobian matrix. ,satisfy:

[0115] ,

[0116] in, The projected Jacobian matrix is ​​used to construct the optimization problem using prior constraints, and then solved using regularized least squares. The specific process is as follows:

[0117] First, define the objective: to solve for the two-dimensional displacement field of each pixel in the bridge crack image. It reflects the positional changes of pixels between adjacent frames;

[0118] First, we construct an optimization problem, the core of which is to balance "data fitting" and "prior constraints". The data fitting part is based on the gray-level consistency of the image: the gray level of the same physical point in adjacent frames should be approximately unchanged. By calculating the residual between the spatial gradient of the image and the temporal gray-level difference, the estimated displacement field is made to match the image observation information.

[0119] Then, prior constraints are added as regularization terms: the first is the smoothness constraint, because the microscopic motion of the crack is locally continuous and the displacement of adjacent pixels will not change abruptly, thereby suppressing displacement field fluctuations caused by noise; the second is the crack region constraint, which uses the segmented crack region to keep the displacement of the non-crack region at 0, avoiding background interference; the third is the rigid body motion constraint, because the non-crack bridge structure is a rigid body and its displacement must conform to the rigid body transformation law of translation plus rotation.

[0120] Next, we use regularized least squares to solve the problem: the displacement field is transformed into pixel-level variables, and the optimization problem is transformed into a system of linear equations. Due to the high dimensionality of the variables, an iterative method is adopted. First, the displacement is initialized to 0, and then the displacement is gradually updated along the negative gradient direction of the objective function until the displacement difference between two iterations is sufficiently small, yielding the final result. .

[0121] Confidence maps were obtained through local signal-to-noise ratio, texture intensity, and phase stability analysis. Based on image confidence Construct fusion weights using the cosine angle of the viewpoint (the angle between the point normal and the line of sight):

[0122] ,

[0123] ,

[0124] in, pixels in an image Confidence level at the point, For point Surface normal, The direction of the camera's line of sight to the point. It is a point Surface normal With the camera at the designated time line of sight The angle between them The larger the value, the more accurate the viewing angle, and the more reliable the image information. It is a fusion weight that combines image confidence and viewpoint reliability.

[0125] Using fusion weights to analyze the micro displacement vectors of point clouds and Perform fusion correction:

[0126] ,

[0127] in, For point cloud microscopic motion displacement based on 2D image fusion correction, It is the original microscopic displacement vector of the point cloud, decoded and output by the spatiotemporal Transformer network. It is a 3D pseudo-displacement vector derived from the image inversion, obtained by inverting the 2D displacement of the image through camera calibration parameters. It is the index of a single point in the point cloud, identifying the specific point; It is a time frame, which identifies the time point corresponding to the displacement.

[0128] (5) Point cloud magnification visualization

[0129] The viewpoint of the crack point cloud is transformed to keep the frontal viewpoint of the point cloud consistent with the viewpoint of the 2D image acquisition. The crack enhancement candidate region is matched with the crack plane mapping based on the 2D image segmentation to distinguish the crack region from the non-crack region in the point cloud.

[0130] To avoid overall deformation distortion, the displacement component is amplified only along the local normal direction within the active region of the crack. The amplified displacement is defined as follows:

[0131] ,

[0132] in, It is a magnified displacement, representing a point in space. In time frame The displacement component magnified along the local normal at the lower edge. displacement after fusion The processing function is used to amplify the displacement components along the local normal. The dynamic amplification factor satisfies:

[0133] ,

[0134] in, Based on magnification factor, Based on signal-to-noise ratio and curvature Adjustments should be made to prevent excessive amplification of high-noise or sharp areas from causing self-crossing. This indicates that the point is located in the crack area.

[0135] The specific reconstructed motion-magnified point cloud sequence based on the initial reference point cloud is as follows:

[0136] ,

[0137] in, It is a time frame after motion magnification Lower point The point cloud coordinates represent the position of the point cloud after micro-motion enhancement; It is a point The initial reference point cloud coordinates are used as the basis for displacement calculation.

[0138] The amplified point cloud sequence is then imported into a 3D visualization platform. The platform reads the 3D coordinates, time frame information, and corresponding displacement attributes of each point in the point cloud data, and loads the point cloud models of different frames in chronological order. During the loading process, the platform automatically calibrates the coordinate system of each frame of the point cloud based on the spatial position of the initial reference point cloud, ensuring that the spatial position of the crack area is continuous and consistent across different time frames, and avoiding misalignment of the motion due to coordinate system deviation.

[0139] For crack areas, the platform will highlight the crack location in the 3D model through differentiated display (such as specific color marking and outline drawing) based on the previously located crack candidate area information. At the same time, it will associate the displacement data of each point to transform the microscopic movement of the point cloud into a visualized dynamic effect. Through frame-by-frame switching or smooth transition, it will present the tiny displacement of the crack over time, allowing the observer to intuitively see the crack's expansion, displacement and other movement states in three-dimensional space.

[0140] Furthermore, the platform supports interactive operation, allowing observers to adjust the viewpoint and zoom the model to view the microscopic motion details of cracks from different angles. Users can also select specific time intervals to focus on the motion characteristics of cracks within that period, or export dynamic sequence files for subsequent analysis. Ultimately, through the dynamic presentation and interactive functions of the point cloud model, the platform provides a visual representation of the adaptive motion magnification of the crack point cloud, offering an intuitive spatial dynamic reference for the microscopic motion analysis of cracks in bridge structures.

[0141] This embodiment 1 has direct application prospects in the field of bridge structural health monitoring, enabling long-term dynamic tracking of the microscopic movement of cracks in operating bridges. By accurately capturing sub-millimeter displacement changes in cracks and presenting them in 3D visualization, it can help monitoring personnel identify crack propagation trends in advance, avoiding structural damage deterioration caused by the hidden movement of cracks. It provides data support for daily bridge maintenance and early warning of bridge defects, and is especially suitable for key monitoring targets such as long-span bridges and old bridges.

[0142] This method can also be extended to engineering testing and evaluation scenarios. For example, in bridge load tests, it can observe the microscopic response of cracks under different loads in real time, helping to verify the bearing capacity of the bridge structure. At the same time, its multimodal data fusion and adaptive motion amplification technology can also provide a reference for crack monitoring of similar concrete buildings, tunnel linings and other structures, promoting the upgrading of structural health monitoring from macroscopic appearance inspection to microscopic dynamic analysis, and improving the accuracy and timeliness of engineering structural safety assessment.

[0143] Example 2

[0144] Please refer to Figure 6 This embodiment 2 provides a modal fusion-based 4D point cloud microscopic motion enhancement system for bridge cracks, comprising:

[0145] The data acquisition and preprocessing synchronization unit is used to complete the acquisition and preprocessing of visible light images and 4D point cloud data; at the same time, the microcontroller performs synchronous control of the lidar and industrial camera to achieve time alignment.

[0146] The image feature extraction unit is used to perform multi-scale decomposition of the crack two-dimensional image using an orientation-controllable pyramid to obtain the 2D displacement field, confidence map and main frequency information of the crack pixels;

[0147] The point cloud motion feature extraction unit is used to select the point cloud sequence within the sliding time window as input based on the spatiotemporal Transformer network. First, it extracts the geometric features of the point cloud through spatial encoding, then calculates the cross-frame dynamic correlation through temporal self-attention, and finally decodes and outputs the original microscopic displacement vector of the point cloud by the regression head to complete the extraction of microscopic motion features of cracks in 4D point cloud.

[0148] The cross-modal displacement correction unit is used to invert the two-dimensional displacement of the image into a three-dimensional pseudo displacement based on the camera calibration parameters. It combines the image confidence and the viewpoint reliability to construct a fusion weight, and performs weighted fusion of the original displacement of the point cloud and the inverted displacement of the image to obtain the corrected micro displacement.

[0149] The point cloud magnification and visualization unit is used to adjust the point cloud viewpoint to be consistent with the image acquisition viewpoint, locate the crack region by combining the crack motion enhancement candidate region, and dynamically magnify the displacement of the region only along the local normal; then, based on the initial reference point cloud, the motion magnification point cloud sequence is reconstructed, and the micro motion of the crack is presented through the 3D visualization platform, thus completing the adaptive motion magnification and visualization of the crack point cloud.

[0150] Example 3

[0151] This embodiment 3 also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement any step of a modal fusion method for enhancing the microscopic motion of 4D point clouds of bridge cracks.

[0152] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0153] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0154] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion, characterized in that, include: S1. Complete the acquisition and preprocessing of visible light images and 4D point cloud data; at the same time, use a microcontroller to synchronously control the lidar and industrial camera to achieve time alignment. S2. Use an orientable pyramid to perform multi-scale decomposition on the two-dimensional image of the crack to obtain the 2D displacement field, confidence map and main frequency information of the crack pixels; S3. Based on the spatiotemporal Transformer network, the point cloud sequence within the sliding time window is selected as input. First, the geometric features of the point cloud are extracted through spatial encoding. Then, the cross-frame dynamic correlation is calculated through temporal self-attention. Finally, the original microscopic displacement vector of the point cloud is decoded by the regression head to complete the extraction of microscopic motion features of cracks in 4D point cloud. S4. Based on the camera calibration parameters, the two-dimensional displacement of the image is inverted into a three-dimensional pseudo displacement. Combining the image confidence and the viewpoint reliability, a fusion weight is constructed. The original displacement of the point cloud and the inverted displacement of the image are weighted and fused to obtain the corrected micro-displacement. S5. Adjust the point cloud viewpoint and the image acquisition viewpoint to be consistent, locate the crack region by combining the crack motion enhancement candidate region, and dynamically amplify the displacement of the crack region along the local normal only; then reconstruct the motion-amplified point cloud sequence based on the initial reference point cloud, and present the micro-motion of the crack through the 3D visualization platform to complete the adaptive motion amplification and visualization of the crack point cloud. The reliability of the viewpoint is the surface normal. With the camera at the designated time line of sight The absolute value of the cosine of the angle between them; The method for calculating the micro-displacement is as follows: Using fusion weights to analyze the micro displacement vectors of point clouds and Perform fusion correction: ， in, For point cloud microscopic motion displacement based on 2D image fusion correction, It is the original microscopic displacement vector of the point cloud, decoded and output by the spatiotemporal Transformer network. It is a 3D pseudo-displacement vector derived from the image inversion, obtained by inverting the 2D displacement of the image through camera calibration parameters. It is the index of a single point in the point cloud, identifying the specific point; It is a time frame, identifying the time point corresponding to the microscopic motion displacement; the... It is a fusion weight that combines image confidence and viewpoint reliability; The specific process of dynamically amplifying the displacement along the local normal in this region is as follows: To avoid overall deformation distortion, the displacement component is amplified only along the local normal direction within the active region of the crack. The amplified displacement is defined as follows: ， in, It is a magnified displacement, representing a point in space. The displacement component magnified along the local normal at time frame t. displacement after fusion The processing function is used to amplify the displacement components along the local normal. The dynamic amplification factor satisfies: ， in, Based on magnification factor, Based on signal-to-noise ratio and curvature Adjustments should be made to prevent excessive amplification of high-noise or sharp areas from causing self-crossing. This indicates that the point is located in the crack area.

2. The method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion according to claim 1, characterized in that, The specific process of extracting the geometric features of the point cloud through spatial encoding in S3 is as follows: The point cloud data in the model input is represented as follows: ， in, This indicates that the time frame represents the time frame. Centered on, the time window range is A set of continuous point cloud sequences, The parameter representing the span of the time window; Indicates the first The set of point cloud coordinates of the frames, totaling One point; The Transformer is used to extract local and global geometric features from each frame of point cloud. For the Frame point cloud The spatial encoder outputs each point Spatial feature vectors: ， in, For point In frame Spatial characteristics, This represents a spatial encoder function that models the spatial relationships between points based on a self-attention mechanism.

3. The method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion according to claim 2, characterized in that, The specific process of calculating cross-frame dynamic correlation through temporal self-attention in S3 is as follows: The same spatial point A self-attention mechanism for feature inputs in different time frames along the temporal dimension, spatial points. The features at different time frames are represented as follows: ， in, Representing a spatial point With the center frame As a benchmark, time window The set of spatial feature vectors for each time frame, each It is a point Spatial features extracted by the spatial encoder in the corresponding time frame Pick arrive Time frames between; Construct the time series feature matrix: ， in, It is a spatial point In the time window The time series feature matrix within; Calculate cross-frame dynamic associations using temporal self-attention: ， in, , , , , , For learnable parameter matrix, The output of the model is the point cloud micro-motion features after fusing the temporal context. These are the query matrix, key matrix, and value matrix. For feature dimensions.

4. The method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion according to claim 1, characterized in that, The specific process of inverting the two-dimensional displacement of the image into a three-dimensional pseudo-displacement in S4 is as follows: Image displacement It is two-dimensional observation data. It is a point The corresponding two-dimensional pixel coordinates in the image are inverted into a three-dimensional pseudo-displacement based on the projection Jacobian matrix. ,satisfy: ， in, The projected Jacobian matrix is ​​used to construct the optimization problem using prior constraints, and then solved using regularized least squares. .

5. The method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion according to claim 1, characterized in that, The specific method for constructing the fusion weights in S4 is as follows: Extracted from images , representing pixels exist Motion vector at time step, dominant frequency information Reflecting the main vibration frequencies in this region, a confidence map was obtained through analysis of local signal-to-noise ratio, texture intensity, and phase stability. ; First, the temporal displacement of the 2D image is mapped to the 4D point cloud space. The camera intrinsic parameter matrix obtained through camera calibration is K, and the extrinsic parameter matrix is... Points in a point cloud The projection in the image is: ， in, For perspective projection functions, These are the coordinates of three-dimensional points in a point cloud. It is a rotation matrix. It is a translation vector. It is a point The corresponding two-dimensional pixel coordinates in the image; ， ， in, pixels in an image Confidence level at the point, For point Surface normal, The direction of the camera's line of sight to the point. It is a point Surface normal With the camera at the designated time line of sight The angle between them The larger the value, the more accurate the viewing angle, and the more reliable the image information. It is a fusion weight that combines image confidence and viewpoint reliability.

6. The method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion according to claim 5, characterized in that, The point cloud sequence reconstructed from the initial reference point cloud after motion amplification in S5 is specifically as follows: ， in, It is a time frame after motion magnification Lower point The point cloud coordinates represent the position of the point cloud after micro-motion enhancement; It is a point The initial reference point cloud coordinates are used as the basis for displacement calculation.

7. A modal fusion-based 4D point cloud microscopic motion enhancement system for bridge cracks, characterized in that, include: The data acquisition and preprocessing synchronization unit is used to complete the acquisition and preprocessing of visible light images and 4D point cloud data; Simultaneously, time alignment is achieved by synchronously controlling the LiDAR and industrial camera using a microcontroller; The image feature extraction unit is used to perform multi-scale decomposition of the crack two-dimensional image using an orientation-controllable pyramid to obtain the 2D displacement field, confidence map and main frequency information of the crack pixels; The point cloud motion feature extraction unit is used to select the point cloud sequence within the sliding time window as input based on the spatiotemporal Transformer network. First, it extracts the geometric features of the point cloud through spatial encoding, then calculates the cross-frame dynamic correlation through temporal self-attention, and finally decodes and outputs the original microscopic displacement vector of the point cloud by the regression head to complete the extraction of microscopic motion features of cracks in 4D point cloud. The cross-modal displacement correction unit is used to invert the two-dimensional displacement of the image into a three-dimensional pseudo displacement based on the camera calibration parameters. It combines the image confidence and the viewpoint reliability to construct a fusion weight, and performs weighted fusion of the original displacement of the point cloud and the inverted displacement of the image to obtain the corrected micro displacement. The point cloud magnification and visualization unit is used to adjust the point cloud viewpoint to be consistent with the image acquisition viewpoint, locate the crack region by combining the crack motion enhancement candidate region, and dynamically magnify the displacement of the crack region only along the local normal; then, based on the initial reference point cloud, the motion magnification point cloud sequence is reconstructed, and the micro motion of the crack is presented through the 3D visualization platform, thus completing the adaptive motion magnification and visualization of the crack point cloud. The reliability of the viewpoint is the surface normal. With the camera at the designated time line of sight The absolute value of the cosine of the angle between them; The method for calculating the micro-displacement is as follows: Using fusion weights to analyze the micro displacement vectors of point clouds and Perform fusion correction: ， in, For point cloud microscopic motion displacement based on 2D image fusion correction, It is the original microscopic displacement vector of the point cloud, decoded and output by the spatiotemporal Transformer network. It is a 3D pseudo-displacement vector derived from the image inversion, obtained by inverting the 2D displacement of the image through camera calibration parameters. It is the index of a single point in the point cloud, identifying the specific point; It is a time frame, identifying the time point corresponding to the microscopic motion displacement; the... It is a fusion weight that combines image confidence and viewpoint reliability; The specific process of dynamically amplifying the displacement along the local normal in this region is as follows: To avoid overall deformation distortion, the displacement component is amplified only along the local normal direction within the active region of the crack. The amplified displacement is defined as follows: ， in, It is a magnified displacement, representing a point in space. The displacement component magnified along the local normal at time frame t. displacement after fusion The processing function is used to amplify the displacement components along the local normal. The dynamic amplification factor satisfies: ， in, Based on magnification factor, Based on signal-to-noise ratio and curvature Adjustments should be made to prevent excessive amplification of high-noise or sharp areas from causing self-crossing. This indicates that the point is located in the crack area.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-6: a method for enhancing the microscopic motion of 4D point clouds of bridge cracks through modal fusion.

Images