Bridge damage monitoring system and method based on image processing

Abstract

The application discloses a bridge damage monitoring system and method based on image processing, which extracts the structural contour of a two-dimensional image through a deep convolutional neural network, and compares it with the rendered edge generated by perspective projection of a preset three-dimensional information model, and uses the chamfer distance to construct a closed-loop feedback mechanism to iteratively optimize the camera pose, so as to eliminate the pose projection drift caused by GPS rejection or weak texture in a complex environment, thereby accurately mapping the two-dimensional microscopic damage to a three-dimensional macroscopic coordinate system. In this way, the reliability of damage physical size quantification and global positioning is significantly improved, the problems of damage monitoring positioning difficulty and low mapping accuracy in complex inspection conditions are effectively solved, and efficient seamless integration of detection data and digital modeling is realized.

Description

Technical Field

[0001] This application relates to the field of intelligent monitoring, and more specifically, to a bridge damage monitoring system and method based on image processing. Background Technology

[0002] As a critical node in the transportation infrastructure network, the safety and durability of bridge structures directly affect the operation of the national economy and the safety of public transportation. During long-term service, bridge structures inevitably suffer microscopic damage such as cracks and spalling due to environmental erosion, material aging, and the continuous action of dynamic and static loads. If these initial damages are not monitored and qualitatively and quantitatively assessed in a timely manner, they can easily evolve into serious structural safety hazards, or even lead to catastrophic accidents. Therefore, constructing an intelligent and high-precision bridge damage monitoring system is of great engineering significance for achieving proactive preventative maintenance of transportation infrastructure.

[0003] Existing technical solutions mostly employ drones equipped with high-resolution cameras for non-contact image acquisition, combined with computer vision algorithms for defect identification. However, in actual bridge inspection scenarios, especially in complex environments such as the bridge web, the inner wall of box girder, or the base of piers, drones are highly susceptible to GPS signal loss or positioning failures due to multipath effects caused by the physical shielding of satellite signals by structures. Simultaneously, bridge component surfaces often exhibit texture deficiencies or highly similar features, rendering traditional visual positioning algorithms ineffective. Under these conditions lacking reliable initial pose constraints, existing technologies face a severe problem of pose projection drift across dimensions. Due to the inability to achieve sub-pixel-level precise correction of camera pose, the system struggles to accurately and perfectly map the two-dimensional micro-crack images captured by the drone into the macroscopic three-dimensional bridge digital twin model's physical coordinate system. This lack of precision in spatial dimension mapping prevents the system from accurately reconstructing the true physical size of the damage and its absolute position in global space, severely impacting the reliability of bridge health monitoring and the efficiency of digital operation and maintenance.

[0004] Therefore, an optimized image processing-based method for bridge damage monitoring is needed. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a bridge damage monitoring system and method based on image processing.

[0006] According to one aspect of this application, a bridge damage monitoring method based on image processing is provided, comprising: S1, the raw sensor data stream containing the pixel matrix, positioning latitude and longitude and inertial attitude angles collected by the UAV is parsed and the parameters are stripped to obtain the two-dimensional image matrix and the initial pose matrix, and the pre-calibrated camera intrinsic parameter matrix is ​​obtained at the same time. S2, based on a pre-trained dual-branch deep convolutional neural network, extracts high semantic information from a two-dimensional image matrix through multi-branch image extraction to obtain crack segmentation masks and structural edge masks; S3, based on the camera intrinsic parameter matrix and the initial pose matrix, constructs a perspective projection transformation, performs dimensionality reduction rendering projection on the outer envelope geometric line group of the preset 3D bridge information model to obtain the rendering edge feature lines that reflect the expected bridge skeleton lines under the current view. S4, construct a cost function using the chamfer distance between the structural edge mask and the rendered edge feature line, and iteratively optimize and correct the initial pose matrix to obtain the optimized pose matrix; S5, based on the optimized pose matrix and camera intrinsic parameter matrix, performs dimensionality reduction and inverse 3D physical mapping of each feature pixel in the crack segmentation mask to obtain a 3D damage feature vector containing crack physical size and global spatial localization.

[0007] According to another aspect of this application, an image processing-based bridge damage monitoring system is provided, comprising: The data stream parsing and parameter stripping module is used to parse and strip parameters from the raw sensor data stream containing pixel matrices, latitude and longitude coordinates, and inertial attitude angles collected by the UAV to obtain a two-dimensional image matrix and an initial pose matrix, while also acquiring a pre-calibrated camera intrinsic parameter matrix. The image dual-branch high semantic information extraction module is used to extract multi-branch high semantic information from a two-dimensional image matrix based on a pre-trained dual-branch deep convolutional neural network to obtain crack segmentation masks and structural edge masks. The 3D model perspective projection dimensionality reduction rendering module is used to construct perspective projection transformation based on the camera intrinsic parameter matrix and the initial pose matrix, and to perform dimensionality reduction rendering projection on the outer envelope geometric line group of the preset 3D bridge information model to obtain the rendering edge feature lines that reflect the expected bridge skeleton lines under the current view. The iterative optimization and correction module is used to construct a cost function based on the chamfer distance between the structural edge mask and the rendered edge feature line, and to iteratively optimize and correct the initial pose matrix to obtain the optimized pose matrix. The dimension reduction and inverse 3D physical mapping module is used to perform dimension reduction and inverse 3D physical mapping on each feature pixel in the crack segmentation mask based on the optimized pose matrix and camera intrinsic parameter matrix to obtain a 3D damage feature vector containing crack physical size and global spatial location.

[0008] Compared with existing technologies, this application provides a bridge damage monitoring system and method based on image processing. It extracts the structural contours of two-dimensional images using a deep convolutional neural network and compares them interactively with the rendered edges generated by perspective projection of a pre-set three-dimensional information model. A closed-loop feedback mechanism is constructed using chamfer distance to iteratively optimize the camera pose, eliminating pose projection drift caused by GPS rejection or weak textures in complex environments. This allows for high-precision mapping of two-dimensional microscopic damage to a three-dimensional macroscopic coordinate system. This approach significantly improves the reliability of damage physical size quantification and global positioning, effectively solving the problems of difficult damage monitoring and positioning and low mapping accuracy under complex inspection conditions, and achieving efficient and seamless integration of detection data and digital modeling. Attached Figure Description

[0009] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0010] Figure 1 This is a flowchart of an image processing-based bridge damage monitoring method according to an embodiment of this application; Figure 2 This is a schematic diagram of the data flow of the image processing-based bridge damage monitoring method according to an embodiment of this application; Figure 3 This is a flowchart of step S2 in the image processing-based bridge damage monitoring method according to an embodiment of this application; Figure 4 This is a block diagram of an image processing-based bridge damage monitoring system according to an embodiment of this application. Detailed Implementation

[0011] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0012] As indicated in this application and claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" are not specifically singular and may include plural forms. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements.

[0013] While this application makes various references to certain modules of the systems according to embodiments of this application, any number of different modules can be used and run on user terminals and / or servers. The modules described are merely illustrative, and different aspects of the systems and methods may use different modules.

[0014] Flowcharts are used in this application to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, various steps can be processed in reverse order or simultaneously as needed. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0015] The technical solution of this application proposes a bridge damage monitoring method based on image processing. Figure 1 This is a flowchart of an image processing-based bridge damage monitoring method according to an embodiment of this application. Figure 2 This is a system architecture diagram of an image processing-based bridge damage monitoring method according to an embodiment of this application. Figure 1 and Figure 2 As shown, the image processing-based bridge damage monitoring method according to an embodiment of this application includes the following steps: S1, parsing and parameter stripping the raw sensor data stream containing pixel matrices, positioning latitude and longitude, and inertial attitude angles collected by a UAV to obtain a two-dimensional image matrix and an initial pose matrix, while obtaining a pre-calibrated camera intrinsic parameter matrix; S2, based on a pre-trained dual-branch deep convolutional neural network, performing multi-branch high semantic information extraction on the two-dimensional image matrix to obtain crack segmentation masks and structural edge masks; S3, constructing a perspective projection based on the camera intrinsic parameter matrix and the initial pose matrix. S4. The outer envelope geometry of the pre-set 3D bridge information model is rendered and projected in a dimensionality reduction manner to obtain the rendering edge feature lines that reflect the expected skeletal lines of the bridge from the current viewpoint. S5. A cost function is constructed using the chamfer distance between the structural edge mask and the rendering edge feature lines, and the initial pose matrix is ​​iteratively optimized and corrected to obtain the optimized pose matrix. S6. Based on the optimized pose matrix and the camera intrinsic parameter matrix, the dimensionality reduction of each feature pixel in the crack segmentation mask is reversed to a 3D physical mapping to obtain a 3D damage feature vector that includes the crack physical size and global spatial positioning.

[0016] Specifically, step S1 involves parsing and parameter stripping the raw sensor data stream, which includes pixel matrices, latitude and longitude coordinates, and inertial attitude angles collected by the UAV, to obtain a two-dimensional image matrix and an initial pose matrix, while simultaneously acquiring a pre-calibrated camera intrinsic parameter matrix. It should be understood that in UAV-based bridge damage monitoring applications, the sensors generate raw data streams containing various data types during UAV flight. These data are encapsulated in a specific protocol format and cannot be directly used by subsequent image processing and 3D mapping modules. Therefore, in the technical solution of this application, the highly coupled, unstructured raw data stream collected by the UAV in a dynamic environment is transformed into logically independent and formatted computational inputs. Through synchronous parsing of the pixel matrix, geographic coordinates, and flight attitude, the visual feature basis to be analyzed is established, and a crucial affine transformation relationship and perspective geometric mapping foundation are constructed for subsequent 3D model rendering and damage physical size mapping.

[0017] In practice, the raw sensor data stream is first unpacked and verified frame by frame, and its payload is separated to extract the pixel binary payload and telemetry metadata. During this process, the data stream is parsed byte by byte according to the pre-defined communication protocol's frame header, frame trailer, checksum, and other information to identify complete data frames. Each identified frame is verified to ensure error-free data transmission. After successful verification, the payload portion of the data frame is unpacked according to the data field structure defined in the protocol, extracting different types of raw data. Specifically, the binary byte sequence containing the raw pixel information captured by the camera is separated as the pixel binary payload; simultaneously, the structured data containing the UAV's real-time positioning (latitude, longitude, altitude) and attitude (pitch angle, roll angle, yaw angle) and other sensor status information is separated as telemetry metadata.

[0018] Next, the pixel binary payload is decoded, converted, and reshaped to generate a two-dimensional image matrix. During this process, for the separated image binary data, if the data is in a compressed encoding format, it is decoded into an RGB or grayscale pixel value array using the appropriate decoding library (such as libjpeg-turbo, FFmpeg). If the data is in the original sensor format (such as Bayer RAW), it is converted to RGB format using image signal processing algorithms such as demosaicing. Furthermore, to facilitate subsequent mathematical calculations, these integer pixel values ​​are normalized, for example, from the integer range of [0, 255] to the floating-point range of [0, 1] or [-1, 1]. Finally, the one-dimensional pixel value array is reshaped according to the image's width, height, and number of channels (e.g., height × width × 1 for grayscale images, and height × width × 3 for RGB images) to obtain a standard two-dimensional image matrix.

[0019] Furthermore, the latitude and longitude parameters in the telemetry metadata are transformed into a local northeast-sky coordinate system to obtain a translation vector, and the three-axis inertial attitude angles are solved to obtain a rotation matrix. The translation vector and the rotation matrix are combined to construct a homogeneous affine transformation matrix to obtain the initial pose matrix. In this process, the latitude, longitude, and altitude coordinates of the UAV are first transformed from the geographic coordinate system (WGS84) to a local rectangular coordinate system (such as the northeast-sky ENU coordinate system) with a fixed reference point on the bridge (such as the center of the pier) as the origin. This transformation process involves the conversion from geodetic coordinates to spatial rectangular coordinates, and finally a three-dimensional translation vector is obtained, representing the position of the camera center relative to the origin. Meanwhile, the roll, pitch, and yaw angles provided by the UAV are in Euler angle form. Based on a specific rotation sequence (usually yaw-pitch-roll, i.e., ZYX sequence), rotation matrices around the Z-axis (yaw), Y-axis (pitch), and X-axis (roll) are calculated respectively. These three basic rotation matrices are then multiplied sequentially to obtain a 3x3 rotation matrix, used to describe the orientation of the camera coordinate system. Finally, this rotation matrix and translation vector are combined into a 4x4 homogeneous transformation matrix, i.e., the initial pose matrix, which fully defines the rigid body motion from the world coordinate system to the camera coordinate system.

[0020] Specifically, S2, based on a pre-trained dual-branch deep convolutional neural network, performs multi-branch high semantic information extraction on the two-dimensional image matrix to obtain crack segmentation masks and structural edge masks. It should be understood that in practical bridge monitoring scenarios, simply acquiring crack pixels cannot determine their absolute physical location on the bridge entity, while merely acquiring the structural outline cannot perceive the nature of the damage. Through synchronous extraction by the dual-branch neural network, it is possible to provide subsequent steps with structural edge masks possessing spatial constraints and crack segmentation masks possessing attribute determination value, thereby ensuring that damage detection and spatial localization are coupled under the same spatiotemporal reference.

[0021] Figure 3 This is a flowchart of step S2 in the image processing-based bridge damage monitoring method according to an embodiment of this application. Figure 3 As shown, S2 includes: S21, performing hierarchical spatial downsampling and cross-scale lateral fusion extraction on the two-dimensional image matrix to generate a shared feature tensor; S22, using the first decoding branch of a dual-branch deep convolutional neural network to classify and binarize the shared feature tensor for microscopic damage pixels to obtain a crack segmentation mask; S23, using the second decoding branch of a dual-branch deep convolutional neural network to extract the macroscopic structural contour gradient response of the shared feature tensor to obtain a structural edge mask.

[0022] Specifically, in step S21, hierarchical spatial downsampling and cross-scale lateral fusion extraction are performed on the two-dimensional image matrix to generate a shared feature tensor. In this process, the two-dimensional image matrix is ​​first input into an encoder, which consists of multiple layers of convolution and pooling operations stacked together. With each layer, the spatial size of the feature map is proportionally reduced (downsampling), allowing the network to capture a wider range of image context and high-level semantics, but also resulting in a loss of detail. Therefore, to balance broad semantic understanding and detail preservation, this embodiment employs cross-scale lateral fusion technology. That is, during the encoding process, feature maps of different depths (i.e., different resolutions) are exchanged and integrated through upsampling and concatenation. For example, deep, high-semantic, low-resolution features are upsampled and combined with shallow, low-semantic, high-resolution features. Finally, a shared feature tensor that integrates multi-scale visual information is generated, containing both global concepts and local details about the image content, providing a common and rich data foundation for two subsequent specialized tasks requiring different focuses.

[0023] Specifically, in step S22, the first decoding branch of a dual-branch deep convolutional neural network performs microscopic damage pixel classification and binarization segmentation on the shared feature tensor to obtain a crack segmentation mask. This branch is a decoder designed to progressively decode and restore the abstract semantic information about cracks contained in the shared feature tensor to each pixel location of the original image. During this process, the branch progressively restores the spatial dimensions of the feature map through a series of upsampling operations (such as transposed convolution), ultimately making it the same height and width as the input image. Specifically, skip connections are introduced during the decoding process to fuse high-resolution detail features from the corresponding layers of the encoder stage, improving the boundary segmentation accuracy for small targets like cracks. The last layer of the decoding branch is a 1x1 convolutional layer, which maps the multi-channel feature map into a single-channel matrix. Each pixel value in this matrix is ​​then passed through a sigmoid activation function, converting it into a probability value between 0 and 1, representing the confidence that the pixel belongs to the crack category. Furthermore, a threshold is set for this probability map to obtain the final binarized crack segmentation mask. For each pixel location in the image, if its calculated crack probability is greater than or equal to this threshold, then in the final mask matrix, that location is marked as 1 (representing a crack); otherwise, it is marked as 0 (representing the background). This results in a matrix of the same size as the input image, consisting only of 0s and 1s, where all pixels with a value of 1 constitute the identified crack region.

[0024] Specifically, in step S23, the second decoding branch of a dual-branch deep convolutional neural network extracts the macroscopic structural contour gradient response of the shared feature tensor to obtain the structural edge mask. This branch works in parallel with the first branch and aims to extract and reconstruct the structural edge mask from the same shared feature tensor. Notably, this branch has a decoder structure similar to the crack segmentation branch but with a different task objective. Its training objective is to learn to identify edge pixels in the image that belong to the macroscopic structural contour of the bridge, such as the bottom boundary of the beam, the vertical edge of the pier, and the projection lines of the stay cables. In this process, this branch also restores the spatial resolution through upsampling operations, and its last layer outputs a single-channel matrix. Each pixel value in this matrix, after being processed by an activation function, represents the intensity or response value of a significant structural edge at that location. Furthermore, to obtain clear, clean, and typically single-pixel-wide structural edge lines, the output edge response map is first thresholded, i.e., an edge intensity threshold is set, and only pixels with response values ​​exceeding the threshold are retained. Secondly, edge thinning is performed, typically using a non-maximum suppression algorithm. The algorithm checks each pixel along its gradient direction (i.e., the normal direction of the edge), retaining only the pixels with the largest local response in that direction while suppressing other non-maximum points. After these two steps, a binary structural edge mask is obtained, where pixels with a value of 1 clearly outline the contours of the main components of the bridge.

[0025] Specifically, in step S3, a perspective projection transformation is constructed based on the camera intrinsic parameter matrix and the initial pose matrix. This transformation is used to perform dimensionality reduction rendering projection on the outer envelope geometric line group of the preset 3D bridge information model to obtain the rendered edge feature lines that reflect the expected bridge skeleton lines from the current viewpoint. Since the initial pose matrix acquired by the UAV often contains measurement errors due to GPS multipath effects or inertial sensor drift, directly using this pose for damage localization will produce significant projection deviations. By rendering the bridge skeleton lines in an ideal state in this step, the system can measure the geometric residual between the ideal projection and the actual edge in subsequent steps, thereby providing the necessary numerical driving target for accurate iterative optimization of the pose.

[0026] It is worth mentioning that the pre-set 3D bridge information model refers to a digital model created before monitoring that contains the bridge's precise geometric dimensions and serves as the geometric truth source for this step. The outer envelope geometric line group refers to the set of line segments extracted from the 3D model that characterize the bridge's external macroscopic shape and contour; it is a simplified wireframe representation of the model. The final rendered edge feature lines refer to the binary image generated through the above projection and rasterization processes. The white lines in the image represent the expected structural skeleton of the bridge as seen from the 3D model from the current camera's perspective, and serve as a theoretical template for edge registration with the real image.

[0027] In practice, the first step involves performing outer envelope geometric topology analysis and wireframe dimensionality reduction on the 3D bridge information model to obtain a group of lines in 3D space. This step aims to extract the simplified wireframe that best represents the macroscopic geometric skeleton from a complex 3D BIM model containing rich details (such as internal structure and surface texture). During this process, the boundary edges of all polygons constituting the model's outer surface are extracted by processing the model's triangular mesh or parametric surfaces. Specifically, this can be achieved by analyzing the model's boundary representation, traversing all faces, and collecting edges shared only by a single polygon (i.e., boundary edges), or by calculating the model's convex hull using an algorithm and extracting its edges. These edges are represented in 3D space as line segments defined by two 3D vertices. Collecting all such line segments yields a group of lines in 3D space, which is a simplified yet precise geometric abstraction of the bridge's macroscopic skeletal structure, preparing the data for subsequent projection rendering.

[0028] Next, based on the camera intrinsic matrix and the initial pose matrix, perspective geometric projection mapping is performed on each 3D vertex in the 3D spatial line group to obtain a 2D structural projection line group. That is, the perspective projection model is applied to project the 3D wireframe obtained in the previous step onto the 2D image plane. Each 3D line segment is defined by two 3D vertices. In this process, the projection process consists of two coordinate transformation steps. First, the initial pose matrix is ​​used to transform the 3D points in the world coordinate system to the camera coordinate system. The pose matrix is ​​a 4x4 transformation matrix that describes how to rotate and translate the coordinates of a point from the global world coordinate system to the camera-centric coordinate system. Second, the camera intrinsic matrix is ​​used to project the 3D points in the camera coordinate system to the pixel coordinate system. The intrinsic matrix is ​​a 3x3 matrix containing internal optical parameters such as the camera's focal length and principal point. Specifically, the intrinsic matrix is ​​multiplied by the 3D point coordinates in the camera coordinate system to obtain a 3D homogeneous coordinate vector; then, the first two components of this vector are divided by the third component (i.e., the depth value) to obtain the final 2D pixel coordinates. The above projection calculation is performed on each vertex of the group of lines in 3D space to obtain its corresponding pixel in the 2D image. Connecting the projection points of the two vertices of the same line segment forms a line segment in the 2D image. All these projected line segments constitute the 2D structure projection line group, which completely simulates the theoretical imaging of the 3D bridge wireframe on the camera sensor under given camera parameters and pose.

[0029] Furthermore, subpixel-level coloring and mask synthesis are performed on the two-dimensional structure projection line group to obtain the rendered edge feature lines. Since the vertex coordinates obtained from projection calculations are usually floating-point numbers, directly rounding them to integer pixel coordinates will introduce jagged edges and quantization errors. Therefore, in the embodiments of this application, firstly, subpixel-level coloring is performed on the two-dimensional structure projection line group, which is usually achieved through a high-precision linear rasterization algorithm. For example, using the Bresenham algorithm or its improved anti-aliasing version, the image pixels that the line segment path should cover are determined based on the subpixel coordinates of the two endpoints of the line segment, and the weight or intensity value of each covered pixel is calculated to achieve smooth, jagged-free line display. Finally, a binary mask is performed for subsequent comparison. Specifically, a threshold is set, and when the intensity value of a pixel calculated by the rasterization algorithm is greater than the threshold, the pixel position is marked as foreground (line, value 1). All pixels covered by the rasterized line segments are marked and synthesized on a blank canvas (an initial matrix of all zeros) of the same size as the real image. The resulting binary image is the rendered edge feature line, which accurately reflects the pure outline that the ideal 3D bridge model skeleton should present under the current initial pose and camera intrinsic parameters.

[0030] Specifically, in step S4, a cost function is constructed using the chamfer distance between the structural edge mask and the rendered edge feature lines. This cost function is then used to iteratively optimize and correct the initial pose matrix to obtain an optimized pose matrix. It should be understood that although step S1 obtains the initial pose, in complex bridge operation environments, the mapping relationship between the two-dimensional pixels and the three-dimensional model directly established by this pose often deviates significantly. This manifests as the rendered bridge skeleton not coinciding with the structural edges in the real image. In the technical solution of this application, by constructing a cost function and performing iterative optimization, this pixel-level geometric residual can be inversely compensated into the pose parameters, thereby obtaining an optimized pose matrix that truly reflects the spatial geometric relationship at the moment of capture, laying the foundation for subsequent high-precision damage prediction.

[0031] It's worth noting that chamfer distance is a geometric metric that measures the similarity between two point sets. It is defined by calculating the sum or sum of squares of the Euclidean distances from each point in one set to its nearest neighbor in the other set. The cost function, on the other hand, is the objective function that correlates the parameters to be optimized (i.e., the six degrees of freedom of pose) with the geometric deviations.

[0032] In the first embodiment, step S4 includes the following steps: S41, determining the edge coincidence error scalar based on the structural edge mask and the rendered edge feature lines; S42, calculating the Jacobian partial derivative of the edge coincidence error scalar with respect to the six degrees of freedom parameters of the initial pose matrix to obtain the pose descent gradient vector; and S43, performing iterative accumulation correction and manifold convergence update on the initial pose matrix based on the pose descent gradient vector to obtain the optimized pose matrix.

[0033] Specifically, in step S41, an edge coincidence error scalar is determined based on the structural edge mask and the rendered edge feature lines. Due to inherent deviations in the initial pose acquired by the UAV, the bridge skeleton lines rendered based on this pose cannot precisely coincide with the structural edges in the actual image. This geometric spatial misalignment can be characterized by a mathematically monotonic value. Specifically, by calculating the edge coincidence error scalar, the system can quantify the degree of defect in the current pose, thereby guiding the optimization algorithm to force the rendered edges to converge towards the real edges by reducing this scalar value, ultimately achieving millimeter-level spatiotemporal alignment.

[0034] In this process, firstly, non-zero elements are extracted from the input 2D structural edge mask and 2D rendered edge feature lines, and the coordinates of the highlighted contour pixels contained therein are projected into two discrete coordinate point sets in a continuous 2D Euclidean space. Next, a bidirectional chamfering distance error metric cost function is constructed, calculating the nearest neighbor Euclidean projection distance between the extracted edge point set in the real world and the rendered projected edge point set, and symmetrically calculating the mean. This yields a global numerical loss model reflecting the severity of the deviation between the two current positions. The cost loss function is defined and calculated as follows: in, The edge overlap error scalar is a real number that reflects the penalty value of pixels that do not fit the bridge structure in the real image with the twin rendered skeleton on the two-dimensional image plane. It refers to the set of two-dimensional pixel Euclidean coordinate positions of positive class (real-world highlight contour points) extracted from the two-dimensional structural edge mask of the input element. It refers to the set of continuous pixel coordinates of the front view of the model skeleton on the two-dimensional image screen, which is extracted from the two-dimensional rendering edge feature lines of the input elements based on spatial deduction. Representative set A discrete physical point randomly sampled from within. Representative set A discrete pre-rendered point in the inner random traversal. The cardinality is the total number of elements in the point set X. It represents the order of magnitude of the number of vertices that make up the point set Y. The L2 Norm constrained distance operator is used to characterize the least squares distance penalty amplification of the absolute straight-line span between two coordinate system points.

[0035] After the calculation is completed, the generated digital loss is packaged and output together with the synchronously transmitted initial pose matrix, thereby ensuring that a state basis is provided for the next step of calculus differentiation calculation.

[0036] Specifically, in step S42, the Jacobian partial derivatives of the edge coincidence error scalar with respect to the six degrees of freedom parameters of the initial pose matrix are calculated to obtain the pose descent gradient vector. Since there is a nonlinear deviation between the initial pose and the true pose, simply observing the error scalar cannot directly reveal how to adjust the six degrees of freedom parameters of the UAV, including three-axis translation and three-axis rotation. In the technical solution of this application, by calculating the Jacobian partial derivative matrix, the influence direction and intensity of the small perturbation of each parameter component on the image plane projection error can be accurately quantified, thereby generating the pose descent gradient and driving the model projection lines to quickly and accurately approach the edge of the real structure. Here, the six degrees of freedom parameters refer to the three translational components and three rotational components describing the spatial state of the camera. These are usually parameterized as vectors in the Lie algebra space during calculation to avoid the computational complexity caused by the orthogonality constraint of the rotation matrix. The Jacobian partial derivative calculation refers to using the principle of partial calculus to calculate the derivative matrix of each observation term in the cost function with respect to the state variable. It reflects the sensitivity of the error function to changes in the state variable. The final pose descent gradient vector is the linear sum of the Jacobian matrices of all observations, which numerically represents the local rate of change of the error function at the current parameter point.

[0037] In this process, the initial pose matrix is ​​first reduced in dimensionality in manifold space, transforming it from a special Euclidean group mapping lacking global additivity to the Lie algebraic morphological domain of its tangent space. This overcomes the gimbal lock and singularity dimensionality explosion problems in Jacobian matrix differentiation. Next, using a deep learning automatic differentiation interface, the chain rule is introduced to calculate the Jacobian partial derivatives of the input edge coincidence error scalar with respect to the six degrees of freedom (6-DoF) of spatial translation and rotation within the Lie algebraic vector. These partial derivatives are then transposed and combined to form a multidimensional spatial gradient compensation column vector that determines the most rapid descent direction. The partial differential derivation of this descent gradient is as follows: in, The pose descent gradient vector generated by the derivative is a 6×16×1 numerical column vector matrix, indicating the steepest spatial descent direction on the error functional surface.

[0038] It is a special mathematical symbol that represents the six-dimensional tangent vector Lie algebraic expression of the initial pose matrix as a bijection and its inverse (i.e., a column vector containing six degrees of freedom of pose). It is the core partial differential operator in calculus that identifies the separation of partial derivatives of multiple variables. Lie algebra The first three sub-measures in the text represent the physical components within the three-dimensional Euclidean geometric space. Translational component matrices of the displacement rates of the three direction-finding axes. Lie algebra The latter three-dimensional sub-measures in the matrix describe the rotational component matrix that describes the pitch, lateral tilt, lateral rotation, and torsional changes around the principal axis of the spatial three-dimensional coordinate system.

[0039] After calculating the descent gradient, the initial pose matrix, which was originally transmitted through the forward channel, is executed together with the descent gradient.

[0040] Specifically, in step S43, the initial pose matrix is ​​iteratively accumulated and corrected based on the pose descent gradient vector, followed by manifold convergence update to obtain the optimized pose matrix. Due to the complexity of lighting changes and perspective distortion in the bridge inspection environment, the residual surface is usually highly nonlinear, and single-dimensional linear interpolation cannot directly obtain the global optimal solution. Furthermore, since the rotation component in the pose matrix belongs to a special Euclidean group in a non-Euclidean space, traditional parameter addition updates will violate the orthogonality constraints of the matrix, leading to distortion of the camera coordinate system. Therefore, in the technical solution of this application, convergence updates are performed through exponential mapping of the manifold space to ensure that the pose matrix maintains its rigorous rigid body transformation physical meaning throughout the correction process. Here, iterative accumulation correction refers to a cyclic refinement process driven by the optimization objective, which involves repeatedly calculating increments and applying them to the current state to gradually approach the global optimal solution. Manifold convergence update refers to the parameter mapping and composite operation performed between the tangent space of the special Euclidean group and its corresponding Lie group manifold. It ensures that rotation and translation conform to the inherent topological structure of the geometric manifold during the update process. The final optimized pose matrix refers to the spatial transformation matrix that most accurately describes the UAV's observation viewpoint when the preset convergence criteria are met, such as the residual change rate being lower than a specific threshold.

[0041] In this process, the built-in adaptive torque prediction optimizer (such as Adam or a Gauss-Newton based second-order descent unit) is first invoked to scale the introduced pose descent gradient vector according to the learning rate step size parameter. Next, the accumulated compensated Lie algebra increment is projected back into the true world Lie group transformation matrix using the exponential mapping equation, and then superimposed onto the input initial pose matrix. This closure iteration is repeated until the compensated vector approaches zero or reaches the global minimum convergence threshold. This step thoroughly corrects and eliminates the positioning drift error, completing the nonlinear feedback approximation operation. The computational evolution rule formula structure is as follows: in, This refers to the final solution product optimization pose matrix obtained when the entire lifecycle of this main process is declared over. For the manifold exponential map in multidimensional space, the Rodrigues equivalent formula can legally transform and wrap the incremental tangents of the unconstrained tangent space in six dimensions into orthogonal standard orientation rotations and offsets. The adaptive descent step size, usually called the system evolution learning rate, controls the pace of each cycle to prevent overcorrection that could lead to non-convergent oscillations in the solution space. It is a dedicated underspin vector outer product symbol, which transforms a one-dimensional 6×1 translation and rotation six-parameter column vector into a 4×4 antisymmetric characteristic matrix form that is allowed for three-dimensional operations, thus laying the underlying structural foundation for left multiplication algebra calculations. The initial pose matrix (Pose_Init) parameters are used as the base text for the coverage and historical residual aggregation to be performed. This ensures that the final derived solution will possess an absolute position with centimeter-level alignment accuracy.

[0042] Specifically, research has revealed that in actual engineering scenarios of bridge structural health monitoring, when high-definition cameras mounted on drones collect images in semi-enclosed spaces such as the bridge web and the inner wall of box girders, the acquired two-dimensional structural edge masks contain rich macroscopic structural geometric information—horizontal beam bottom boundary lines, vertical pier outlines, and inclined cable-stayed bridge projection lines, etc. These edge pixels are not isolated, scattered zero-dimensional points, but belong to continuous structural line segments with clear physical orientation characteristics. Each edge pixel naturally carries a local tangent direction vector on its respective line segment, reflecting the physical orientation of the structural component at that point. Similarly, each pixel in the two-dimensional rendered edge feature lines generated by perspective projection and differential rasterization rendering from a three-dimensional bridge information model also possesses a deterministic local tangent direction determined by the model's geometric topology. However, in the first embodiment described above, step S41 only extracts non-zero elements from the two-dimensional structural edge mask and the two-dimensional rendered edge feature lines, projecting the coordinates of the highlighted contour pixels contained therein into two sets of attribute-free discrete coordinate points in a continuous two-dimensional Euclidean space. Then, using the standard bidirectional chamfer distance as the error metric, it only calculates the Euclidean positional distance between points, completely ignoring the tangent direction angle deviation between matching point pairs. This approach, which degenerates structured edges with rich geometric direction semantics into a set of zero-dimensional scattered points without direction attributes, loses the crucial geometric constraint of the consistency of local tangent directions between edge points. This makes the cost function unable to distinguish between pseudo-matches that are spatially close but have incorrect directions and true matches that are spatially close and have correct directions.

[0043] Under real-world bridge inspection conditions, the engineering consequences of this defect are particularly severe. Specifically, when a drone experiences significant rotational deviations in its initial pose due to interference from the multipath effect of the bridge's steel box girder in narrow spaces such as the bridge deck, edge segments that originally belong to different structural components in three-dimensional space (such as the bottom boundary of a horizontal beam and the outline of a vertical pier) will overlap extensively in the two-dimensional image plane due to rotational misalignment. In this situation, a real edge point x on the bottom edge of a horizontal beam is very likely to be matched with a rendered point y on the edge of a vertical pier in terms of Euclidean distance—although the spatial distance between the two is small, their tangent directions are nearly orthogonal (the included angle is close to 90°), which is a typical cross-component mismatch. This mismatch with inconsistent directions injects seriously misleading signals into the Jacobian gradient field of the subsequent step S42, causing the pose optimizer to converge along the wrong direction, ultimately resulting in local optimum traps or oscillating non-convergence, severely weakening the overall pose correction accuracy and robustness of step four.

[0044] To address the deficiency in step S41 where the standard chamfer distance ignores the consistency of local tangent directions at edge points, this application proposes a preferred embodiment. This embodiment designs a direction-aware anisotropic weighted chamfer distance mechanism around the special geometric constraint of local tangent direction consistency. It introduces a tangent direction angle deviation penalty factor on top of the original spatial Euclidean distance, enabling the cost function to simultaneously constrain the positional proximity and directional consistency of matching point pairs, fundamentally eliminating the contamination of the gradient field by cross-component erroneous matching.

[0045] Specifically, firstly, edge pixel extraction and local tangent direction field estimation are performed on the structural edge mask and rendered edge feature lines to obtain the real edge direction point set and the rendered edge direction point set. Since the first embodiment completely discards the direction information of the structural line segment where each pixel is located when extracting the edge point set, the improved mechanism first needs to recover and retain this direction semantics while extracting coordinates. Specifically, the input two-dimensional structural edge mask is subjected to non-zero pixel traversal extraction to obtain the real edge discrete coordinate point set. Simultaneously, within the local neighborhood window of each extracted point, the unit tangent direction vector of that point is estimated using the orthogonal direction of the image gradient (i.e., along the edge direction). The same extraction and direction estimation operations are performed on the two-dimensional rendered edge feature lines. For any edge point... Its local tangent direction vector is obtained by rotating the image gradient vector in the neighborhood by 90° and normalizing it. This process can be expressed by the formula: in, The unit tangent direction vector at edge point p represents the physical orientation of the structural line segment containing that point. It is a 90° rotation matrix in a two-dimensional plane, used to rotate the gradient normal perpendicular to the edge to the tangent direction along the edge. Let be the two-dimensional image spatial gradient vector at edge point p, which consists of partial derivative components in the horizontal and vertical directions; This is an L2 norm normalization operator that ensures the output is a unit-length direction vector.

[0046] In bridge inspection scenarios, for pixels on the bottom boundary of horizontal beams, the estimated tangent direction will be approximately horizontal; for pixels on the outline of vertical piers, the tangent direction will be approximately vertical—this difference in direction is the key criterion for distinguishing true from false matches. After this processing step, the output sets of true edge direction points and rendered edge direction points are obtained, both containing the two-dimensional coordinates of each edge point and its corresponding unit tangent direction vector, providing a complete data base with direction attributes for subsequent direction consistency evaluation.

[0047] Next, directional consistency matching weight estimation is performed on the real edge direction point set and the rendered edge direction point set to obtain a forward matching weight set and a reverse matching weight set. That is, after obtaining the edge point set with directional attributes, the degree of directional deviation between each pair of nearest neighbor matching points is further quantified and transformed into a weight scalar that can be used to modulate the distance contribution. However, the reason for constructing weights separately instead of directly adding the directional deviation to the distance formula is that directional deviation and positional distance are physical quantities with different dimensions. Only after mapping them to dimensionless (0,1] interval weights through an exponential kernel function can they be coupled with the Euclidean distance squared term in a multiplicative gating manner without introducing dimensional confusion. Specifically, for each directional edge point in the real edge direction point set... The process involves retrieving the nearest neighbor point in the Euclidean distance from the point set at the rendering edge direction, then calculating the absolute value of the cosine of the angle between the tangent directions of the matched point pair—the absolute value is used because there is 180° equivalence in the edge tangent directions, meaning that the forward and reverse orientations of the tangent directions on the same line segment are geometrically equivalent—and mapping the direction deviation to a direction consistency weight scalar using an exponential kernel function. The same reverse matching and weight calculation are then performed symmetrically on each point in the rendering edge direction point set. The calculation process for the direction consistency weight can be expressed by the formula: in, This indicates exponentiation. For edge points Matching point with its nearest neighbor The directional consistency weight scalar between them has a value range of (0,1]. For point The unit tangent direction vector at that point originates from the set of real edge direction points; Nearest neighbor matching point The unit tangent direction vector at that location is derived from the set of rendering edge direction points; The dot product of two unit vectors is equal to the absolute value of the cosine of the angle between them. This is an absolute value operator used to eliminate ambiguity regarding the 180° equivalence of the tangent direction; This is a hyperparameter for orientation sensitivity, which controls the decay rate of the orientation deviation penalty. The smaller the value, the stronger the suppression of orientation inconsistency.

[0048] This means that in bridge inspection scenarios, when the tangent directions of the matching point pairs are completely consistent (for example, both are points on the bottom edge of a horizontal beam), the matching points will be in the same position. When the numerator is zero and the weight approaches 1, the distance contribution of the matching pair is fully preserved; when the tangent directions are nearly orthogonal (e.g., the edge point of a horizontal beam is mismatched to the edge point of a vertical pier column). As the numerator approaches 1, the weights decay exponentially to approach 0, strongly suppressing the distance contribution of the incorrect match. After this step, the forward matching weight set and the reverse matching weight set are output, corresponding to the scalar set of directional consistency weights for each point in the real edge direction point set and the rendered edge direction point set, respectively.

[0049] Furthermore, based on the forward and reverse matching weight sets, a point-by-point multiplicative gated weighting and normalized accumulation of the forward and reverse nearest neighbor Euclidean distances are performed to obtain the edge coincidence error scalar. That is, after obtaining the point-by-point directional consistency weights, they are incorporated into the chamfer distance accumulation framework to replace the original equal-weight accumulation mechanism. Specifically, the forward and reverse matching weight sets are used to perform point-by-point weighted modulation of the forward and reverse nearest neighbor Euclidean distances to construct a direction-aware anisotropic bidirectional chamfer distance cost function. Compared with the original standard chamfer distance, this improved cost function uses the directional consistency weight as a multiplicative gate factor when accumulating the distance contribution of each pair of matching points, ensuring that directionally consistent matching pairs dominate the gradient signal, while directionally contradictory pseudo-matching pairs are adaptively suppressed. The complete expression of the improved cost function is as follows: in, This is a scalar cost for the direction-aware anisotropic chamfer distance, which is a replacement for the improved edge coincidence error scalar. This is a set of two-dimensional coordinate points extracted from the set of points along the true edge direction; This is a set of two-dimensional coordinate points extracted from the set of points along the rendering edge direction; For positive matching weight concentration point The corresponding directional consistency weight scalar; This means for each ,exist Find the point with the smallest squared Euclidean distance. , used to measure matching error; For reverse matching weight concentration point The corresponding directional consistency weight scalar; This means for each ,exist Find the point with the smallest squared Euclidean distance. , used for reverse matching error; This is the L2 norm squared distance operator; the weighted summation term in the denominator is used for normalization to ensure that the numerical scale of the cost function is not affected by fluctuations in the number of effective matching points.

[0050] This means that in bridge inspection scenarios, matching errors along the same structural component are given high weight to drive fine alignment, while incorrect matching errors across different component orientations are given extremely low weight to avoid gradient contamination. This ensures that the gradient signal received by the Jacobian partial derivative matrix in subsequent sub-step 4.2 is pure and correctly oriented. After this step, the improved edge coincidence error scalar is output.

[0051] Specifically, the preferred embodiment proposed in this application introduces an exponential kernel-gated weight mechanism based on the cosine similarity of the local tangent direction into the point-to-point matching evaluation system of the chamfer distance. This upgrades the improved cost function from an isotropic pure position metric to an anisotropic joint position-direction metric, thereby adaptively identifying and suppressing erroneous nearest neighbor matching across components at the cost function level, retaining only true matching with the same component direction as an effective gradient source. Ultimately, in actual engineering scenarios of bridge inspection, when the initial pose rotation deviation of the UAV is large due to GPS multipath effect in narrow spaces such as the bridge overpass, and the bottom edge of the horizontal beam and the edge of the vertical pier overlap extensively in pixel space, this preferred embodiment suppresses the contribution of pseudo-matches that are spatially close but orthogonally orthogonal to near zero through the exponential decay characteristic of the direction consistency weight. This significantly improves the convergence robustness and final alignment accuracy of the pose optimizer under the initial condition of large rotation deviation, effectively avoiding local optimum traps or oscillatory non-convergence caused by gradient signal contamination by erroneous matching, thus ensuring the millimeter-level positioning reliability of subsequent mapping of two-dimensional micro-damage to three-dimensional physical coordinates.

[0052] Specifically, in step S5, based on the optimized pose matrix and camera intrinsic parameter matrix, a dimensionality reduction and inverse 3D physical mapping of each feature pixel in the crack segmentation mask is performed to obtain a 3D damage feature vector containing the crack's physical size and global spatial location. It should be understood that in 2D images acquired by UAVs, the pixel width occupied by cracks undergoes drastic visual scaling with changes in shooting distance, pitch angle, and lens focal length, making it unsuitable for direct engineering evaluation. By performing this step, the system can utilize the corrected precise pose and pre-calibrated intrinsic parameters to restore the lightweight 2D semantic mask into a 3D feature with accurate geographic coordinates and physical dimensions (such as millimeter-level width), thereby providing bridge maintenance units with comparable and traceable structural damage data. The final 3D damage feature vector is a structured data vector encapsulating the crack's center position, orientation angle, physical length, and maximum width in a global coordinate system.

[0053] In practice, firstly, based on the camera intrinsic matrix and the optimized pose matrix, the coordinates of each active pixel in the crack segmentation mask are inversely projected using perspective to obtain a global set of ray equations. Inverse projection refers to the process of inverting discrete pixel coordinates on the image plane into ray beams in three-dimensional space, following the backward propagation principle of perspective geometry. This establishes a corresponding backward projection path in the three-dimensional world for each pixel representing a crack in the image. During this process, a binary crack segmentation mask is input, where all pixels with a value of 1 (active pixels) constitute a coordinate set. For each pixel coordinate in the set, it is first inversely projected from the pixel coordinate system back to the normalized imaging plane in the camera coordinate system using the inverse of the camera intrinsic matrix. Specifically, a three-dimensional homogeneous vector containing the pixel coordinate and the constant 1 is constructed, and then this vector is left-multiplied by the inverse of the intrinsic matrix. The result is a two-dimensional point on the normalized plane located one unit distance in front of the camera in the camera coordinate system. Since the actual depth information is unknown, this point actually defines a direction vector originating from the camera's optical center (origin) and passing through the pixel. Next, the ray in the camera coordinate system is transformed to a global coordinate system describing the real world. This is achieved by optimizing the pose matrix, which describes the rotation and translation of the camera coordinate system relative to the world coordinate system. Through the inverse transformation of the pose matrix, the specific position of the camera optical center in the world coordinate system can be calculated, and the ray direction vector is rotated to align with the world coordinate system. Furthermore, in the world coordinate system, the 3D spatial path corresponding to this pixel can be described by a parametric ray equation: a starting point (the position of the camera optical center) plus a direction vector multiplied by a distance parameter greater than zero. Repeating the above calculation for each active pixel in the crack mask yields a set of global spatial ray equations, where each ray represents the theoretical ray path from the camera lens to an unknown point on the bridge surface corresponding to a pixel.

[0054] Next, collision and intersection mapping is performed on each ray in the global spatial ray equation set and the mesh surface of the 3D bridge information model to obtain a 3D absolute coordinate point cloud. The 3D bridge information model is typically represented by a mesh surface composed of countless triangular facets. In this process, firstly, for each ray in the ray set, its intersection points with all triangular facets of the bridge model are calculated. Specifically, an efficient algorithm (such as the Möller-Trumbore algorithm) is used to solve for the intersection parameters of the ray and the plane containing the triangle, and further determines whether the intersection point lies within the boundary of the triangle. In particular, due to the large number of bridge model facets, spatially accelerated data structures (such as bounding box hierarchies BVH or KD trees) can be used to quickly eliminate a large number of obviously non-intersecting triangles, thus significantly improving the intersection efficiency. For each ray, the algorithm finds the effective intersection point closest to the camera along the ray direction from all possible intersecting facets. If a ray successfully intersects a triangular facet, the 3D world coordinates of that intersection point are recorded. After traversing all rays representing crack pixels and performing intersection calculations, a set of three-dimensional spatial points is obtained, namely a three-dimensional absolute coordinate point cloud, which accurately depicts the actual spatial distribution of the crack pattern in the image on the real three-dimensional bridge surface.

[0055] Furthermore, the physical dimensions of paired points at the crack boundary in the 3D absolute coordinate point cloud are quantized and spatially located to obtain a 3D damage feature vector. In this process, firstly, the geometric structure of the crack is identified from the point cloud, such as distinguishing the two sides of the crack. This can be achieved by analyzing the spatial density and principal direction of the point cloud, or by combining the connected component information of the original 2D mask. For a single crack, paired points representing the crack opening are found. Typically, on a cross-section perpendicular to the local crack direction, two points in the 3D point cloud that are closest to each other and belong to the two sides of the crack are searched. These two points constitute a boundary pairing point, and the Euclidean distance between them is the physical width of the crack at that location. Multiple cross-sections are selected along the crack's length for this type of calculation to obtain a set of width samples, from which statistical features such as maximum width and average width are extracted. The physical length of the crack can be calculated by reconstructing its 3D centerline. Specifically, the centerline can be extracted by skeletonizing the point cloud or approximated by a sequence of midpoints connecting the two sides of the crack's edge. The total length of the crack is obtained by summing the 3D lengths of all continuous line segments on the centerline. The global spatial location of a crack is typically represented by the three-dimensional coordinates of its point cloud's geometric center (centroid), which directly indicates the location of the damage within the overall bridge structure. Furthermore, spatial relationships can be used to determine the specific component to which the crack belongs (e.g., a box girder or a pier). Then, all the calculated key features—such as crack identifier, length, maximum width, average width, centroid coordinates, component to which it belongs, and approximate orientation—are organized and encapsulated according to a predefined data structure (which can be a multi-dimensional vector, a JSON object, or a database record). This final output structured data object is the three-dimensional damage feature vector, which comprehensively and quantitatively describes the physical and spatial properties of the damage.

[0056] In summary, the image processing-based bridge damage monitoring method according to the embodiments of this application is explained. It extracts the structural contour of a two-dimensional image using a deep convolutional neural network and compares it interactively with the rendered edges generated by perspective projection of a pre-set three-dimensional information model. A closed-loop feedback mechanism is constructed using chamfer distance to iteratively optimize the camera pose, eliminating pose projection drift caused by GPS rejection or weak textures in complex environments. This allows for high-precision mapping of two-dimensional microscopic damage to a three-dimensional macroscopic coordinate system. This approach significantly improves the reliability of damage physical size quantification and global positioning, effectively solving the problems of difficult damage monitoring and positioning and low mapping accuracy under complex inspection conditions, and achieving efficient and seamless integration of detection data and digital modeling.

[0057] Furthermore, an image processing-based bridge damage monitoring system is also provided.

[0058] Figure 4This is a block diagram of an image processing-based bridge damage monitoring system according to an embodiment of this application. Figure 4 As shown, the image processing-based bridge damage monitoring system 300 according to an embodiment of this application includes: a data stream parsing and parameter stripping module 310, used to parse and strip parameters from the raw sensor data stream containing pixel matrices, positioning latitude and longitude, and inertial attitude angles collected by a UAV to obtain a two-dimensional image matrix and an initial pose matrix, while simultaneously acquiring a pre-calibrated camera intrinsic parameter matrix; an image dual-branch high semantic information extraction module 320, used to extract multi-branch high semantic information from the two-dimensional image matrix based on a pre-trained dual-branch deep convolutional neural network to obtain crack segmentation masks and structural edge masks; and a three-dimensional model perspective projection dimensionality reduction rendering module 330, used to extract dimensionality based on the camera intrinsic parameter matrix and... An initial pose matrix is ​​used to construct a perspective projection transformation, and a dimensionality reduction rendering projection is performed on the outer envelope geometric line group of the preset 3D bridge information model to obtain the rendering edge feature lines that reflect the expected bridge skeleton lines under the current viewpoint. An iterative optimization and correction module 340 is used to construct a cost function based on the chamfer distance between the structural edge mask and the rendering edge feature lines, and to iteratively optimize and correct the initial pose matrix to obtain an optimized pose matrix. A dimensionality reduction inverse 3D physical mapping module 350 is used to perform dimensionality reduction inverse 3D physical mapping of each feature pixel in the crack segmentation mask based on the optimized pose matrix and the camera intrinsic parameter matrix to obtain a 3D damage feature vector containing the crack physical size and global spatial positioning.

[0059] As described above, the image processing-based bridge damage monitoring system 300 according to the embodiments of this application can be implemented in various wireless terminals, such as servers with image processing-based bridge damage monitoring algorithms. In one possible implementation, the image processing-based bridge damage monitoring system 300 according to the embodiments of this application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the image processing-based bridge damage monitoring system 300 can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the image processing-based bridge damage monitoring system 300 can also be one of many hardware modules of the wireless terminal.

[0060] Alternatively, in another example, the image processing-based bridge damage monitoring system 300 and the wireless terminal can also be separate devices, and the image processing-based bridge damage monitoring system 300 can be connected to the wireless terminal via wired and / or wireless networks, and transmit interactive information in accordance with an agreed data format.

[0061] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A bridge damage monitoring method based on image processing, characterized in that, include: S1, the raw sensor data stream containing the pixel matrix, positioning latitude and longitude and inertial attitude angles collected by the UAV is parsed and the parameters are stripped to obtain the two-dimensional image matrix and the initial pose matrix, and the pre-calibrated camera intrinsic parameter matrix is ​​obtained at the same time. S2, based on a pre-trained dual-branch deep convolutional neural network, extracts high semantic information from a two-dimensional image matrix through multi-branch image extraction to obtain crack segmentation masks and structural edge masks; S3, based on the camera intrinsic parameter matrix and the initial pose matrix, constructs a perspective projection transformation, performs dimensionality reduction rendering projection on the outer envelope geometric line group of the preset 3D bridge information model to obtain the rendering edge feature lines that reflect the expected bridge skeleton lines under the current view. S4, construct a cost function using the chamfer distance between the structural edge mask and the rendered edge feature line, and iteratively optimize and correct the initial pose matrix to obtain the optimized pose matrix; S5, based on the optimized pose matrix and camera intrinsic parameter matrix, performs dimensionality reduction and inverse 3D physical mapping of each feature pixel in the crack segmentation mask to obtain a 3D damage feature vector containing crack physical size and global spatial localization.

2. The bridge damage monitoring method based on image processing according to claim 1, characterized in that, Step S1 includes: The raw sensor data stream is unpacked frame by frame for verification and load separation to extract the pixel binary load and telemetry metadata. The pixel binary payload is decoded, converted, and reshaped to generate a two-dimensional image matrix; The latitude and longitude parameters in the telemetry metadata are transformed into the local northeast-sky coordinate system to obtain the translation vector, and the rotation matrix is ​​calculated for the three-axis inertial attitude angles to obtain the rotation matrix. The translation vector and the rotation matrix are combined to construct a homogeneous affine transformation matrix to obtain the initial pose matrix.

3. The bridge damage monitoring method based on image processing according to claim 1, characterized in that, Step S2 includes: Hierarchical spatial downsampling and cross-scale lateral fusion extraction are performed on the two-dimensional image matrix to generate a shared feature tensor; The first decoding branch of the dual-branch deep convolutional neural network is used to classify and binarize the shared feature tensor for micro-damage pixels to obtain the crack segmentation mask. The second decoding branch of the dual-branch deep convolutional neural network is used to extract the macroscopic structural contour gradient response of the shared feature tensor to obtain the structural edge mask.

4. The bridge damage monitoring method based on image processing according to claim 1, characterized in that, Step S3 includes: The outer envelope geometric topology analysis and wireframe dimensionality reduction extraction of the 3D bridge information model are performed to obtain a group of straight lines in 3D space; Based on the camera intrinsic parameter matrix and the initial pose matrix, perspective geometric projection mapping is performed on each three-dimensional vertex in the three-dimensional spatial line group to obtain the two-dimensional structure projection line group. Subpixel-level coloring and mask synthesis are performed on the two-dimensional structure projection line group to obtain the rendered edge feature lines.

5. The bridge damage monitoring method based on image processing according to claim 1, characterized in that, Step S4 includes: Based on the structural edge mask and rendered edge feature lines, the edge coincidence error scalar is determined; Jacobi partial derivatives of the edge coincidence error scalar with respect to the six degrees of freedom parameters of the initial pose matrix are calculated to obtain the pose descent gradient vector; The initial pose matrix is ​​iteratively accumulated and corrected based on the pose descent gradient vector, and the manifold is updated to obtain the optimized pose matrix.

6. The bridge damage monitoring method based on image processing according to claim 1, characterized in that, Step S5 includes: Based on the camera intrinsic parameter matrix and the optimized pose matrix, the coordinates of each active pixel in the crack segmentation mask are back-perspective inverse projection to obtain the global spatial ray equation set. Collision and intersection mapping are performed between each ray in the global spatial ray equation set and the mesh surface of the 3D bridge information model to obtain the 3D absolute coordinate point cloud; Physical size quantization calculation and spatial positioning encapsulation are performed on paired points at the crack boundary in a 3D absolute coordinate point cloud to obtain a 3D damage feature vector.

7. The image processing-based bridge damage monitoring method according to claim 5, characterized in that, Based on the structural edge mask and rendered edge feature lines, the edge coincidence error scalar is determined, including: Edge pixel extraction and local tangent direction field estimation are performed on the structural edge mask and rendered edge feature lines to obtain the real edge direction point set and the rendered edge direction point set; Perform orientation consistency matching weight estimation on the real edge orientation point set and the rendered edge orientation point set to obtain the forward matching weight set and the reverse matching weight set; The edge overlap error scalar is obtained by performing point-by-point multiplicative gating weighting and normalized accumulation on the forward and reverse nearest neighbor Euclidean distances based on the forward matching weight set and the reverse matching weight set, respectively.

8. A bridge damage monitoring system based on image processing, characterized in that, include: The data stream parsing and parameter stripping module is used to parse and strip parameters from the raw sensor data stream containing pixel matrices, latitude and longitude coordinates, and inertial attitude angles collected by the UAV to obtain a two-dimensional image matrix and an initial pose matrix, while also acquiring a pre-calibrated camera intrinsic parameter matrix. The image dual-branch high semantic information extraction module is used to extract multi-branch high semantic information from a two-dimensional image matrix based on a pre-trained dual-branch deep convolutional neural network to obtain crack segmentation masks and structural edge masks. The 3D model perspective projection dimensionality reduction rendering module is used to construct perspective projection transformation based on the camera intrinsic parameter matrix and the initial pose matrix, and to perform dimensionality reduction rendering projection on the outer envelope geometric line group of the preset 3D bridge information model to obtain the rendering edge feature lines that reflect the expected bridge skeleton lines under the current view. The iterative optimization and correction module is used to construct a cost function based on the chamfer distance between the structural edge mask and the rendered edge feature line, and to iteratively optimize and correct the initial pose matrix to obtain the optimized pose matrix. The dimension reduction and inverse 3D physical mapping module is used to perform dimension reduction and inverse 3D physical mapping on each feature pixel in the crack segmentation mask based on the optimized pose matrix and camera intrinsic parameter matrix to obtain a 3D damage feature vector containing crack physical size and global spatial location.

Images