An end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints

Abstract

This invention provides an end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints, relating to the field of image processing technology. The method comprises the following steps: S1, establishing a dual-branch encoding-decoding structure; S2, performing cross-dimensional feature fusion on the image global features, point cloud global features, pixel-level features, and point-level features to obtain image-dominant fusion features and point-dominant fusion features; and outputting a pose estimation matrix. S3, establishing a temporal damage prediction model and constraining it using a multidimensional loss function. S4, inputting the registered image pixel-level features, point cloud global features, and relative time information into the temporal damage prediction model and outputting a damage prediction value. This invention demonstrates superior performance in feature extraction, registration accuracy, and robustness, outperforming multiple cutting-edge networks and providing key technical support for multi-scale feature quantization, evaluation, and reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete nodes. Background Art

[0002] Semi-rigid precast concrete (PC) connections offer outstanding seismic designability due to their wide and adjustable connection stiffness. However, their complex, non-integrated, asymmetric, and multi-scale damage characteristics and failure modes pose challenges to damage assessment.

[0003] Existing image-point cloud registration methods and damage prediction technologies still have several key flaws:

[0004] Insufficient registration accuracy: Existing methods (such as algorithms based on feature matching or projection) perform poorly in complex earthquake damage scenarios and have difficulty adapting to the asymmetric deformation and reciprocating motion of semi-rigid nodes, resulting in large errors in damage location and quantification. Multi-scale feature imbalance: The network pays uneven attention to macroscopic deformation (such as beam-column displacement) and microscopic damage (such as fine cracks), and small-scale features are easily overlooked, affecting the comprehensiveness of the assessment. Lack of temporal modeling: Most methods only target static scenarios and cannot capture the time lag effects (such as delayed crack propagation) and cumulative effects of damage under strong earthquakes, resulting in inaccurate dynamic damage prediction. Modal singleness: Reliance on a single data source (only images or only point clouds) limits the accuracy of damage identification—images lack spatial information, and point clouds lack texture details, making it difficult to fully quantify damage. Detachment from physical mechanisms: Existing deep learning models are mostly data-driven black boxes that do not incorporate structural mechanics priors (such as the irreversibility of cracks), which may lead to unrealistic prediction results (such as cracks appearing in compression zones).

[0005] These deficiencies collectively result in poor robustness and limited generalization of existing methods in practical engineering applications, making them incapable of meeting the requirements for high-precision damage assessment of semi-rigid PC nodes under strong earthquakes. Future work will require improvements in the practicality and reliability of these methods through cross-modal fusion registration, joint temporal-multiscale modeling, and physically constrained loss functions. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to propose an end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete nodes to solve the technical problem of poor assessment accuracy of existing damage assessment methods.

[0007] The technical means adopted in the present invention are as follows:

[0008] An end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints includes the following steps:

[0009] S1. Establish a dual-branch encoding-decoding structure, which includes an image branch and a point cloud branch. The time-series images of the semi-rigid precast concrete nodes after the earthquake are passed through the encoder of the image branch to generate image global features, and then through the decoder of the image branch to generate pixel-level features. The time-series point clouds of the post-earthquake images at the same time are passed through the encoder of the point cloud branch to generate point cloud global features, and then through the encoder of the point cloud branch to generate point-level features.

[0010] S2. Perform cross-dimensional feature fusion on the image global features, point cloud global features, pixel-level features, and point-level features to obtain image-dominant fusion features and point-dominant fusion features; obtain feature matching point information based on the image-dominant fusion features and point-dominant fusion features, and output a pose estimation matrix;

[0011] S3. Establish a time-series damage prediction model for semi-rigid precast concrete joints and use a multi-dimensional loss function to constrain the time-series damage prediction model. When the total loss and each loss component converge, output the time-series damage prediction model.

[0012] S4. Input the registered image pixel-level features, point cloud global features, and relative time information into the temporal damage prediction model and output the damage prediction value.

[0013] Furthermore, in S1, the encoder of the image branch includes a ResNet module and a Transformer module;

[0014] The ResNet module is used to extract deep semantics; the ResNet module adopts ResNet34, which includes four residual blocks. The ResNet module extracts multi-level features of the image layer by layer;

[0015] The Transformer module is used to enhance global attention; the Transformer module introduces two layers of Transformers to strengthen attention to local key features through the global self-attention mechanism;

[0016] The decoder of the image branch is used to generate pixel-level features; the decoder of the image branch restores pixel-level features through a three-layer decoding module combined with a jump connection.

[0017] Furthermore, in S1, the encoder of the point cloud branch includes a Set Abstraction module and a PointTransformer module;

[0018] The Set Abstraction module is used to select key points; the Set Abstraction module selects key points by sampling the farthest point and extracts features in the local neighborhood in combination with a multi-layer perceptron;

[0019] The Point Transformer module is used to effectively model the relationship between points using the local self-attention mechanism. The Point Transformer module designs a local self-attention mechanism for point clouds, limits the attention range through K-nearest neighbors, and captures long-range dependencies.

[0020] The decoder of the point cloud branch uses the Feature Propagation module and Point Transformer V to upsample and restore point-level features.

[0021] Furthermore, S2 specifically includes the following steps:

[0022] The global features of the image , global features of point cloud , pixel-level features , point-level features Perform cross fusion to obtain the image-dominant fusion features Point-dominated fusion features , the formula is as follows:

[0023] ;

[0024] Among them, C is the feature channel, D is the dimension; H is the image height; W is the image width; N is the number of point clouds;

[0025] Image-dominated fusion features Use the two-dimensional convolution module to perform downsampling processing, and obtain the pixel-level score matrix after passing through the Sigmoid module. ; The point-dominated fusion feature Use one-dimensional convolution module to perform downsampling processing, and obtain the point-level score vector after passing through the Sigmoid module;

[0026] The feature matching point information is obtained based on the pixel-level score matrix and the point-level score vector. The feature matching point information with a higher score is the feature matching point information with a higher score. The PnP and RANSAC algorithms are used to estimate the rigid body transformation matrix Γ of the feature matching point information with a higher score to obtain the pose estimation matrix.

[0027] Furthermore, the timing damage prediction model in S3 includes:

[0028] Input features, including relative time information t and point cloud global features , pixel-level features F I ;

[0029] A bidirectional long short-term memory module based on a recurrent neural network is used to model spatiotemporal features;

[0030] The time encoding module is used to enhance the network's understanding of relative time and non-uniform time variations, mapping relative time information into a compact time feature vector; the time encoding module consists of two fully connected layers;

[0031] Fusing temporal feature vectors with spatial features to enhance the network’s spatiotemporal understanding capabilities;

[0032] The damage prediction head consists of two fully connected layers.

[0033] Furthermore, the prediction method of the timing damage prediction model in S3 is as follows:

[0034] With F I First, the element-by-element superposition is achieved through the broadcast mechanism, and the spatial encoding module composed of standard convolution modules is passed in for feature downsampling. t Then it is passed into the time coding module for dimension expansion;

[0035] The output features of the spatial encoding module and the temporal encoding module are fused to obtain the spatiotemporal feature F st , the spatiotemporal features are passed into a two-layer bidirectional long short-term memory module based on a recurrent neural network to model the spatiotemporal features;

[0036] The modeled spatiotemporal features are input into a prediction module consisting of two fully connected layers, which outputs timing damage to achieve the prediction of timing damage.

[0037] Furthermore, in S3, the loss components of the multi-dimensional loss function include: multi-scale feature similarity loss, differentiable pose loss, and hysteresis damage loss;

[0038] Multi-scale feature similarity loss includes boundary feature loss and coarse / fine registration loss;

[0039] Boundary feature loss l b As follows: In the two fusion features and The score is below the preset threshold s The features are considered to be outside the overlapping area and are regarded as abnormal features; l b It consists of two parts, namely the loss associated with the overlapping area and the loss associated with the abnormal feature area; the loss associated with the overlapping area is used to make the network pay more attention to the features in the overlapping area, and the loss associated with the abnormal feature area is used to make the network stay away from the abnormal feature area; l b The calculation method is as follows:

[0040] ;

[0041] in, is the score of the i-th image feature in the overlapping area; is the score of the j-th point cloud feature in the overlapping area; is the score of the k-th image feature in the abnormal area; is the score of the m-th point cloud feature in the abnormal area; Z I, in represents the number of image features in the overlapping area, Z P, in Indicates the number of point cloud features in the overlapping area; Z I, out represents the number of image features in the abnormal area, Z P, out Indicates the number of point cloud features within the abnormal area;

[0042] Coarse / fine registration loss l c / l f as follows: l c Used to calculate the feature reprojection distance between point cloud and image oh and through the preset safety radius r c Define positive feature pairs and negative feature pairs in the matrix mask; use distance function D (·)=1- s (·) Calculate the distance between the positive feature pairs D pos (.) Distance to negative feature pair D neg (.); the coarse registration feature loss is defined as follows:

[0043] ;

[0044] in, n is the number of positive feature pairs in the coarse registration; M pos is the positive eigenpair boundary value; p is the number of negative feature pairs in the coarse registration; M neg is the negative feature pair boundary value;

[0045] Adopting an adaptive weighted optimization loss strategy to learn the correspondence of precise features, combined with feature reprojection distance oh , using a safety radius r f Divide positive samples into negative samples; the definition of fine registration feature loss is as follows:

[0046] ;

[0047] in, q The number of positive feature pairs for fine registration; For the i Similarity of positive feature pairs; v The number of negative feature pairs for fine registration;

[0048] Positive adaptive weight factor r p , negative adaptive weight factor r n The calculation method is as follows:

[0049] ;

[0050] in, is the adaptive weight function; c is a scaling factor; M Fine feature separation boundary value; q and v Represent the number of positive feature pairs and negative feature pairs respectively; For the j Similarity of negative feature pairs;

[0051] Differentiable pose loss l pose As follows: Combining the pose error and the confidence distribution constraint, the differentiable pose loss function l pose The definition is as follows:

[0052] ;

[0053] Among them, H( O )=- O log O -(1- O )log(1- O ), O is the confidence weight matrix, R pred To predict the rotation matrix, R gt is the real rotation matrix, T pred To predict the translation vector, T gt is the real translation vector;

[0054] Hysteresis damage loss l d as follows:

[0055] The damage index based on stiffness degradation is used to describe the performance degradation phenomenon during the hysteresis process. The specific calculation method is as follows:

[0056] ;

[0057] in, K initialrepresents the tangential stiffness of the semi-rigid precast concrete node before damage, K final represents the tangential stiffness of the semi-rigid precast concrete joint after damage;

[0058] The real damage state evolution of semi-rigid precast concrete joints in the hysteretic process is studied based on the formula D gt The Huber loss is used to define the overall time series damage loss function. In the time series damage prediction task, the Huber loss can stabilize the deep network training by balancing the error sensitivity and suppressing the interference of outliers. The hysteresis damage loss function l d The specific definition is as follows:

[0059] ;

[0060] Among them, D pred is the damage state predicted by the network, D gt is the true damage state predicted by the network, δ is the preset threshold that controls the loss function to switch between MSE and MAE;

[0061] The total loss of the multi-dimensional loss function is as follows:

[0062] ;

[0063] Among them, λ b ,λ c ,λ f ,λ pose and λ d l b 、l c 、l f 、l pose and l d The loss component weight coefficient.

[0064] The present invention also provides a storage medium comprising a stored program, wherein when the program is run, any of the above-mentioned end-to-end image-point cloud registration and damage prediction methods for semi-rigid precast concrete nodes is executed.

[0065] The present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes any of the above-mentioned end-to-end image-point cloud registration and damage prediction methods for semi-rigid precast concrete nodes through the computer program.

[0066] Compared with the prior art, the present invention has the following advantages:

[0067] In the study of semi-rigid precast concrete nodes, the advantages of image and point cloud data were integrated to construct a network that achieved high-precision registration and damage prediction. The network's image-point cloud comprehensive registration error is less than 2mm, and it can take into account the quantitative analysis of large-scale boundaries and small-scale features, providing strong support for multi-scale damage state assessment and three-dimensional reconstruction. In terms of temporal damage prediction, its error is less than 6.81%, and the error at most time points is within the limit of 15%, which can accurately assess the damage state of semi-rigid precast concrete nodes. After comparison and verification, the network has excellent performance in feature extraction, registration accuracy and robustness, and is superior to many cutting-edge networks, providing key technical support for multi-scale feature quantification, evaluation and reconstruction.

[0068] This paper establishes a dual-branch structure of image and point cloud. In the image branch, ResNet+Transformer extracts multi-scale deep features, enhancing attention to local key features (such as cracks). In the point cloud branch, PointNet++PointTransformer preserves geometric structure information and handles local long-range dependencies of the point cloud. This paper balances semantic alignment and geometric accuracy through cross-fusion (concat) of global features and upsampled features. This paper considers both boundary registration accuracy and local feature matching (such as microscopic cracks). This paper adapts to the disorder of point clouds and enhances the robustness of cross-modal matching. This paper implements gradient backpropagation through a differentiable pose loss (Gumbel Softmax+PnP), improving registration stability.

[0069] This method establishes a damage model (time series prediction) and inputs registered time series features (image upsampled features, point cloud global features, and time encoding). Using BiLSTM, it captures temporal dependencies and combines them with physical constraints (stiffness degradation loss) to model damage evolution. The results are as follows: Time lag effects are modeled, and bidirectional LSTM correlates historical and future information, resolving the issue of apparent damage lag. Physical mechanism enhancement: The introduction of stiffness degradation loss improves the network's understanding of physical laws and avoids overfitting that would be driven purely by data. Closed-loop optimization: Damage prediction is fed back into the registration phase, and overall accuracy is improved through feature similarity constraints (such as loss weight adjustment). BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0071] Figure 1 Flow chart of the method of the present invention.

[0072] Figure 2 This is a comparison diagram of the Point Transformer and Transformer architectures of the present invention.

[0073] Figure 3 This is a graph comparing the pose prediction accuracy of the network for experimental and simulation data. DETAILED DESCRIPTION

[0074] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0075] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0076] like Figure 1 As shown, the present invention provides an end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete nodes, comprising the following steps:

[0077] S1. Establish a dual-branch encoding-decoding structure, which includes an image branch and a point cloud branch. The time-series images of the semi-rigid precast concrete nodes after the earthquake are passed through the encoder of the image branch to generate image global features, and then through the decoder of the image branch to generate pixel-level features. The time-series point clouds of the post-earthquake images at the same time are passed through the encoder of the point cloud branch to generate point cloud global features, and then through the encoder of the point cloud branch to generate point-level features.

[0078] S2. Perform cross-dimensional feature fusion on the image global features, point cloud global features, pixel-level features, and point-level features to obtain image-dominant fusion features and point-dominant fusion features; obtain feature matching point information based on the image-dominant fusion features and point-dominant fusion features, and output a pose estimation matrix;

[0079] Image branch: ResNet+Transformer extracts multi-level features (ResNet34 extracts deep semantics, Transformer enhances global attention), and the decoder generates pixel-level features.

[0080] Encoder:

[0081] 1. ResNet module: ResNet34 is used, which includes four residual blocks (ResNet34 I-IV). It extracts multi-level features of the image layer by layer, and the output dimensions are increased from (B, 64, H / 2, W / 2) to (B, 512, H / 16, W / 16) to avoid gradient vanishing.

[0082] 2. Transformer module: Introducing a two-layer Transformer (Transformer I-II), using a global self-attention mechanism to strengthen attention to local key features, while maintaining the output dimensions (B, 512, H / 16, W / 16).

[0083] Decoder: Through 3 layers of decoding modules (decoding modules I-III), combined with skip connections, pixel-level features are restored, and the output dimension is upsampled from (B, 256, H / 8, W / 8) to (B, 64, H / 2, W / 2).

[0084] The image passes through the network's encoder to generate global features , and then generates rich pixel-level features through the decoder part with skip connections .

[0085] Point cloud branch: Based on PointNet++ and Point Transformer, the Set Abstraction module filters key points, extracts local geometric features (such as normal vectors and curvature), and the decoder recovers point-level features.

[0086] Encoder: 1. Set Abstraction (SA) module: SA-Ⅰ / Ⅱ / Ⅲ: Filter key points through farthest point sampling (FPS) and extract features in local neighborhoods in combination with multi-layer perceptron (MLP).

[0087] Output dimension changes: SA-Ⅰ output (B, 64, 1024, 64), SA-Ⅱ output (B, 131, 256, 32), SA-Ⅲ output (B, 259, 1, 256), gradually reducing the number of point clouds and concentrating features.

[0088] 2. Point Transformer module:

[0089] Point TransformerⅠ-Ⅱ: Design a local self-attention mechanism for point clouds, limit the attention range through K-nearest neighbors, and capture long-range dependencies.

[0090] Output dimensions: Point Transformer I output (B, 128, 1024), Point Transformer II output (B, 256, 256).

[0091] Decoder: Through the Feature Propagation (FP) module and Point Transformer V, upsampling is performed to restore point-level features, and the output dimension is increased from (B, 256, 256) to (B, 64, 1024).

[0092] The temporal point cloud can establish global features through the encoding layer Similarly, rich point-level features are generated through a decoder with skip connections .

[0093] Cross-dimensional feature fusion: The cross-dimensional registration of images and point clouds is achieved by generating global features through the encoding layer ( and ) and the upsampled features generated by the decoding layer (i.e., pixel-level features F I and point-level features F P ). The global features extracted by global pooling or Transformer operations can provide highly condensed cross-modal semantic alignment information, further suppress the impact of noise on registration, and ensure the stability of the overall pose estimation. Since the feature information carried by the upsampled features is more comprehensive and rich, the features after cross fusion are dominated by the upsampled features, so the fused features are image-dominant fusion features. Point-dominated fusion features ,The fusion method adopts the method of superposition along the feature dimension, and the fusion process is shown in the following formula.

[0094] ;

[0095] Among them, C is the feature channel, D is the dimension; H is the image height; W is the image width; N is the number of point clouds;

[0096] Obtained and Use two-dimensional convolution module and one-dimensional convolution module for downsampling respectively, and obtain the corresponding pixel-level score matrix after passing through the Sigmoid module. The point-level score vector is used to describe the importance of each feature used for registration and is also an important indicator for the subsequent key point selection. Combined with the feature matching point information with high scores, a score greater than 0.95 is defined as a high score. The PnP and RANSAC algorithms are used to estimate the rigid body transformation matrix Γ.

[0097] S3. Establish a time-series damage prediction model for semi-rigid precast concrete joints and use a multi-dimensional loss function to constrain the time-series damage prediction model. When the total loss and each loss component converge, output the time-series damage prediction model.

[0098] S4. Input the registered image pixel-level features, point cloud global features, and relative time information into the temporal damage prediction model and output the damage prediction value.

[0099] Timing damage prediction:

[0100] Input: pixel-level features of the registered image, global features of the point cloud, and relative time information.

[0101] Model: BiLSTM captures spatiotemporal correlations, the time encoding module processes non-uniformly spaced time series, and the fully connected layer outputs damage prediction values.

[0102] In view of the inherent sparsity of post-earthquake images and point cloud time series data and the time lag effect of internal degradation, this study uses a bidirectional long short-term memory (BiLSTM) module based on a recurrent neural network to model its spatiotemporal features. In addition, in order to enhance the network's ability to understand relative time and non-uniformly spaced time variations, a time encoding module consisting of two fully connected layers is introduced to map relative time information into a compact time feature vector. The time feature vector is then fused with the spatial feature to enhance the network's spatiotemporal understanding ability. Considering the advantages of information expression and the alignment network structure, the relative time (t), the global features of the point cloud ( The final upsampled features (FI) of the image are used as input features for the time series prediction stage. The final damage prediction head consists of two fully connected layers. This completes the overall network structure.

[0103] Select relative time (t), global features of point cloud ( ), the final upsampled features (FI) of the image are used as input features in the time series prediction stage. FI first implements element-by-element superposition through a broadcasting mechanism. The feature is then fed into a spatial encoding module composed of standard convolutional modules for feature downsampling. The output features of the spatial and temporal encoding modules are then fused to obtain spatiotemporal features (Fst). This is then fed into a two-layer BiLSTM module for spatiotemporal feature modeling. Finally, this is fed into a prediction module composed of two fully connected layers to predict temporal impairments.

[0104] Multi-dimensional loss function design:

[0105] Loss amount:

[0106] 1. Multi-scale feature similarity loss:

[0107] Boundary feature loss ( l b ): In the two fusion features and The score is below the preset threshold s Those features of are considered to be outside the overlapping area and are considered as abnormal features. l b It consists of two parts, namely the loss associated with the overlapping area and the loss associated with the abnormal feature area. The former is used to make the network pay more attention to the features in the overlapping area, while the latter is used to keep the network away from the abnormal feature area. l b The calculation method is as follows:

[0108] ;

[0109] in, is the score of the i-th image feature in the overlapping area; is the score of the j-th point cloud feature in the overlapping area; is the score of the k-th image feature in the abnormal area; is the score of the m-th point cloud feature in the abnormal area; Z I, in and Z P, in Respectively represent the number of image features and point cloud features in the overlapping area; Z I, out and Z P, out Represent the number of image features and point cloud features in the abnormal area respectively.

[0110] Coarse / fine registration loss ( l c / l f ):

[0111] l c Used to calculate the feature reprojection distance between point cloud and image oh , and through a preset safety radius r c Define positive eigenpairs ( oh <rc) and negative eigenpairs (ω > rc) in the matrix mask. The distance function D (·) = 1 - s (·) is used to calculate the determined positive eigenpair distance D pos (.) and the negative eigenpair distance D neg (.). The definition of the rough registration feature loss is as follows:

[0112] ;

[0113] where n is the number of positive eigenpairs in rough registration; M pos is the boundary value of the positive eigenpair; p is the number of negative eigenpairs in rough registration; M neg is the boundary value of the negative eigenpair;

[0114] In order to obtain more accurate pixel - point registration pairs for pose regression, the present invention introduces an optimized loss strategy with adaptive weights to learn the corresponding relationship of its precise features. To achieve high - precision registration at the small - scale feature level, combined with the feature reprojection distance oh , a smaller safety radius r f is used to divide positive samples ( oh < r f ) and negative samples (ω > r f ). The definition of the fine registration feature loss is as follows:

[0115] ;

[0116] where q is the number of positive eigenpairs in fine registration; is the i th positive eigenpair similarity; <00-00446>is the number of negative eigenpairs in fine registration;

[0117] Positive and negative adaptive weight factors r p and r n are calculated as follows:

[0118] ;

[0119] where is the adaptive weight function; c is a scaling factor; M Fine feature separation boundary value; q and v Represent the number of positive feature pairs and negative feature pairs respectively; For the j Negative feature pair similarity.

[0120] 2. Differentiable pose loss ( l pose ):

[0121] This paper proposes a differentiable framework that combines differentiable probability sampling, weighted PnP optimization and joint loss function design to realize an end-to-end image-point cloud registration network.

[0122] Combining the pose error and the confidence distribution constraint, the differentiable pose loss function ( l pose ) is defined as follows:

[0123] ;

[0124] Among them, H( O )=- O log O -(1- O )log(1- O ), O is the confidence weight matrix, R pred With R gt are the predicted and true rotation matrices, T pred With T gt are the predicted and true translation vectors, respectively.

[0125] 3. Hysteresis damage loss ( l d ):

[0126] Under the action of reciprocating loads, the strength, stiffness and energy dissipation capacity of semi-rigid PC nodes will show significant degradation, and this process will be accompanied by obvious overall deformation, local cracks and surface peeling and other apparent phenomena. There must be some implicit mapping relationships between physical degradation and surface features. Based on this, the introduction of a time damage prediction module and the corresponding loss function related to hysteresis damage should help enhance the network's ability to understand these mapping relationships. Based on this, the present invention uses a damage index based on stiffness degradation to describe its performance degradation phenomenon during the hysteresis process. The specific calculation method is as follows:

[0127] ;

[0128] in, Kinitial Initial and K final Finally, they represent the tangential stiffness of the semi-rigid PC node before and after damage.

[0129] The real damage state evolution of the hysteresis process of semi-rigid PC nodes is analyzed based on the formula D gt The Huber loss is used to define the overall time series damage loss function. The Huber loss is a loss function that combines the mean square error (MSE) and the mean absolute error (MAE). It uses a quadratic function when the error is small and a linear function when the error is large. This can take into account both the sensitivity to small errors and the robustness to large errors. In the task of time series damage prediction, the Huber loss can stabilize the deep network training by balancing the error sensitivity and suppressing the interference of outliers. Hysteresis damage loss function ( l d ) The specific definition is as follows:

[0130] ;

[0131] Among them, D pred With D gt are the damage state predicted by the network and the actual damage state, respectively, and δ is the preset threshold that controls the loss function to switch between MSE and MAE.

[0132] Total loss and stage-by-stage training:

[0133] Total loss:

[0134] ;

[0135] Among them, λb, λc, λf, λpose and λd are the loss component weight coefficients of lb, lc, lf, lpose and ld respectively.

[0136] Adjust weights in stages: This invention adopts a staged training method, which divides the entire training process into three stages, namely the posture prediction stage, the damage prediction stage and the balance adjustment stage. l b 、 l c 、 l f 、 l pose and l dThe values ​​of are [1, 1, 0.5, 1, 0.1]; [0.5, 1, 0.5, 1, 0.6]; [0.1, 1, 0.5, 1, 1]. The setting of this parameter is mainly based on the importance of each component, the changing trend of each loss component and the rationality of the overall loss development law, and is set through enumeration method after multiple attempts.

[0137] Figure 1 In the first phase, the network is primarily used for image-point cloud registration. In this phase, the network is divided into two branches, one for processing the two-dimensional image I and the other for processing the three-dimensional point cloud P at the corresponding moment. Here, t is the time dimension, C is the number of image channels (typically 3), H and W are the height and width of the image, and N is the number of points. T is the translation matrix, and R is the rotation matrix. The network's multi-scale characteristics are enhanced by introducing an attention mechanism and combining it with targeted loss function design. The second phase is primarily used for the prediction of time-series earthquake damage to semi-rigid PC nodes. The network aims to establish a module with time-series modeling capabilities to incorporate the time-series damage evolution and its physical mechanisms into the network.

[0138] like Figure 2 As shown in the figure, the backbone network for the image-point cloud registration stage established in this paper consists of two encoding-decoding branches, one for extracting deep features from the image and the other for extracting deep features from the point cloud data. The image branch is composed of a ResNet module and a Transformer module. The image passes through the network's encoder to generate global features, which are then passed through the decoder with skip connections to generate rich pixel-level features. The point cloud feature processing branch also uses an encoding-decoding architecture, primarily consisting of a Set Abstraction module and a Point Transformer module. The temporal point cloud is passed through the encoding layer to generate global features, and then through the decoder with skip connections to generate rich point-level features.

[0139] like Figure 3 As shown in the figure, to intuitively assess the degree of matching between deep features in the image and point cloud, the t-SNE module is used to visualize feature matching in the latent 2D space based on cosine similarity. This paper performs t-SNE visualization on all time series samples, but only five equally spaced time series samples from the start to the end of time are shown in the figure. As can be seen from the figure, while the pose estimation accuracy of all networks in the real test scene is slightly lower than that of the simulated data, the accuracy gap between the experimental and simulated samples is not large, demonstrating the effectiveness of the dataset established in this paper.

[0140] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints, characterized by: The steps include: S1. Establishing a dual-branch encoding-decoding structure, the dual-branch encoding-decoding structure includes an image branch and a point cloud branch; allowing the time-series image of the semi-rigid precast concrete node after the earthquake to pass through the encoder of the image branch to generate image global features, and then pass through the decoder of the image branch to generate pixel-level features; The time series point cloud of the post-earthquake image at the same time is passed through the encoder of the point cloud branch to generate point cloud global features, and then passed through the encoder of the point cloud branch to generate point-level features; The encoder of the image branch includes a ResNet module and a Transformer module; The ResNet module is used to extract deep semantics; the ResNet module adopts ResNet34, which includes four residual blocks. The ResNet module extracts multi-level features of the image layer by layer; The Transformer module is used to enhance global attention; the Transformer module introduces two layers of Transformers to strengthen attention to local key features through the global self-attention mechanism; The decoder of the image branch is used to generate pixel-level features; the decoder of the image branch recovers pixel-level features through a three-layer decoding module combined with a skip connection; The encoder of the point cloud branch includes a Set Abstraction module and a Point Transformer module; The Set Abstraction module is used to select key points; the Set Abstraction module selects key points by sampling the farthest point and extracts features in the local neighborhood in combination with a multi-layer perceptron; The Point Transformer module is used to effectively model the relationship between points using the local self-attention mechanism. The Point Transformer module designs a local self-attention mechanism for point clouds, limits the attention range through K-nearest neighbors, and captures long-range dependencies. The decoder of the point cloud branch uses the Feature Propagation module and Point Transformer V to upsample and restore point-level features; S2. Perform cross-dimensional feature fusion on the image global features, point cloud global features, pixel-level features, and point-level features to obtain image-dominant fusion features and point-dominant fusion features; Based on the image-dominant fusion features and point-dominant fusion features, feature matching point information is obtained and the pose estimation matrix is ​​output; S3. Establish a time-series damage prediction model for semi-rigid precast concrete joints and use a multi-dimensional loss function to constrain the time-series damage prediction model. When the total loss and each loss component converge, output the time-series damage prediction model. The time series damage prediction model includes: Input features, including relative time information t and point cloud global features , pixel-level features F I ; A bidirectional long short-term memory module based on a recurrent neural network is used to model spatiotemporal features; The time encoding module is used to enhance the network's understanding of relative time and non-uniform time variations, mapping relative time information into a compact time feature vector; the time encoding module consists of two fully connected layers; Fusing temporal feature vectors with spatial features to enhance the network’s spatiotemporal understanding capabilities; The damage prediction head consists of two fully connected layers; S4. Input the registered image pixel-level features, point cloud global features, and relative time information into the temporal damage prediction model and output the damage prediction value.

2. The end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints according to claim 1 is characterized in that: S2 specifically includes the following steps: The global features of the image , global features of point cloud , pixel-level features , point-level features Perform cross fusion to obtain the image-dominant fusion features Point-dominated fusion features , the formula is as follows: Among them, C is the feature channel, D is the dimension; H is the image height; W is the image width; N is the number of point clouds; Image-dominated fusion features Use the two-dimensional convolution module to perform downsampling processing, and obtain the pixel-level score matrix after passing through the Sigmoid module. ; The point-dominated fusion feature Use one-dimensional convolution module to perform downsampling processing, and obtain the point-level score vector after passing through the Sigmoid module; The feature matching point information is obtained based on the pixel-level score matrix and the point-level score vector. The feature matching point information with a higher score is the feature matching point information with a higher score. The PnP and RANSAC algorithms are used to estimate the rigid body transformation matrix Γ of the feature matching point information with a higher score to obtain the pose estimation matrix.

3. The end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints according to claim 1 is characterized in that: The prediction method of the timing damage prediction model in S3 is as follows: With F I First, the element-by-element superposition is achieved through the broadcast mechanism, and the spatial encoding module composed of standard convolution modules is passed in for feature downsampling. t Then it is passed into the time coding module for dimension expansion; The output features of the spatial encoding module and the temporal encoding module are fused to obtain the spatiotemporal feature F st , the spatiotemporal features are passed into a two-layer bidirectional long short-term memory module based on a recurrent neural network to model the spatiotemporal features; The modeled spatiotemporal features are input into a prediction module consisting of two fully connected layers, which outputs timing damage to achieve the prediction of timing damage.

4. The end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete joints according to claim 1 is characterized in that: In S3, the loss components of the multi-dimensional loss function include: multi-scale feature similarity loss, differentiable pose loss, and hysteresis damage loss; Multi-scale feature similarity loss includes boundary feature loss and coarse / fine registration loss; Boundary feature loss l b As follows: In the two fusion features and The score is below the preset threshold σ The features are considered to be outside the overlapping area and are regarded as abnormal features; l b It consists of two parts, namely the loss associated with the overlapping area and the loss associated with the abnormal feature area; the loss associated with the overlapping area is used to make the network pay more attention to the features in the overlapping area, and the loss associated with the abnormal feature area is used to make the network stay away from the abnormal feature area; l b The calculation method is as follows: in, is the score of the i-th image feature in the overlapping area; is the score of the j-th point cloud feature in the overlapping area; is the score of the k-th image feature in the abnormal area; is the score of the mth point cloud feature in the abnormal area; Z I, in represents the number of image features in the overlapping area, Z P, in Indicates the number of point cloud features in the overlapping area; Z I, out represents the number of image features in the abnormal area, Z P, out Indicates the number of point cloud features within the abnormal area; Coarse / fine registration loss l c / l f as follows: l c Used to calculate the feature reprojection distance between point cloud and image ω and through the preset safety radius r c Define positive feature pairs and negative feature pairs in the matrix mask; use distance function D (·)=1- s (·) Calculate the distance between the positive feature pairs D pos (.) Distance from negative feature pair D neg (.); the coarse registration feature loss is defined as follows: in, n is the number of positive feature pairs in the coarse registration; M pos is the positive eigenpair boundary value; p is the number of negative feature pairs in the coarse registration; M neg is the negative feature pair boundary value; Adopting an adaptive weighted optimization loss strategy to learn the correspondence of precise features, combined with feature reprojection distance ω , using a safety radius r f Divide positive samples into negative samples; the definition of fine registration feature loss is as follows: in, q The number of positive feature pairs for fine registration; For the i Similarity of positive feature pairs; v The number of negative feature pairs for fine registration; Positive adaptive weight factor ρ p , negative adaptive weight factor ρ n The calculation method is as follows: in, is the adaptive weight function; γ is a scaling factor; M Fine feature separation boundary value; q and v Represent the number of positive feature pairs and negative feature pairs respectively; For the j Similarity of negative feature pairs; Differentiable pose loss l pose As follows: Combining the pose error and the confidence distribution constraint, the differentiable pose loss function l pose The definition is as follows: Among them, H( O )=- O log O -(1- O )log(1- O ), O is the confidence weight matrix, R pred To predict the rotation matrix, R gt is the real rotation matrix, T pred To predict the translation vector, T gt is the real translation vector; Hysteresis damage loss l d as follows: The damage index based on stiffness degradation is used to describe the performance degradation phenomenon during the hysteresis process. The specific calculation method is as follows: in, K initial represents the tangential stiffness of the semi-rigid precast concrete node before damage, K final represents the tangential stiffness of the semi-rigid precast concrete joint after damage; The real damage state evolution of semi-rigid precast concrete joints in the hysteretic process is studied based on the formula D gt The Huber loss is used to define the overall time series damage loss function. In the time series damage prediction task, the Huber loss stabilizes the deep network training by balancing the error sensitivity and suppressing the interference of outliers. The hysteresis damage loss function l d The specific definition is as follows: Among them, D pred is the damage state predicted by the network, D gt is the true damage state predicted by the network, δ is the preset threshold that controls the loss function to switch between MSE and MAE; The total loss of the multi-dimensional loss function is as follows: Among them, λ b ,λ c ,λ f ,λ pose and λ d l b 、l c 、l f 、l pose and l d The loss component weight coefficient.

5. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is run, the end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete nodes according to any one of claims 1 to 4 is executed.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: The processor executes the end-to-end image-point cloud registration and damage prediction method for semi-rigid precast concrete nodes according to any one of claims 1 to 4 by running the computer program.

Images