Three-dimensional reconstruction and damage detection method for intelligent driving environment building

Abstract

The invention relates to the technical field of building geometric reconstruction and crack detection, in particular to a three-dimensional reconstruction and damage detection method for a building in an intelligent driving environment, and the method comprises the steps: collecting a building video in a road environment during intelligent driving, and reconstructing a sparse point cloud of the building through an SfM algorithm; inputting each frame of time sequence image into a pre-trained semantic segmentation model to generate a crack highlight image; according to the sparse point cloud, initializing a corresponding parameterized 3D Gaussian ellipsoid, and optimizing parameters of the 3D Gaussian ellipsoid; mapping the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model, and generating a rendered image through a differentiable rendering algorithm; and comparing the rendered image with the crack highlight image, iteratively calculating the overall loss, and transmitting the overall loss through a back propagation gradient until a preset convergence condition is reached, thereby obtaining a final 3D model which simultaneously expresses the overall structure of the building and crack details. According to the invention, accurate environment data can be provided for the automatic driving system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of building geometry reconstruction and crack detection, and particularly relates to a three-dimensional reconstruction and damage detection method for buildings in an intelligent driving environment. BACKGROUND

[0002] At present, building 3D reconstruction technology mainly relies on structured light, laser scanning and image-based reconstruction methods. Among them, image-based reconstruction technology is widely used due to its lower cost and wide applicability. In recent years, deep learning technology has made great progress in image semantic segmentation field, especially the segmentation model based on convolutional neural network (CNN), such as UNet, FCN, etc., which has shown excellent performance in crack detection.

[0003] The prior art has relatively mature implementation schemes in building 3D reconstruction and crack detection, but in the intelligent driving application, the existing building 3D reconstruction method cannot introduce the detection and feedback of the subtle damage (such as cracks) on the surface of the building in the reconstruction process at the same time, and it is difficult to realize real-time monitoring and updating of buildings and infrastructure in a complex road environment.

[0004] At present, there is a lack of end-to-end differentiable optimization process, so that 3D model reconstruction cannot be directly guided by two-dimensional crack detection results. In the intelligent driving scene, how to effectively feed back the crack detection results to the 3D reconstruction model in order to provide accurate environmental data for the autonomous driving system is a key challenge. SUMMARY

[0005] Therefore, it is necessary to provide a three-dimensional reconstruction and damage detection method for buildings in an intelligent driving environment, which can provide accurate environmental data for the autonomous driving system.

[0006] In a first aspect, the present application provides a three-dimensional reconstruction and damage detection method for buildings in an intelligent driving environment, which comprises: Collecting a building video in a road environment during intelligent driving, dividing the building video into time sequence images, and reconstructing a sparse point cloud of the building by performing feature extraction, matching and camera pose estimation on the time sequence images through an SfM algorithm; Inputting each frame of time sequence image into a pre-trained semantic segmentation model, detecting and marking the crack area to generate a crack highlight image; Initializing a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud, and optimizing the 3D Gaussian ellipsoid parameters according to the local information of the time sequence image; Mapping the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters, and generating a rendering image through a differentiable rendering algorithm. The rendered image is compared with the crack highlight image, an overall loss is iteratively calculated, the overall loss is transmitted by back propagation of a gradient and the 3D Gaussian ellipsoid parameters are updated until a preset convergence condition is reached, and a final 3D model is obtained, which simultaneously represents the overall structure and crack details of the building.

[0007] In one of the embodiments, the inputting of each frame of the time sequence image into the pre-trained semantic segmentation model, the detection and marking of the crack region, and the generation of the crack highlight image comprise: The inputting of each frame of the time sequence image into the pre-trained semantic segmentation model to obtain a pixel-level segmentation map of the crack region; The fusion of the pixel-level segmentation map and the original time sequence image, the marking of the crack region with a preset color, and the generation of the crack highlight image.

[0008] In one of the embodiments, the local information of the time sequence image comprises local color, brightness, and texture information of the time sequence image; The initialization of a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud, and the optimization of the 3D Gaussian ellipsoid parameters according to the local information of the time sequence image comprise: The initialization of a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud; The pre-optimization of the 3D Gaussian ellipsoid parameters according to the local color, brightness, and texture information of the time sequence image, so that the 3D Gaussian ellipsoid is more fitted to the actual building surface features; The fitting and optimization of the 3D Gaussian ellipsoid parameters by the least square method.

[0009] In one of the embodiments, the differentiable rendering algorithm is an alpha blending method based on Gaussian distribution; The mapping of the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters, and the generation of a rendered image through a differentiable rendering algorithm comprise: The mapping of the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters; The rasterization of the two-dimensional image plane through the alpha blending method based on Gaussian distribution to sort the 3D Gaussian ellipsoids overlaid on the same two-dimensional image plane pixel; The calculation of the contribution of the 3D Gaussian ellipsoid on the two-dimensional image plane pixel according to the sorting, and the obtaining of the color value of each pixel on the two-dimensional image plane according to the contribution of the 3D Gaussian ellipsoid on the two-dimensional image plane pixel to generate a rendered image.

[0010] In one of the embodiments, the comparing the rendered image and the crack highlight image, iteratively calculating a whole loss, passing the whole loss by back propagation of gradients and updating the 3D Gaussian ellipsoid parameters until a preset convergence condition is reached, obtaining a final 3D model simultaneously expressing the overall structure and crack details of the building includes: comparing the rendered image and the crack highlight image, iteratively calculating a pixel-level difference loss, a crack area key loss and a regularization loss; weighting and summing the pixel-level difference loss, the crack area key loss and the regularization loss to obtain a whole loss; passing the whole loss by back propagation of gradients and updating the 3D Gaussian ellipsoid parameters until a preset convergence condition is reached, obtaining a final 3D model simultaneously expressing the overall structure and crack details of the building.

[0011] In one of the embodiments, the passing the whole loss by back propagation of gradients and updating the 3D Gaussian ellipsoid parameters until a preset convergence condition is reached, obtaining a final 3D model simultaneously expressing the overall structure and crack details of the building includes: passing the whole loss gradient to the corresponding 3D Gaussian ellipsoid parameters by back propagation; updating the 3D Gaussian ellipsoid parameters by an Adam optimizer until a preset convergence condition is reached, obtaining a final 3D model simultaneously expressing the overall structure and crack details of the building.

[0012] In one of the embodiments, the feature extraction, matching and camera pose estimation of the time sequence images by the SfM algorithm, and the reconstruction of the sparse point cloud of the building includes: extracting feature points from each frame of time sequence image by a local feature extraction algorithm, and realizing feature correspondence between adjacent or overlapping time sequence images by a feature matching algorithm to construct a feature point matching relationship; estimating the camera internal and external parameters of each frame of time sequence image according to the matching features by the SfM algorithm; reconstructing the sparse point cloud of the building according to the camera internal and external parameters and the feature point matching relationship.

[0013] In a second aspect, the present application also provides a device for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment. The device includes: a sparse point cloud reconstruction module for collecting building videos in a road environment during intelligent driving, cutting the building videos into time sequence images, and reconstructing a sparse point cloud of the building by feature extraction, matching and camera pose estimation of the time sequence images by the SfM algorithm; a crack highlight image generation module, configured to input each frame of the time sequence image into a pre-trained semantic segmentation model, detect and mark a crack region, and generate a crack highlight image; a Gaussian ellipsoid generation module, configured to initialize a corresponding parametric 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud, and optimize the 3D Gaussian ellipsoid parameters according to local information of the time sequence image; a rendered image generation module, configured to map the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters, and generate a rendered image through a differentiable rendering algorithm; a 3D model generation module, configured to compare the rendered image with the crack highlight image, iteratively calculate an overall loss, transfer the overall loss through a back propagation gradient, update the 3D Gaussian ellipsoid parameters, and obtain a final 3D model that simultaneously expresses the overall structure and crack details of the building until a preset convergence condition is reached.

[0014] In a third aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program: collecting a building video in a road environment during intelligent driving, cutting the building video into time sequence images, and performing feature extraction, matching and camera pose estimation on the time sequence images through an SfM algorithm to reconstruct a sparse point cloud of the building; inputting each frame of the time sequence image into a pre-trained semantic segmentation model, detecting and marking a crack region, and generating a crack highlight image; initializing a corresponding parametric 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud, and optimizing the 3D Gaussian ellipsoid parameters according to local information of the time sequence image; mapping the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters, and generating a rendered image through a differentiable rendering algorithm; comparing the rendered image with the crack highlight image, iteratively calculating an overall loss, transferring the overall loss through a back propagation gradient, updating the 3D Gaussian ellipsoid parameters, and obtaining a final 3D model that simultaneously expresses the overall structure and crack details of the building until a preset convergence condition is reached.

[0015] In a fourth aspect, the present application further provides a computer readable storage medium. The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the following steps: Collect a building video in a road environment during intelligent driving, split the building video into time sequence images, and perform feature extraction, matching and camera pose estimation on the time sequence images through an SfM algorithm to reconstruct a sparse point cloud of the building; Input each time sequence image into a pre-trained semantic segmentation model, detect and mark a crack region to generate a crack highlight image; According to the sparse point cloud, initialize a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building, and optimize the 3D Gaussian ellipsoid parameters according to local information of the time sequence images; According to the 3D Gaussian ellipsoid parameters, map the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model, and generate a rendering image through a differentiable rendering algorithm; Compare the rendering image with the crack highlight image, iteratively calculate an overall loss, transfer the overall loss through a back propagation gradient and update the 3D Gaussian ellipsoid parameters until a preset convergence condition is reached to obtain a final 3D model that simultaneously expresses the overall structure and crack details of the building.

[0016] In summary, the present application has the following beneficial technical effects: According to the sparse point cloud, initialize a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building, so that the local details (cracks) of the building surface can be accurately presented in the 3D model of the building; generate a rendering image through a differentiable rendering algorithm, compare the rendering image with the crack highlight image, iteratively calculate an overall loss, transfer the overall loss through a back propagation gradient, which can realize the joint optimization of the crack detection result and the 3D model of the building, so that the 3D model can not only reconstruct the overall geometric structure of the building, but also accurately express the local damage information such as cracks, thereby providing accurate environmental data for the automatic driving system. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A flowchart of a three-dimensional reconstruction and damage detection method for a building in an intelligent driving environment in an embodiment; Figure 2 A flowchart of a three-dimensional reconstruction and damage detection method for a building in an intelligent driving environment in another embodiment; Figure 3 A data processing flowchart of a three-dimensional reconstruction and damage detection system for a building in an intelligent driving environment; Figure 4 A structural block diagram of a three-dimensional reconstruction and damage detection device for a building in an intelligent driving environment in an embodiment. DETAILED DESCRIPTION

[0018] The embodiment of the present application provides a three-dimensional reconstruction and damage detection method for a building in an intelligent driving environment.

[0019] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather, these embodiments are provided to more thoroughly and completely understand the present disclosure. It is understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0020] In the description of embodiments of the present disclosure, the term "comprising" and similar terms are to be understood as open-ended, i.e., "including but not limited to". The term "based on" is to be understood as "based, at least in part, on". The term "one embodiment" or "the embodiment" is to be understood as "at least one embodiment". The terms "first", "second", etc. can refer to different or identical objects. Other explicit and implicit definitions can also be included below.

[0021] For ease of understanding, the specific process of the embodiments of the present disclosure is described below. Please refer to Figure 1 One embodiment of the three-dimensional reconstruction and damage detection method for intelligent driving environment buildings in the embodiments of the present disclosure includes: S100, collecting building videos in the road environment during intelligent driving, cutting the building videos into time sequence images, and performing feature extraction, matching and camera pose estimation on the time sequence images through the SfM algorithm to reconstruct the sparse point cloud of the building.

[0022] Specifically, building video data in the road environment during intelligent driving is collected on site to ensure that the collected video has sufficient clarity and stability; the video is cut into continuous time sequence images, and each frame of image is preprocessed (denoising, distortion correction, brightness equalization) as necessary to ensure the accuracy of subsequent feature extraction and crack detection. Specifically, the building video is recorded using a high-definition camera device, and optionally, a fixed or smooth following camera mode can be used to reduce motion blur; each frame of image is denoised, lens distortion corrected and brightness contrast adjusted, etc., to ensure that the image quality meets the subsequent processing requirements. Based on the preprocessed time sequence images, the SfM algorithm is used to reconstruct the preliminary geometric structure of the building to obtain the sparse point cloud and camera internal and external parameters, providing basic data for the initialization of the subsequent 3D Gaussian ellipsoid.

[0023] S200, inputting each frame of time sequence image into a pre-trained semantic segmentation model, detecting and marking the crack area to generate a crack highlight image.

[0024] Specifically, first, a semantic segmentation model is trained using a U-Net network structure. The model has a double-channel structure of an encoder-decoder, which can effectively capture the context information and detailed features of the image. During the training process, a pre-labeled building crack dataset is used for training to ensure that the model can accurately segment the crack area. After training, each frame of time-series image is input into the pre-trained semantic segmentation model to detect the crack area. The crack area is highlighted with a predetermined color (such as green) to generate a picture with highlighted crack information, providing supervision information for subsequent joint optimization with the 3D model.

[0025] S300, according to the sparse point cloud, initializing a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building, and optimizing the 3D Gaussian ellipsoid parameters according to the local information of the time-series image.

[0026] Specifically, a set of parameterized 3D Gaussian ellipsoids is initialized based on the sparse point cloud, and the parameters are further refined through image local information. Specifically, a parameterized 3D Gaussian ellipsoid is initialized at each key position of the building using the sparse point cloud information obtained above, which is used to refine the reconstruction result, so that the surface details of the building (especially the crack part) can be more accurately expressed in the 3D model. The core parameters of the 3D Gaussian ellipsoid include position parameters, covariance matrix, color information and opacity. The position parameters represent the center position of the Gaussian ellipsoid, which is represented by a floating-point number with a precision of 10 -3 meters; the covariance matrix describes the shape, direction and extension range of the ellipsoid, the initial value of which can be determined according to the local point cloud density and continuously adjusted in the subsequent optimization; the color information represents the color information of the local area, the initial value of which can be obtained from the image color statistical data, represented by RGB three channels, with a value range of [0, 255]; the opacity represents the opacity of the local area, which is initially set in the interval [0.5, 1.0] and automatically adjusted according to the loss feedback to ensure reasonable superimposition effect in the rendering process. Then, the 3D Gaussian ellipsoid parameters are optimized using the local information of the time-series image, so that the 3D Gaussian ellipsoid is more consistent with the actual building surface features.

[0027] In this embodiment, through the parameterized 3D Gaussian ellipsoid, not only the overall geometric structure of the building is restored, but also the tiny cracks and damage features on the building surface are captured and finely described, so that the final output 3D model has a significant improvement in accuracy, providing reliable data support for subsequent structure safety evaluation and maintenance decision-making.

[0028] S400, according to the 3D Gaussian ellipsoid parameters, mapping the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model, and generating a rendered image through a differentiable rendering algorithm.

[0029] Among them, differentiable rendering is a rendering technology based on mathematical derivability, and its core goal is to make the rendering process differentiable to the changes of input parameters (such as geometry, lighting, material, etc.), so as to support gradient back propagation and realize end-to-end optimization.

[0030] Specifically, by using the 3D Gaussian Splatting technology, the image generation is realized through the differentiable rendering algorithm based on Gaussian point cloud, so that the rendering speed can reach more than 100 FPS, which is much faster than the response speed of traditional SfM post-processing or body rendering method. While ensuring real-time preview, instant feedback of local details of buildings (such as cracks and damaged areas) is realized, so that users can obtain accurate building state information in a short time.

[0031] Compared with traditional rendering methods, the 3D Gaussian Splatting technology in the embodiment greatly reduces the calculation amount, thereby reducing energy consumption and hardware resource consumption, which is conducive to deployment and application on embedded or mobile devices. The set of Gaussian ellipsoids in three-dimensional space is mapped to a two-dimensional image plane through a camera model, and a differentiable rendering algorithm is used to generate a rendered image, so that the parameters of the entire 3D model have gradient transmission capability to the rendering result, supporting end-to-end joint optimization. The efficient 3D Gaussian Splatting rendering technology and end-to-end optimization process make the system have both real-time response capability and the ability to finely capture small cracks and local damage information on the surface of the building.

[0032] S500, compare the rendered image with the crack highlight image, iteratively calculate the overall loss, and update the 3D Gaussian ellipsoid parameters by back-propagating the overall loss and updating the 3D Gaussian ellipsoid parameters until the preset convergence condition is reached, to obtain the final 3D model that simultaneously expresses the overall structure and crack details of the building.

[0033] Specifically, by comparing the differences between the rendered image and the crack highlight image, the overall loss is calculated, and the error gradient is transmitted to each parameter of the 3D Gaussian ellipsoid by using the back propagation algorithm, and the 3D Gaussian ellipsoid parameters are updated, to realize the joint optimization of the building geometry and crack details. The optimization process of steps S300 to S400 is repeatedly executed until the overall loss converges or the preset number of iterations is reached, so as to obtain the final 3D model that simultaneously expresses the overall structure and crack details of the building. The high-precision 3D model containing the overall geometry of the building and the local crack damage information is finally output, and the corresponding 2D rendered image can be generated for subsequent analysis and display.

[0034] In this embodiment, the overall loss is propagated by backpropagating the gradient and updating the 3D Gaussian ellipsoid parameters, so that the two-dimensional crack detection information is directly fed back to the 3D model parameters, realizing the deep fusion of the overall geometric structure of the building and the local damage information, and greatly improving the overall accuracy and robustness of the 3D model reconstruction.

[0035] The overall scheme adopts an end-to-end differentiable rendering and joint optimization process, fuses the two-dimensional image information, crack detection results and the preliminary obtained sparse 3D point cloud data in the building video data, refines the reconstruction details through the parameterized representation of the 3D Gaussian ellipsoid, and feeds back the semantic information of the crack area to the 3D model in the iterative optimization process.

[0036] In one embodiment, each frame of time series image is input into a pre-trained semantic segmentation model to detect and mark the crack area, and generate a crack highlight image, including: Each frame of time series image is input into a pre-trained semantic segmentation model to obtain a pixel-level segmentation map of the crack area; the pixel-level segmentation map and the original time series image are fused, and the crack area is marked with a predetermined color to generate a crack highlight image.

[0037] Specifically, first, each frame of preprocessed time series image is adjusted to the same size (512x512 pixels), then input into the trained semantic segmentation model to obtain a pixel-level segmentation map of the crack area, wherein the model loss function combines cross-entropy loss and Dice loss, and the weight can be set to 0.5:0.5, and the model is trained based on thousands of building crack labeled images; the pixel-level segmentation map and the original image are fused, and the crack area is marked with a predetermined color such as green to generate a picture with highlighted crack information.

[0038] In one embodiment, as shown in Figure 2 S300 includes: S320, according to the sparse point cloud, initializing a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building; S340, pre-optimizing the 3D Gaussian ellipsoid parameters according to the local color, brightness and texture information of the time series image, so that the 3D Gaussian ellipsoid is more fitted to the actual building surface features; S360, optimizing the 3D Gaussian ellipsoid parameters by least squares method.

[0039] The least squares method is a method of finding the best fitting parameters by minimizing the sum of squared residuals. In 3D ellipsoid fitting, the goal is to find a set of parameters that minimize the geometric distance between data points and the ellipsoid surface.

[0040] Specifically, sparse point cloud information is used to initialize a parameterized 3D Gaussian ellipsoid at each key position of the building; according to the spatial distribution of the sparse point cloud, the corresponding Gaussian ellipsoid parameters are initialized in each local area, and more Gaussian elements can be set for the dense point cloud area, and the number of Gaussian elements can be reduced for the sparse area; the Gaussian ellipsoid parameters are pre-optimized using local color, brightness and texture information of the image, so that they are more consistent with the actual building surface characteristics; the 3D Gaussian ellipsoid parameters are optimized by using a preliminary least squares fitting or other optimization method to determine the initial parameter values. In addition, during the generation and optimization of the 3D Gaussian ellipsoid, a covariance matrix regularization term is introduced to prevent the parameters from fluctuating too much and to ensure the overall smoothness and stability of the model.

[0041] In one embodiment, according to the 3D Gaussian ellipsoid parameters, the 3D Gaussian ellipsoid is mapped to a two-dimensional image plane through a camera model, and a rendered image is generated by a differentiable rendering algorithm, including: According to the 3D Gaussian ellipsoid parameters, the 3D Gaussian ellipsoid is mapped to a two-dimensional image plane through a camera model; the two-dimensional image plane is rasterized by an alpha blending method based on Gaussian distribution to sort the 3D Gaussian ellipsoids overlaid on the same two-dimensional image plane pixel; the contribution of the 3D Gaussian ellipsoid to the two-dimensional image plane pixel is calculated according to the sorting, and the color value of each pixel on the two-dimensional image plane is obtained according to the contribution of the 3D Gaussian ellipsoid to the two-dimensional image plane pixel, to generate a rendered image.

[0042] The alpha blending method based on Gaussian distribution generally refers to using the probability density function of Gaussian distribution to calculate the blending effect of multiple translucent objects in rendering or image processing. Alpha blending is an image compositing technique that generates the final pixel color by weighting and averaging the colors of multiple translucent objects.

[0043] Specifically, according to the camera internal and external parameters and the 3D Gaussian ellipsoid parameters, each 3D Gaussian ellipsoid is mapped to a two-dimensional image plane through a camera model, and the projection area and shape parameters of each ellipsoid on the image plane are calculated. Then, an alpha blending method based on Gaussian distribution is used for rasterization, and in the rasterization process, the 3D Gaussian ellipsoids overlaid on the same two-dimensional image plane pixel are sorted, the sorting is usually based on depth value, the contribution of each 3D Gaussian ellipsoid to the two-dimensional image plane pixel is calculated by the rendering formula from near to far, and the color value of each pixel on the two-dimensional image plane is calculated according to the contribution. For each pixel on the image plane The color value of each pixel on the image plane can be expressed as: wherein, represents the contribution of the i-th Gaussian ellipsoid on the pixel according to the two-dimensional projection of the ellipsoid, is the opacity, is the color information.

[0044] All mapping and blending operations are designed to be differentiable, so that when the error is backpropagated, the gradient can be passed to the parameters (position, covariance, color, opacity) of each Gaussian ellipsoid.

[0045] In one embodiment, the rendered image and the crack highlight image are compared, the overall loss is iteratively calculated, the overall loss is transmitted by backpropagating the gradient and the 3D Gaussian ellipsoid parameters are updated until a preset convergence condition is reached, and a final 3D model that simultaneously expresses the overall structure of the building and the details of the cracks is obtained, including: The rendered image and the crack highlight image are compared, the pixel-level difference loss, the crack area key loss, and the regularization loss are iteratively calculated, the pixel-level difference loss, the crack area key loss, and the regularization loss are weighted and summed to obtain the overall loss, and the overall loss is transmitted by backpropagating the gradient and the 3D Gaussian ellipsoid parameters are updated until a preset convergence condition is reached, and a final 3D model that simultaneously expresses the overall structure of the building and the details of the cracks is obtained.

[0046] Specifically, the loss function includes a pixel-level difference loss, a crack area key loss, and a regularization loss, wherein the pixel-level difference loss calculates the error in overall brightness and color between the rendered image and the crack highlight image using L1 loss; the crack area key loss gives a higher weight (multiplied by a coefficient λ>1) to the area marked by the crack detection module, ensuring that the crack details are fully focused; the regularization loss designs a regularization term for the Gaussian ellipsoid parameters (covariance matrix) to prevent abnormal fluctuations in the parameters and maintain the smoothness of the model. The overall loss L is obtained by weighting and summing the various losses, and the overall loss is transmitted by backpropagation.

[0047] In this embodiment, by designing the loss function to include both the overall pixel-level difference loss and the error in the crack area, a higher weight can be applied to the damaged area during optimization, making the expression of cracks and damage information in the final model more obvious and accurate.

[0048] In one embodiment, the overall loss is transmitted by backpropagating the gradient and the 3D Gaussian ellipsoid parameters are updated until a preset convergence condition is reached, and a final 3D model that simultaneously expresses the overall structure of the building and the details of the cracks is obtained, including: The overall loss gradient is transmitted to the corresponding 3D Gaussian ellipsoid parameters through back propagation; the 3D Gaussian ellipsoid parameters are updated by an Adam optimizer until a preset convergence condition is reached, and the final 3D model expressing the overall structure and crack details of the building is obtained.

[0049] Specifically, the overall loss L is iteratively calculated, the gradient of the overall loss L is transmitted to the corresponding 3D Gaussian ellipsoid parameters by using a back propagation algorithm, then the Gaussian ellipsoid parameters are updated by using an Adam optimizer until a preset convergence condition is reached, and the final 3D model expressing the overall structure and crack details of the building is obtained, wherein the initial learning rate of the Adam optimizer is set to 1e-3, and the number of iterations is determined according to the loss convergence condition.

[0050] In this embodiment, through fine control of gradient feedback, stable transmission of gradient can be ensured in the back propagation process, and the problems of gradient vanishing or explosion that may occur in traditional differentiable rendering are avoided, thereby improving the overall stability and convergence speed of the system.

[0051] In one embodiment, the SfM algorithm is used to perform feature extraction, matching and camera pose estimation on the time-series images, and a sparse point cloud of the building is reconstructed, including: The local feature extraction algorithm is used to extract feature points from each time-series image, and the feature matching algorithm is used to realize feature correspondence between adjacent or overlapping time-series images, and a feature point matching relationship is constructed; the camera internal and external parameters of each time-series image are estimated by the SfM algorithm according to the matching features; and the sparse point cloud of the building is reconstructed according to the camera internal and external parameters and the feature point matching relationship.

[0052] In this embodiment, the SIFT, ORB and other local feature extraction algorithms are used to extract feature points from each time-series image, and the feature matching algorithm is used to realize feature correspondence between adjacent or overlapping time-series images according to the extracted feature points, and a feature point matching relationship is constructed, and then the camera internal and external parameters of each time-series image are calculated by using the SfM algorithm based on the feature matching results. Secondly, the sparse point cloud data of the building is reconstructed by using the triangulation method according to the matching relationship and the camera parameters, and the three-dimensional structure of the building is preliminarily reflected.

[0053] In addition, the system can provide an intuitive graphical user interface (GUI), and the user can see the changes of the rendered images and detection results in real time during the video acquisition, preprocessing, 3D reconstruction and crack detection process. This instant feedback mechanism enables the operator to quickly adjust the acquisition angle, illumination and other parameters to ensure the quality of data input, thereby improving the overall detection effect. From video acquisition, image preprocessing to sparse point cloud generation, crack region detection and 3D model joint optimization, the present application constructs an automatic and adjustable processing flow, greatly reducing the need for manual intervention.

[0054] To illustrate the technical solutions and effects of the three-dimensional reconstruction and damage detection method for buildings in an intelligent driving environment of the present application, specific application examples will be used in the following description. As shown in Figure 3 The data processing flow of the entire three-dimensional reconstruction and damage detection system for buildings in an intelligent driving environment includes the following steps: 1. Video acquisition and image preprocessing: After the building video is acquired, it is cut into time-series images and processed by the preprocessing module for denoising, correction, etc. to ensure image quality.

[0055] 2. Sparse point cloud generation: The SfM algorithm is used to extract features, match, and estimate camera pose from the preprocessed images, and reconstruct the sparse point cloud data of the building.

[0056] 3. Crack detection: Each frame of image is input into the pre-trained U-Net model to obtain the crack segmentation result, and is fused with the original image to form a crack highlighted picture.

[0057] 4. 3D Gaussian ellipsoid initialization: According to the distribution of the sparse point cloud in space, the corresponding 3D Gaussian ellipsoid set is initialized, and each ellipsoid is described by parameters such as position, covariance matrix, color, opacity, etc.

[0058] 5. Camera mapping and rendering: According to the camera internal and external parameters, the 3D Gaussian ellipsoid is projected onto the two-dimensional image plane, and the differentiable rendering algorithm is used to generate the rendered image.

[0059] 6. Loss calculation and feedback optimization: The total loss between the rendered image and the crack highlighted picture is calculated, and the 3D Gaussian ellipsoid parameters are updated through backpropagation. This process is iterated until the convergence criterion is met.

[0060] 7. Output: The final output is a high-precision 3D model containing the overall geometric structure and local crack damage information of the building, and the corresponding 2D rendered image can be generated for subsequent analysis and display.

[0061] The entire system realizes seamless fusion of two-dimensional image and three-dimensional model information. Through the end-to-end differentiable rendering process, the crack detection result is not only used for separate two-dimensional analysis, but also directly used as a supervision signal to feedback to the parameter optimization of the 3D model, so that the surface damage information of the building is accurately described in the 3D model. This joint optimization mechanism makes up for the shortcomings of traditional SfM, which only generates sparse point clouds and two-dimensional crack detection is executed independently, achieving collaborative optimization and real-time response of building 3D reconstruction and damage detection.

[0062] In one embodiment, as Figure 4As shown, a three-dimensional reconstruction and damage detection device for intelligent driving environment buildings is provided, comprising a sparse point cloud reconstruction module 10, a crack highlight image generation module 20, a Gaussian ellipsoid generation module 30, a rendered image generation module 40 and a 3D model generation module 50, wherein: The sparse point cloud reconstruction module 10 is used to collect building videos in the road environment during intelligent driving, cut the building videos into time sequence images, and extract, match and estimate the camera pose of the time sequence images through the SfM algorithm to reconstruct the sparse point cloud of the building. The crack highlight image generation module 20 is used to input each frame of time sequence image into a pre-trained semantic segmentation model, detect and mark the crack area, and generate a crack highlight image. The Gaussian ellipsoid generation module 30 is used to initialize a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud, and optimize the 3D Gaussian ellipsoid parameters according to the local information of the time sequence image. The rendered image generation module 40 is used to map the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters, and generate a rendered image through a differentiable rendering algorithm. The 3D model generation module 50 is used to compare the rendered image with the crack highlight image, iteratively calculate the overall loss, transfer the overall loss through back propagation gradient and update the 3D Gaussian ellipsoid parameters until the preset convergence condition is reached, and obtain the final 3D model which expresses the overall structure and crack details of the building.

[0063] In one embodiment, the crack highlight image generation module 20 is also used to input each frame of time sequence image into a pre-trained semantic segmentation model to obtain a pixel-level segmentation map of the crack area; fuse the pixel-level segmentation map with the original time sequence image, and mark the crack area with a preset color to generate a crack highlight image.

[0064] In one embodiment, the Gaussian ellipsoid generation module 30 is also used to initialize a corresponding parameterized 3D Gaussian ellipsoid at each key position of the building according to the sparse point cloud; pre-optimize the 3D Gaussian ellipsoid parameters according to the local color, brightness and texture information of the time sequence image to make the 3D Gaussian ellipsoid more consistent with the actual building surface features; and optimize the 3D Gaussian ellipsoid parameters through the least squares method.

[0065] In an embodiment, the rendering image generation module 40 is further configured to map the 3D Gaussian ellipsoid to a two-dimensional image plane through a camera model according to the 3D Gaussian ellipsoid parameters; rasterize the two-dimensional image plane through an alpha blending method based on Gaussian distribution to sort the 3D Gaussian ellipsoids overlaid on the same two-dimensional image plane pixel; calculate the contribution of the 3D Gaussian ellipsoid on the two-dimensional image plane pixel according to the sorting, and obtain the color value of each pixel on the two-dimensional image plane according to the contribution of the 3D Gaussian ellipsoid on the two-dimensional image plane pixel to generate the rendering image.

[0066] In an embodiment, the 3D model generation module 50 is further configured to compare the rendering image with the crack highlight image, iteratively calculate a pixel-level difference loss, a crack area emphasis loss and a regularization loss, sum the pixel-level difference loss, the crack area emphasis loss and the regularization loss to obtain an overall loss, and pass the overall loss through back propagation gradient transmission and update the 3D Gaussian ellipsoid parameters until a preset convergence condition is reached to obtain the final 3D model expressing the overall structure and crack details of the building.

[0067] In an embodiment, the 3D model generation module 50 is further configured to pass the overall loss gradient to the corresponding 3D Gaussian ellipsoid parameters through back propagation, and update the 3D Gaussian ellipsoid parameters through an Adam optimizer until a preset convergence condition is reached to obtain the final 3D model expressing the overall structure and crack details of the building.

[0068] In an embodiment, the sparse point cloud reconstruction module 10 is further configured to extract feature points from each frame of time sequence image through a local feature extraction algorithm, and realize feature correspondence between adjacent or overlapping time sequence images through a feature matching algorithm to construct a feature point matching relationship; estimate the camera internal and external parameters of each frame of time sequence image through an SfM algorithm according to the matching features; and reconstruct the sparse point cloud of the building according to the camera internal and external parameters and the feature point matching relationship.

[0069] The above-mentioned various modules in the device for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment can be realized by software, hardware and combinations thereof, in whole or in part. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.

[0070] In an embodiment, the present application discloses a computer device comprising a memory and a processor, the memory is configured to store a computer program capable of running on the processor, and the processor is configured to load the computer program to execute the above-mentioned embodiment of the method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment.

[0071] In one embodiment, the embodiment of the application discloses a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is loaded by a processor to execute the three-dimensional reconstruction and damage detection method for intelligent driving environment buildings.

[0072] The above are preferred embodiments of the application, and do not limit the protection scope of the application, so: all equivalent changes made according to the structure, shape, principle of the application should be covered within the protection scope of the application.

Claims

1. A method for three-dimensional reconstruction and damage detection of buildings in intelligent driving environments, characterized in that, include: Videos of buildings in the road environment during intelligent driving are collected, the building videos are segmented into time-series images, and the SfM algorithm is used to extract features, match and estimate camera pose of the time-series images to reconstruct the sparse point cloud of the buildings. Each frame of temporal image is input into a pre-trained semantic segmentation model to detect and label crack regions, generating crack highlight images; Based on the sparse point cloud, a corresponding parameterized 3D Gaussian ellipsoid is initialized at each key location of the building, and the parameters of the 3D Gaussian ellipsoid are optimized based on the local information of the time-series image. Based on the 3D Gaussian ellipsoid parameters, the 3D Gaussian ellipsoid is mapped onto a two-dimensional image plane through a camera model, and a rendered image is generated through a differentiable rendering algorithm. The rendered image is compared with the crack highlight image, the overall loss is calculated iteratively, the overall loss is passed through backpropagation gradient and the 3D Gaussian ellipsoid parameters are updated until the preset convergence condition is reached, and the final 3D model that simultaneously expresses the overall structure of the building and the crack details is obtained.

2. The method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment according to claim 1, characterized in that, The step of inputting each frame of temporal image into a pre-trained semantic segmentation model to detect and label crack regions and generate crack highlight images includes: Each frame of temporal image is input into a pre-trained semantic segmentation model to obtain a pixel-level segmentation map of the crack region; The pixel-level segmentation image is fused with the original time-series image, and the crack region is marked with a preset color to generate a crack highlight image.

3. The method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment according to claim 1, characterized in that, The local information of the time-series image includes the local color, brightness, and texture information of the time-series image; The step of initializing a corresponding parameterized 3D Gaussian ellipsoid at each key location of the building based on the sparse point cloud, and optimizing the 3D Gaussian ellipsoid parameters based on the local information of the time-series image, includes: Based on the sparse point cloud, initialize the corresponding parameterized 3D Gaussian ellipsoid at each key location of the building; The parameters of the 3D Gaussian ellipsoid are pre-optimized based on the local color, brightness and texture information of the time series image, so that the 3D Gaussian ellipsoid better fits the actual building surface features. The parameters of the 3D Gaussian ellipsoid are optimized by fitting using the least squares method.

4. The method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment according to claim 1, characterized in that, The differentiable rendering algorithm is an alpha blending method based on Gaussian distribution; The step of mapping the 3D Gaussian ellipsoid onto a two-dimensional image plane using a camera model based on the 3D Gaussian ellipsoid parameters, and generating a rendered image using a differentiable rendering algorithm, includes: Based on the 3D Gaussian ellipsoid parameters, the 3D Gaussian ellipsoid is mapped onto a two-dimensional image plane through a camera model; The two-dimensional image plane is rasterized using an alpha mixing method based on Gaussian distribution to sort the 3D Gaussian ellipsoids covering pixels on the same two-dimensional image plane; The contribution of the 3D Gaussian ellipsoid to the pixels in the two-dimensional image plane is calculated based on the sorting, and the color value of each pixel in the two-dimensional image plane is obtained based on the contribution of the 3D Gaussian ellipsoid to the pixels in the two-dimensional image plane, thereby generating a rendered image.

5. The method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment according to claim 1, characterized in that, The process of comparing the rendered image with the crack highlight image, iteratively calculating the overall loss, propagating the overall loss through backpropagation gradients and updating the 3D Gaussian ellipsoid parameters, until a preset convergence condition is met, yields the final 3D model that simultaneously represents the overall structure of the building and the details of the cracks, including: The rendered image is compared with the crack highlight image, and the pixel-level difference loss, crack region focus loss, and regularization loss are calculated iteratively. The overall loss is obtained by weighted summing of the pixel-level difference loss, the crack region focus loss, and the regularization loss. The overall loss is propagated by backpropagation gradients and the 3D Gaussian ellipsoid parameters are updated until a preset convergence condition is met, resulting in a final 3D model that simultaneously expresses the overall structure of the building and the details of cracks.

6. The method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment according to claim 5, characterized in that, The process of propagating the overall loss through backpropagation gradients and updating the 3D Gaussian ellipsoid parameters until a preset convergence condition is met, to obtain the final 3D model that simultaneously represents the overall structure and crack details of the building, includes: The overall loss gradient is passed to the corresponding 3D Gaussian ellipsoid parameters through backpropagation. The 3D Gaussian ellipsoid parameters are updated by the Adam optimizer until a preset convergence condition is met, resulting in a final 3D model that simultaneously expresses the overall structure of the building and the details of cracks.

7. The method for three-dimensional reconstruction and damage detection of buildings in an intelligent driving environment according to claim 1, characterized in that, The step of reconstructing the sparse point cloud of buildings by performing feature extraction, matching, and camera pose estimation on the temporal image using the SfM algorithm includes: Feature points are extracted from each frame of temporal image using a local feature extraction algorithm, and feature matching is used to establish feature correspondence between adjacent or overlapping temporal images and construct feature point matching relationships. Based on the matching features, the camera intrinsic and extrinsic parameters of each frame of temporal image are estimated using the SfM algorithm; Based on the camera's intrinsic and extrinsic parameters and the feature point matching relationship, the sparse point cloud of the building is reconstructed.

8. A three-dimensional reconstruction and damage detection device for buildings in intelligent driving environments, characterized in that, include: The sparse point cloud reconstruction module is used to collect building videos in the road environment during intelligent driving, segment the building videos into time-series images, and use the SfM algorithm to extract features, match and estimate camera poses of the time-series images to reconstruct the sparse point cloud of the buildings. The crack highlight image generation module is used to input each frame of temporal image into a pre-trained semantic segmentation model, detect and label crack regions, and generate crack highlight images. The Gaussian ellipsoid generation module is used to initialize the corresponding parameterized 3D Gaussian ellipsoid at each key location of the building based on the sparse point cloud, and optimize the 3D Gaussian ellipsoid parameters based on the local information of the time series image. The rendering image generation module is used to map the 3D Gaussian ellipsoid onto a two-dimensional image plane through a camera model based on the 3D Gaussian ellipsoid parameters, and generate a rendering image through a differentiable rendering algorithm. The 3D model generation module is used to compare the rendered image with the crack highlight image, iteratively calculate the overall loss, pass the overall loss through backpropagation gradient and update the 3D Gaussian ellipsoid parameters until a preset convergence condition is reached, so as to obtain the final 3D model that simultaneously expresses the overall structure of the building and the details of the crack.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images