Automatic detection method for building window-wall ratio based on deep learning and 3DGS

Abstract

The present application relates to the field of building energy-saving evaluation and computer vision, and discloses a kind of based on deep learning and 3DGS building window wall ratio automation detection method, comprising the following steps: the multi-view facade image of target building is collected by unmanned aerial vehicle;Sparse three-dimensional reconstruction is carried out on the collected image using motion recovery structure method, to obtain camera internal parameter, camera pose and sparse three-dimensional point cloud;Based on sparse three-dimensional point cloud, camera internal parameter and camera pose, three-dimensional Gaussian splashing technology is used to construct three-dimensional scene representation, and training optimization is carried out through differentiable rasterization renderer;The facade parameter of target building is extracted, and the orthographic projection image of the facade is generated;The window in the orthographic projection image is detected using a deep learning object detection model, and the window region pixel coordinates are output;The window wall ratio of a single facade is calculated according to the window pixel area and the facade pixel area, and the overall building window wall ratio is calculated in combination with the area weight of each facade. The scheme improves the detection accuracy.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of building energy efficiency assessment and computer vision, specifically to an automated detection method for the window-to-wall ratio of buildings based on deep learning and 3DGS. Background Technology

[0002] The window-to-wall ratio, a key indicator for measuring the thermal performance of a building envelope, directly impacts building energy consumption, indoor lighting quality, and thermal comfort, making it a core control parameter in building energy efficiency design codes. Currently, window-to-wall ratio data is primarily obtained through two methods: manual on-site measurement or reviewing historical as-built drawings. However, manual measurement requires professionals to conduct building-by-building surveys with ranging equipment. For city-level building cluster energy efficiency diagnostic projects, this often requires dozens of person-months of manpower and several months of time. Furthermore, many existing buildings suffer from missing or incomplete drawings due to their age, making systematic data collection difficult. Meanwhile, while deep learning detection technology based on single 2D images has made progress in the field of building element recognition, it is limited by the geometric characteristics of image perspective projection. Non-uniform scale deformation occurs in wall and window areas during imaging, causing pixel areas to fail to accurately reflect actual physical dimensions. Furthermore, visual occlusions formed by tree branches, adjacent buildings, or temporary facilities can cover parts of the window area, causing target detection algorithms to miss or misjudge detections. This results in a significant deviation between the calculated window pixel area and the facade area based on single-view images, ultimately affecting the engineering application value of the window-to-wall ratio. The root of these problems lies in the inability of traditional manual methods to overcome the efficiency bottleneck of labor-intensive operations. Furthermore, 2D image information itself lacks depth dimension data support, making it impossible to correct area distortion caused by perspective distortion and to effectively separate foreground occlusion from the target building structure. This makes it difficult for existing technologies to meet the urgent need for large-scale, high-precision acquisition of window-to-wall ratio parameters in building energy efficiency assessments. Therefore, there is an urgent need to propose a detection method for building window-to-wall ratio that is efficient, automated, and highly accurate. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a novel automated window-to-wall ratio detection method based on deep learning and 3DGS to solve the perspective distortion and occlusion problems of existing two-dimensional images.

[0004] To achieve the above objectives, the present invention employs the following technical solution: An automated method for detecting the window-to-wall ratio of buildings based on deep learning and 3DGS includes the following steps: Step 1: Collect multi-view facade images of the target building using a drone; Step 2: Use the structure-of-motion reconstruction method to perform sparse 3D reconstruction on the acquired images to obtain camera intrinsic parameters, camera pose, and sparse 3D point cloud; Step 3: Based on the sparse 3D point cloud, camera intrinsic parameters and camera pose, a 3D scene representation is constructed using 3D Gaussian splashing technology, and then trained and optimized using a differentiable rasterizer renderer. Step 4: Extract the facade parameters of the target building, and use the three-dimensional Gaussian splash model to generate an orthographic projection image of the facade using the orthographic projection method, wherein the interference of foreground occlusions is eliminated by setting the rendering distance; Step 5: Use a deep learning object detection model to detect windows in the orthographic projection image and output the pixel coordinates of the window region; Step 6: Calculate the window-to-wall ratio of a single facade based on the window pixel area and the facade pixel area, and calculate the overall building window-to-wall ratio by combining the area weights of each facade.

[0005] Furthermore, in step 1, the overlapping area of ​​adjacent images is greater than 30% when the drone takes pictures, and the camera intrinsic parameters, including focal length, optical center position and distortion parameters, are recorded.

[0006] Furthermore, in step 2, the open-source software COLMAP is used for motion recovery structure reconstruction, and a sequential matching strategy is used for feature point matching for continuously captured image data.

[0007] Furthermore, in step 3, the parameters of each Gaussian element include position, covariance matrix, opacity, and color coefficients represented by spherical harmonic functions. The rotation matrix is ​​parameterized by quaternions, and the covariance matrix is ​​decomposed into the product of the rotation matrix and the scaling matrix.

[0008] Furthermore, in step 3, during initialization, each point in the sparse 3D point cloud output by the motion recovery structure is initialized as a 3D Gaussian element, with its position set to point cloud coordinates, its covariance set to an isotropic small sphere, its color set to the RGB value of the point cloud, and its opacity set to 0.1.

[0009] Furthermore, in step 3, the training optimization uses a weighted combination of L1 loss and structural similarity loss as the loss function, dynamically adjusts the number of Gaussians through an adaptive density control strategy, and sets different learning steps for position, scale, rotation, opacity, and color coefficient.

[0010] Furthermore, in step 4, the building facade extraction adopts an automatic method based on RANSAC plane segmentation and minimum bounding box, or a semi-automatic method through manual corner point calibration. The facade parameters include corner point coordinates, normal direction, width and height information, and area weight.

[0011] Furthermore, in step 4, spatial filtering is performed before orthographic projection rendering. The in-plane boundary is widened by 2% according to the width and height of the facade, and a certain thickness range is left on the front and back sides in the normal direction, so as to preserve the effective structure of the facade and suppress interference from irrelevant scenes.

[0012] Furthermore, in step 5, the window detection model is based on the YOLOv11 architecture, pre-trained on the COCO dataset, and trained using transfer learning on the street view window dataset. At least one of the following constraints is introduced during the testing process: Enhanced testing: Repeat the detection process after flipping, enlarging, and shrinking the image; Abnormal bounding box filtering: Detection boxes are filtered based on aspect ratio, area size, and the difference between brightness and median value; Detection box clustering: For windows arranged in a regular pattern, the position of the detection boxes is corrected by a clustering algorithm, and the undetected positions are re-detected with low confidence.

[0013] Furthermore, in step 6, the formula for calculating the window-to-wall ratio of a single facade is: ; in, express WWR on the facade express Total pixel area of ​​the facade window region. express Total pixel area of ​​the facade wall area; The formula for calculating the overall window-to-wall ratio of a building is: ; in, for The area weight of the facade, where n is the total number of facades.

[0014] Compared to existing technologies, the advantages of this invention, which adopts the above-mentioned scheme, are that the automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS can obtain the camera pose and the 3D sparse point cloud of the target building through SfM; obtain the 3D model of the target building through 3DGS and render the building facade image from any viewpoint; obtain the position and size of windows in the facade image through a trained window detection model; and finally obtain the WWR of any facade or the entire facade of the target building. By automatically acquiring multi-view images, generating orthographic projection through 3D reconstruction, detecting windows, and calculating the window-to-wall ratio, it solves the problems of inefficiency in manual measurement and perspective distortion occlusion in 2D images, and has the advantages of efficient, automated, and high-precision detection of building window-to-wall ratio. Attached Figure Description

[0015] Figure 1This is a flowchart illustrating an embodiment of the automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a projection method according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the shooting path according to an embodiment of the present invention; Figure 4 This is a flowchart of a facade window inspection process according to an embodiment of the present invention. Detailed Implementation

[0016] To facilitate understanding of the present invention by those skilled in the art, specific embodiments of the present invention are described below with reference to the accompanying drawings. Preferred embodiments of the present invention are shown in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. Rather, these embodiments are provided to provide a thorough and complete understanding of the disclosure of the present invention.

[0017] It should be noted that when a component is said to be "fixed to" another component, it can be directly attached to the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component.

[0018] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0019] One embodiment of the present invention is as follows: Figure 1-4 As shown, this automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS acquires multi-view facade images of the target building using a drone, performs sparse 3D reconstruction using the structure-by-motion method, and constructs a 3D scene representation based on the reconstruction results using 3D Gaussian splashing technology. Further, the method extracts building facade parameters, generates orthographic projection images of the facades through orthographic projection, and sets a rendering distance to eliminate interference from foreground occlusions. Subsequently, a deep learning object detection model is used to detect windows in the orthographic projection images. Finally, the window-to-wall ratio of a single facade is calculated based on the window pixel area and the facade pixel area, and the overall building window-to-wall ratio is calculated by combining the area weights of each facade.

[0020] For ease of understanding, the following explains some key terms in this embodiment: A drone, generally speaking, refers to an unmanned aircraft controlled by radio remote control equipment or its own programs, capable of carrying sensors such as cameras to collect multi-view image data from the air. In this application, a drone is used to efficiently and flexibly acquire facade images of a target building.

[0021] Structure of Motion (SfM) is a computer vision technique that reconstructs the 3D structure of a scene, along with the camera pose and intrinsic parameters of the camera that captured the images, by analyzing a series of 2D images. This technique can recover sparse 3D point clouds, camera intrinsic parameters, and camera pose from an unordered set of images, providing foundational data for subsequent 3D modeling.

[0022] Sparse 3D point clouds refer to a collection of points with three-dimensional coordinates reconstructed using methods such as SfM. These points typically correspond to feature points in a scene and can roughly represent the 3D geometry of the scene.

[0023] 3D Gaussian Splashing (3DGS) is an emerging method for representing and rendering 3D scenes. Its core idea is to use a large number of 3D Gaussian primitives to represent the scene, each primitive having parameters such as position, scale, rotation, opacity, and color. By rendering and optimizing these Gaussian primitives, high-quality 3D scenes can be efficiently reconstructed.

[0024] A differentiable rasterizer is a renderer capable of rendering a 3D scene representation (e.g., 3D Gaussian primitives) into a 2D image, and whose rendering process is differentiable with respect to the 3D scene parameters. This allows for the adjustment of 3D scene parameters based on the differences between the rendered image and the real image using optimization algorithms such as gradient descent, thereby achieving training optimization of the 3D model.

[0025] Orthographic projection is a type of parallel projection in which the projection lines are parallel to each other and perpendicular to the projection plane. Unlike perspective projection, orthographic projection does not produce the perspective effect of objects appearing larger when closer and smaller when farther away, and it can maintain the true size proportion of objects on the projection plane, making it suitable for precise measurement.

[0026] A deep learning object detection model is a computational model based on deep learning technology that can automatically identify the location of specific objects in an image and classify them. This model learns image features through training, thereby accurately detecting predefined targets, such as windows, in new images.

[0027] The window-to-wall ratio (WWR) is the ratio of the area of ​​windows on a building's exterior walls to the total area of ​​the walls. It is one of the important indicators for measuring a building's energy efficiency. Accurate calculation of the WWR is of great significance for building design, energy consumption assessment, and energy-saving retrofitting.

[0028] This embodiment provides an automated method for detecting the window-to-wall ratio of buildings based on deep learning and 3DGS, specifically including the following steps: First, multi-view facade images of the target building are acquired using drones. The drones can fly along a preset flight path or be manually controlled, capturing multiple images. During image acquisition, it's easy to ensure coverage of all facades of the target building and record basic information such as shooting time and GPS location. Camera intrinsics can be provided by the camera manufacturer or obtained through simple calibration methods.

[0029] Secondly, the Structure-of-Motion (SOM) method is used to perform sparse 3D reconstruction on the acquired images, obtaining camera intrinsics, camera pose, and sparse 3D point clouds. The SOM method utilizes feature points in the images for matching and triangulation to estimate the camera pose and intrinsics for each image, generating a sparse 3D point cloud of the scene. For example, the general SfM algorithm library can be used, with feature point matching strategies including brute-force matching or approximate nearest neighbor matching.

[0030] Next, based on the aforementioned sparse 3D point cloud, camera intrinsics, and camera pose, a 3D Gaussian splashing technique is used to construct a 3D scene representation, which is then trained and optimized using a differentiable rasterizer renderer. The 3D Gaussian splashing technique utilizes the sparse 3D point cloud as initial points, creating a 3D Gaussian primitive for each point. Each Gaussian primitive can be assigned initial position, scale, rotation, and color information. Subsequently, these Gaussian primitives are rendered into an image using a differentiable rasterizer renderer and compared with the actual acquired image. Based on the comparison results, optimization algorithms can be used to iteratively adjust the parameters of each Gaussian primitive to minimize the difference between the rendered image and the real image, thereby progressively optimizing the 3D scene representation.

[0031] Furthermore, the facade parameters of the target building are extracted, and an orthographic projection image of the facade is generated using the aforementioned 3D Gaussian splash model via orthographic projection. The rendering distance is set to exclude interference from foreground occlusion. The facade parameters of the target building can be manually selected and defined in the 3D scene, for example, by manually selecting the four corner points of the facade and calculating their normal direction and width and height information. Once the facade is defined, the orthographic projection image of the facade can be generated using the constructed 3D Gaussian splash model via orthographic projection. During the rendering process, a rendering distance range can be set so that only 3D Gaussian primitives within this range are rendered, thereby mitigating the influence of foreground occlusion on the generation of the facade image to some extent.

[0032] Subsequently, a deep learning object detection model is used to detect windows in the orthographic projection image, outputting the pixel coordinates of the window regions. The deep learning object detection model can be a general convolutional neural network (CNN) architecture, such as Faster R-CNN, SSD, or any of the YOLO series. This model is trained on an image dataset containing window annotations to learn how to recognize window features in images. During the detection phase, the generated orthographic projection image of the facade is input into the trained model, which outputs the bounding boxes of the detected window regions and their pixel coordinates.

[0033] Finally, the window-to-wall ratio of a single facade is calculated based on the window pixel area and the facade pixel area, and the overall building window-to-wall ratio is calculated by combining the area weights of each facade. After obtaining the pixel coordinates of the window areas, the pixel area of ​​each window area can be calculated, and the total pixel area of ​​all windows on a single facade can be obtained by summing them. At the same time, the total pixel area of ​​that single facade is calculated. The window-to-wall ratio of a single facade can be obtained by dividing the total pixel area of ​​the windows by the total pixel area of ​​the facade. For the entire building, the window-to-wall ratio of each facade can be weighted and averaged, where the weights can be set according to the actual area or importance of each facade, thus obtaining the overall building window-to-wall ratio.

[0034] This embodiment effectively solves the problems of low data acquisition efficiency and high cost in traditional building window-to-wall ratio detection by integrating multi-view image acquisition from UAVs, motion-reconstructed 3D structural reconstruction, 3D Gaussian splash modeling, orthogonal projection rendering, and deep learning-based object detection, achieving automated and large-scale data acquisition. Simultaneously, this method eliminates the influence of perspective distortion through 3D scene reconstruction and orthogonal projection, and effectively eliminates interference from foreground occlusions by setting the rendering distance, significantly improving the accuracy and practicality of window detection and window-to-wall ratio calculation, providing reliable data support for building energy efficiency assessment.

[0035] In this embodiment, in step 1, when the drone takes pictures, the overlapping area of ​​adjacent images is greater than 30%, and the camera intrinsic parameters are recorded, including focal length, optical center position and distortion parameters.

[0036] Specifically, "the overlap area of ​​adjacent images during drone shooting is greater than 30%" means that when a drone is acquiring multi-view facade images, the area of ​​the scene covered by any two adjacent images exceeds 30% of the total area of ​​a single image. This overlap rate can be controlled in several ways. For example, flight parameters such as flight altitude, camera focal length, shooting interval, and flight speed can be preset through drone flight path planning software to ensure that the required overlap rate is met under different flight conditions. Alternatively, during drone flight, real-time image processing algorithms can monitor the feature point matching of adjacent images and dynamically adjust the shooting strategy based on the matching results, such as slowing down the flight speed or increasing the shooting frequency, to maintain the required overlap rate.

[0037] "Recording camera intrinsic parameters" refers to acquiring and storing the internal geometric parameters of the camera mounted on the drone. These parameters are crucial for describing the camera's internal geometric properties; they project points in three-dimensional space onto a two-dimensional image plane, forming the basis for accurate 3D reconstruction and geometric correction. Recording camera intrinsic parameters can be achieved by offline calibration of the drone's camera using professional camera calibration tools (such as a checkerboard calibration board) before the drone takes pictures, obtaining its focal length, optical center position, and distortion parameters, and storing these parameters in the system. Alternatively, some high-end drone systems may have built-in high-precision camera intrinsic parameter sensors or self-calibration modules that can acquire and record camera intrinsic parameters in real-time or near real-time during flight.

[0038] "Focal length, optical center position, and distortion parameters" are the core components of camera intrinsic parameters. The focal length determines the image magnification and angle of view; the optical center position is the projection of the image coordinate system origin onto the image plane, usually expressed as pixel coordinates; distortion parameters describe the inherent geometric distortion of the lens, including radial and tangential distortion, which need to be corrected to obtain accurate image geometric information. These parameters are typically stored in matrix form, such as the camera intrinsic parameter matrix K and the distortion coefficient vector, and are calculated and recorded during camera calibration.

[0039] In the image acquisition stage of step 1, by ensuring that the overlap area of ​​adjacent images captured by the UAV is greater than 30%, sufficient feature point matching is provided for the subsequent motion reconstruction of the structure, significantly improving the success rate and robustness of feature point matching, effectively reducing matching ambiguity, and thus ensuring the density and accuracy of the sparse 3D point cloud. Simultaneously, camera intrinsic parameters, including focal length, optical center position, and distortion parameters, are recorded, providing precise camera geometry information for motion reconstruction. This allows for accurate distortion correction of the original image, avoiding the introduction of geometric errors during 3D reconstruction. This precise image acquisition strategy lays a solid foundation for the construction and optimization of the 3D Gaussian splash model in step 3, ensuring the geometric accuracy and texture details of the 3D scene representation. This, in turn, improves the generation quality of the facade orthographic projection image in step 4 and the accuracy of window detection in step 5. Ultimately, it ensures the reliability and accuracy of the window-to-wall ratio calculation in step 6, effectively solving the problem of decreased accuracy in 3D reconstruction and window-to-wall ratio calculation caused by insufficient image acquisition quality.

[0040] In this embodiment, in step 2, the open-source software COLMAP is used to perform motion recovery structure reconstruction, and a sequential matching strategy is used to match feature points for continuously captured image data.

[0041] Specifically, the open-source software COLMAP is a mature and widely used Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline that can automatically reconstruct 3D structures and camera poses from unordered image sets. COLMAP integrates various efficient feature detection, feature matching, geometric verification, and bundle adjustment algorithms, enabling it to stably output high-precision camera intrinsics, camera poses, and sparse 3D point clouds. Besides COLMAP, those skilled in the art can also choose other similar open-source or commercial SfM software, such as OpenMVG, VisualSFM, or Agisoft Metashape, which can also achieve sparse 3D reconstruction from multi-view images and provide the required camera parameters and point cloud data. The sequential matching strategy refers to prioritizing or only performing feature point matching on adjacent image pairs in the image sequence during the feature point matching stage. For example, for a sequence of images captured continuously by a drone, matching can be performed only on feature points between the i-th frame and the (i+1)-th frame, or within a certain window (such as between the i-th frame and the (i+k)-th frame, where k is a preset small integer). This strategy differs from traditional global matching (attempting to match all image pairs) or matching methods based on image content similarity. In practice, this can be achieved by configuring the matcher parameters in COLMAP or other SfM software, such as setting the matching mode to "sequential matching" or "sequence-based matching," or by controlling the generation of matching pairs programmatically.

[0042] In this embodiment, in step 3, the parameters of each Gaussian element include position, covariance matrix, opacity, and color coefficients represented by spherical harmonic functions. The rotation matrix is ​​parameterized by quaternions, and the covariance matrix is ​​decomposed into the product of the rotation matrix and the scaling matrix.

[0043] Specifically, the position parameter of the Gaussian primitive is used to determine its precise geometric center in 3D space, ensuring accurate placement in the scene. This position parameter can be represented using a (x, y, z) triple in a Cartesian coordinate system, or it can be directly set to the coordinates of a point obtained from a sparse 3D point cloud during initialization. The covariance matrix describes the shape, size, and orientation of the Gaussian primitive in 3D space, capturing the local structure of geometric details in the scene. This covariance matrix can be directly represented using a 3x3 symmetric positive definite matrix, or indirectly represented by decomposing it into the product of a rotation matrix and a scaling matrix, allowing for independent control of orientation and scale. The opacity parameter represents the degree to which the Gaussian primitive blocks light, used to simulate occlusion relationships and semi-transparency effects of objects in the scene, making the rendering results more realistic. This opacity can be a scalar value between 0 and 1, where 0 represents complete transparency and 1 represents complete opacity, and can be optimized during training. Color coefficients represented by spherical harmonics are used to efficiently encode color information of Gaussian elements in different viewing directions to adapt to complex lighting and material reflection characteristics, resulting in a realistic appearance of the rendered scene from different viewpoints. This can be achieved by using a set of spherical harmonic coefficients to represent each color channel (R, G, B), for example, using a third-order spherical harmonic function.

[0044] To further optimize the rotation representation, the rotation matrix is ​​parameterized using quaternions. Quaternions, as a singularity-free, compact, and computationally efficient rotation representation method, effectively avoid the gimbal lock problem that may occur with Euler angles and other representation methods at specific angles, thus making the optimization process more stable and efficient. During optimization, the quaternions can be directly updated and then converted into a rotation matrix for covariance matrix decomposition. Furthermore, the covariance matrix is ​​decomposed into the product of the rotation matrix and the scaling matrix. This decomposition decouples the orientation and scale information of the Gaussian elements, allowing the model to independently adjust the orientation and size of the Gaussian elements during optimization. This can be achieved by obtaining the rotation and scaling matrices through singular value decomposition (SVD) or eigenvalue decomposition. During optimization, the diagonal elements (representing the scale of the three principal axes) of the rotation matrix (using quaternions) and the scaling matrix are optimized separately, and then combined into the covariance matrix.

[0045] In this embodiment, the formula for calculating the window-to-wall ratio of a single facade in step 6 is as follows: ; in, express WWR on the facade express Total pixel area of ​​the facade window region. express Total pixel area of ​​the facade wall area; The formula for calculating the overall window-to-wall ratio of a building is: ; in, for The area weight of the facade, where n is the total number of facades.

[0046] Specifically, the formula for calculating the window-to-wall ratio of a single facade is used to quantify the window-to-wall ratio of a single building facade. Among other things, express The total pixel area of ​​the facade window region is obtained by accumulating the pixels of the detected window region in the orthographic projection image by the deep learning object detection model in step 5. express The total pixel area of ​​the facade wall region represents the total pixel area of ​​the wall region on this facade. Its value can be obtained by subtracting the total pixel area from the total pixel area of ​​the facade orthographic projection image generated in step 4. The window-to-wall ratio can be obtained either directly by calculating the total pixel area of ​​the facade and subtracting the window pixel area. This formula is based directly on pixel area calculation, avoiding area estimation errors caused by perspective distortion or occlusion in traditional methods, and ensuring the accuracy of the window-to-wall ratio calculation for a single facade.

[0047] Formula for calculating the window-to-wall ratio of an overall building Used to comprehensively evaluate the window-to-wall ratio of the entire building. for The area weight of the facade can be set based on factors such as the actual physical area of ​​the facade, its importance in the overall building energy consumption, or its orientation. For example, The area information from the facade parameters extracted in step 4 can be used directly, or a weighted average can be applied based on the contribution of each facade to the overall thermal performance of the building. 'n' represents the total number of facades in the target building, obtained by counting the facades extracted in step 4. This formula uses a weighted average, rather than a simple arithmetic average, which more reasonably reflects the contribution of different facades to the overall window-to-wall ratio, thereby improving the accuracy and representativeness of the overall window-to-wall ratio calculation.

[0048] This embodiment clarifies the specific calculation formulas for the window-to-wall ratio of a single facade and the overall building, effectively solving the problems of inaccurate calculations, improper weight allocation, and errors easily introduced when processing multiple facades due to the lack of a unified calculation standard. Specifically, the calculation of the window-to-wall ratio of a single facade is based on the ratio of the total pixel area of ​​the window area to the total pixel area of ​​the wall area, directly utilizing the perspective-free orthographic projection image generated in step 4 and the window area accurately detected in step 5, ensuring the accuracy and reliability of the calculation. Building upon this, the calculation of the overall building window-to-wall ratio introduces the area weight of each facade, comprehensively considering the actual size and importance of different facades through a weighted average, avoiding the bias that may be caused by simple averaging, so that the final overall window-to-wall ratio result can more realistically and comprehensively reflect the actual situation of the building. This explicit and weighted calculation method not only improves the accuracy and consistency of the window-to-wall ratio data, but also provides a solid data foundation for subsequent building energy efficiency assessments and design optimizations.

[0049] In this embodiment, in step 3, the training optimization uses a weighted combination of L1 loss and structural similarity loss as the loss function, dynamically adjusts the number of Gaussians through an adaptive density control strategy, and sets different learning steps for position, scale, rotation, opacity and color coefficient.

[0050] The loss function is a key metric for measuring the difference between the model's predictions and the true values. L1 loss, or Mean Absolute Error, assesses the difference by calculating the average of the absolute differences between the predicted and true values. Its characteristic is that it is insensitive to outliers, helping to maintain gradient stability during training. Structural Similarity Loss (SSIM), on the other hand, comprehensively evaluates the similarity between two images from three dimensions: brightness, contrast, and structure. This evaluation method is more in line with human perception of image quality. In practical applications, this weighted combination can be implemented in various ways.

[0051] The Gaussian quantity refers to the total number of basic units used to represent a 3D scene in a 3D Gaussian splash model. An adaptive density control strategy automatically increases or decreases the number of these Gaussian primitives during model training, based on scene complexity and rendering requirements. This strategy can be implemented in ways including, but not limited to, the following two: One approach is to continuously monitor the contribution or gradient magnitude of each Gaussian primitive in the rendered image. For regions with high contribution or large gradients, indicating rich detail or significant reconstruction errors, the corresponding Gaussian primitives can be split to increase their density and capture more detail. Conversely, for regions with low contribution or small gradients, redundant Gaussian primitives can be merged or removed to reduce computational resource consumption. Another approach is to periodically evaluate the "opacity" or "scale" parameters of the Gaussian primitives. For example, primitives with excessively low opacity or small scale can be considered unimportant or redundant and removed. Meanwhile, in areas with large rendering errors or insufficient Gaussian density, new primitives can be added by copying existing Gaussian primitives and making minor perturbations to their position, scale, and other parameters, thereby improving local reconstruction accuracy. This adaptive adjustment strategy of the number of Gaussians can effectively optimize the efficiency and accuracy of scene representation, avoid wasting computational resources in simple areas, and provide sufficient detail representation in complex areas.

[0052] The learning step size is a key hyperparameter in deep learning optimization algorithms, determining the magnitude of model parameter updates in each iteration. In a 3D Gaussian splatter model, position, scale, rotation, opacity, and color coefficient are the core parameters describing each Gaussian primitive, and their impact on the model's convergence speed and final rendering quality varies. Differentiated learning steps for these different parameter types can be implemented in several ways. One approach is to manually set different fixed learning steps for each parameter group. For example, since position and scale parameters are crucial for the accurate representation of scene geometry, smaller learning steps may be needed for fine-tuning, thus avoiding drastic oscillations during training. Opacity and color coefficient, on the other hand, might benefit from slightly larger learning steps to accelerate convergence to a suitable visual representation. Another approach is to combine more advanced optimizers (such as AdamW, SGD with momentum, etc.) and learning rate schedulers (such as cosine annealing, step decay, etc.) to set different initial learning step sizes or learning rate decay strategies for different parameter groups. For example, a slower decay curve can be set for position parameters, while a faster decay curve can be set for color coefficients, thus achieving differentiated learning step size adjustments throughout the training process. This differentiated learning step size setting can more effectively guide the update of model parameters, accelerate the training process, improve the stability of model convergence, and ultimately enhance the rendering quality.

[0053] In this embodiment, in step 4, the building facade extraction adopts an automatic method based on RANSAC plane segmentation and minimum bounding box, or a semi-automatic method by manually calibrating corner points. The facade parameters include corner point coordinates, normal direction, width and height information, and area weight.

[0054] Specifically, building facade extraction refers to identifying and separating the various vertical surfaces of a target building from a 3D scene representation. Its purpose is to provide clear geometric boundaries and spatial references for subsequent orthographic projection image generation, forming the basis for accurate calculation of the window-to-wall ratio. This extraction process can be achieved by analyzing the geometric features in 3D point clouds or 3D models, such as using techniques like clustering along normal directions and plane fitting to identify potential facade areas; or by combining image processing techniques to perform edge detection and region segmentation on rendered 2D images, and then backprojecting the results back into 3D space for verification and refinement.

[0055] The automated method based on RANSAC plane segmentation and minimum bounding boxes aims to automate the identification and parameter acquisition of building facades, improving efficiency and reducing manual intervention. RANSAC plane segmentation can robustly detect the main planar structures from 3D data containing noise and outliers, effectively handling complex scenarios. The minimum bounding box is used to accurately define the boundaries of the identified planes, ensuring the accuracy of the facade area. This method first extracts or generates point cloud data from a 3D Gaussian splash model, then applies the RANSAC algorithm to iteratively select random point sets to fit the plane model, and counts the number of points that meet the in-model point threshold, repeating this process multiple times to find the optimal plane. The minimum bounding box can be calculated based on the in-plane points segmented by RANSAC, calculating their minimum axis-aligned bounding boxes in the planar coordinate system. Another implementation is that RANSAC plane segmentation can be optimized by combining normal information, prioritizing points with similar normal directions for plane fitting. The calculation of the minimum bounding box can also be performed using principal component analysis (PCA) to determine the optimal bounding box orientation for a closer fit to the facade shape.

[0056] As a complement to automated methods, the semi-automatic method of manually calibrating corner points aims to handle complex or non-standard building structures where automated methods may fail. It ensures the reliability and accuracy of facade extraction through manual intervention. By precisely calibrating key corner points manually, the system can be provided with a clear definition of facade boundaries. For example, users can select four or more corner points of the building facade through an interactive interface on a rendered 3D scene view or orthographic projection image. The system automatically fits the plane and calculates facade parameters based on these corner points. Alternatively, an initial facade area can be provided, allowing users to drag or adjust the boundary points of that area to refine the facade scope, with the system updating facade parameters in real time.

[0057] The acquired facade parameters are a key dataset describing the geometric features and spatial attributes of the building facade. They are essential inputs for subsequent orthographic image generation, window detection, and window-to-wall ratio calculation. These parameters include: Corner coordinates define the precise position and shape of the facade in three-dimensional space and are the basis for determining the facade boundary. For example, they can be the three-dimensional coordinates of the four vertices of the facade or the two-dimensional coordinates of the facade in a certain local coordinate system. They are combined with the facade's normal and center point to define its three-dimensional position.

[0058] The normal direction indicates the orientation of the facade in three-dimensional space. It is crucial for determining the viewing angle of the orthographic projection and eliminating foreground obstructions. For example, it can be a three-dimensional vector representing the unit normal vector of the facade plane, or it can be calculated by the cross product of three non-collinear points of the facade.

[0059] Width and height information provides the dimensions of the facade for calculating the facade area. It is a direct input for calculating the window-to-wall ratio. For example, it can be the length and width values ​​of the facade in its own coordinate system, or it can be obtained by calculating the distance between the corner points of the facade.

[0060] Area weighting is used to weight the contribution of different facades when calculating the window-to-wall ratio of the overall building, reflecting the relative importance or area proportion of each facade in the overall building. For example, it can be the actual area value of each facade, or the proportion of the area of ​​each facade to the total area of ​​all facades.

[0061] In this embodiment, spatial filtering is performed before orthographic projection rendering. The in-plane boundary is widened by 2% for both the width and height of the facade, and a certain thickness range is left on the front and back sides in the normal direction to preserve the effective structure of the facade and suppress interference from irrelevant scenes.

[0062] Specifically, spatial filtering is performed before orthographic projection rendering to precisely define the building facade area to be rendered, thereby avoiding the inclusion of irrelevant scene elements in the orthographic image. This spatial filtering can be implemented in several ways. For example, a 3D bounding box or truncated cone can be constructed based on extracted building facade parameters (such as corner coordinates, normal direction, and width and height information), and only the 3D Gaussian primitives located within this bounding box or truncated cone can be selected for rendering. Another approach is to organize the Gaussian primitives in the 3D Gaussian splash model using a spatial index structure (such as an octree or KD tree), and quickly filter out the Gaussian primitives intersecting with the target facade area by querying this index structure before rendering.

[0063] Furthermore, widening the in-plane boundary by 2% for both width and height is to moderately expand the facade boundary on the two-dimensional plane. This expansion effectively compensates for minor errors that may exist during the facade parameter extraction process, or for incomplete boundary recognition due to the complexity of the building structure. For example, after determining the minimum bounding rectangle of the facade boundary, the width and height of this rectangle can be increased by 2% respectively, resulting in a new, slightly larger rendering area. Alternatively, a geometric expansion operation can be performed on the two-dimensional polygon representing the facade boundary to ensure that windows or other effective structures at the facade edges are not lost due to clipping.

[0064] Simultaneously, a certain thickness range is reserved on both the front and back sides along the normal direction, aiming to define a three-dimensional depth interval along the normal direction of the building facade. This not only includes the physical thickness of the facade itself (such as walls, window frames, etc.), but also effectively excludes non-facade components located far in front of or behind the facade, such as trees in the foreground, vehicles, or other buildings in the background. This thickness range can be set based on empirical values, such as leaving 0.5 meters to 2 meters on both sides, or dynamically adjusted according to the building type and the actual scene. In practice, two parallel planes can be set on both sides of the facade's central plane according to the facade's normal vector, and only the three-dimensional Gaussian elements located between these two planes can be rendered.

[0065] In this embodiment, in step 5, the window detection model is based on the YOLOv11 architecture, pre-trained on the COCO dataset, and then trained using transfer learning on the street view window dataset.

[0066] Specifically, the window detection model is based on the YOLOv11 architecture. YOLO (You Only Look Once) is a highly efficient single-stage object detection algorithm, renowned for its speed and high accuracy. YOLOv11, as its latest iteration, further optimizes the network structure, loss function, and training strategy, enabling end-to-end real-time object detection. This architecture ensures the real-time performance and accuracy of window detection, avoiding the computational redundancy of traditional multi-stage detection methods (such as the R-CNN series). In practical applications, the official implementation of YOLOv11 can be used, configured with its recommended backbone networks (such as CSPDarknet53, YOLOv8-C2f, etc.) and detection heads. Alternatively, other efficient versions of the YOLO series, such as YOLOv8 or YOLOv9, can also be used, which also possess the advantages of single-stage detection and improve performance through lightweight design or more advanced feature fusion mechanisms.

[0067] This window detection model is pre-trained on the COCO dataset. Pre-training refers to training a deep learning model on a large-scale, diverse dataset to learn general feature representation capabilities. COCO (Common Objects in Context) is a large image dataset containing a large number of everyday object categories and rich scenes. Pre-training on the COCO dataset aims to provide a strong generalization foundation for the window detection model, enabling it to recognize basic visual patterns and object features in images, thereby accelerating the learning of subsequent specific tasks and improving model performance. Typically, the weights of a YOLOv11 model pre-trained on the COCO dataset are used as initial parameters. In addition to the COCO dataset, pre-training on other large-scale image classification or object detection datasets such as ImageNet can also be chosen to obtain a wider range of visual feature extraction capabilities.

[0068] The model was trained using transfer learning on the Street View Window dataset. Transfer learning refers to applying a model trained on a source task to a target task through fine-tuning. The Street View Window dataset is a dataset specifically designed for building facade window detection, containing a large number of window images with different types, lighting conditions, and occlusions. By training the model on the Street View Window dataset, the model can adapt to the feature distribution of a specific domain (street view building windows), overcoming challenges that general models may encounter in complex street view environments, such as perspective distortion, occlusion, lighting variations, and window diversity, thereby significantly improving the accuracy and robustness of window detection. In implementation, based on the COCO pre-trained model, the Street View Window dataset can be used to fine-tune all or some layers of the model, adjusting the learning rate and training cycle to avoid overfitting. Furthermore, more refined transfer learning strategies can be employed, such as freezing the feature extraction layer of the pre-trained model and training only the detection head, or using domain adaptation techniques to better bridge the differences between the source and target domains.

[0069] In this embodiment, in step 5, at least one of the following constraints are introduced during the detection process: test enhancement, that is, the image is flipped, enlarged, and reduced and then repeatedly detected; abnormal box filtering, that is, the detection boxes are filtered according to the aspect ratio, area size, brightness and median value difference; and detection box clustering, that is, for regularly arranged windows, the detection box position is corrected by clustering algorithm, and the undetected position is re-detected with low confidence.

[0070] Specifically, test augmentation is a method that applies data augmentation techniques during the inference phase to improve the model's robustness to changes in the input image. This can be achieved by horizontally or vertically flipping the original image to generate a new image for detection; or by enlarging or reducing the image at different scales to simulate targets at different scales, and then detecting each of these transformed images separately. By repeatedly detecting multiple transformed images, more stable and comprehensive detection results can be obtained, effectively addressing detection instability caused by viewpoint, distance, or occlusion.

[0071] Outlier box filtering aims to identify and remove bounding boxes that do not conform to the physical characteristics of windows or detection logic, thereby improving the accuracy of detection results. Its implementation can include: setting a threshold based on the aspect ratio of the detection box, for example, excluding rectangles with excessively large or small aspect ratios, since windows typically have relatively regular proportions; setting upper and lower limits based on the pixel area of ​​the detection box to filter out detection boxes that are too small (potentially noise or false detections) or too large (potentially containing non-window areas); or analyzing the brightness distribution of pixels within the detection box, such as calculating the average or median brightness and comparing it with the overall or local background brightness of the image to exclude detection boxes with abnormal brightness, such as excessively low brightness indicating occlusion or excessively high brightness indicating reflection.

[0072] Detection box clustering focuses on optimizing detection results by leveraging the typically regular arrangement of windows on building facades. Its implementation can include: First, using the center points of the initially detected window bounding boxes as clustering objects, and employing clustering algorithms such as K-means or DBSCAN to group these center points, identifying groups of windows with regular arrangement patterns. Then, based on the clustering results, the positions of the detection boxes within each group are corrected. For example, the average position of all detection boxes within the group is calculated, or their positions are adjusted by fitting lines / grids to better conform to the geometric characteristics of the regular arrangement. Furthermore, for potentially undetected gaps within the regular arrangement, predictions can be made based on the size and spacing of surrounding detected windows, and low-confidence re-detections can be performed at these predicted locations. This involves lowering the confidence threshold of the detection model to attempt to recall real windows that were initially filtered out due to slightly lower confidence levels.

[0073] In this embodiment, a thermal performance parameter association step is also included: establishing a transparent component thermal performance parameter database, querying and associating the component's thermal performance parameters from the database based on the component category output by semantic segmentation, wherein the thermal performance parameters include at least one of the heat transfer coefficient and the solar heat gain coefficient.

[0074] The thermal performance parameter association step aims to bind the identified transparent facade components (such as windows, glass doors, and glass curtain walls) to their corresponding physical and thermal properties, providing fundamental data support for subsequent building energy consumption assessments, energy-saving renovation strategy formulation, or more refined window-to-wall ratio calculations. This step can use a programming interface (API) or data connector to retrieve the corresponding thermal performance parameters from a pre-set database in real time, using the component category information output by the semantic segmentation model as a key. Alternatively, after the semantic segmentation model completes identification, it generates an intermediate file containing component category and location information. A separate post-processing module then reads this file and performs batch database queries based on component category, appending thermal parameters to each identified component instance.

[0075] The transparent component thermal performance parameter database is a knowledge base storing the thermal performance parameters of various transparent facade components. It provides a structured, queryable data source, ensuring that transparent components of different types, specifications, and materials can all find their corresponding thermal properties. This database can be a relational database (such as MySQL or PostgreSQL), containing fields such as component category ID, component name, heat transfer coefficient (U-value), and solar heat gain coefficient (SHGC). Data can be entered and maintained according to national or industry standards, product manuals, experimental data, etc. Alternatively, this database can also be a non-relational database (such as MongoDB) or a simple key-value store, storing the mapping relationship between component categories and thermal parameters in JSON or XML format.

[0076] In this embodiment, in step 6, the window-to-wall ratio calculation includes either a categorized window-to-wall ratio calculation or a weighted window-to-wall ratio calculation, wherein the weight of the weighted window-to-wall ratio is set according to the thermal performance parameters.

[0077] Specifically, "classified window-to-wall ratio calculation" refers to calculating the window-to-wall ratio separately for different categories of transparent facade components. For example, in the method described above, after identifying various transparent component categories such as standard windows, glass doors, and glass curtain walls through a semantic segmentation model, the window-to-wall ratio for each component type can be calculated separately. One implementation is to sum the pixel areas of all components in each identified transparent component category, and then calculate the ratio between this ratio and the total pixel area of ​​the corresponding facade to obtain the classified window-to-wall ratio for that category. Another implementation is to calculate the proportion of each transparent component category in the entire transparent area of ​​the facade, and combine this with the overall window-to-wall ratio to estimate their respective contributions.

[0078] The "weighted window-to-wall ratio calculation" assigns different weights to the thermal performance parameters of different transparent components when calculating the overall window-to-wall ratio. For example, in the methods described above, thermal performance parameters of the transparent components, such as the heat transfer coefficient (U-value) or the solar heat gain coefficient (SHGC), are already associated. When calculating the weighted window-to-wall ratio, weights can be set based on these parameters. One implementation is to set the weights as functions of the thermal performance parameters. For example, for the heat transfer coefficient, the weight can be set to be inversely proportional to the heat transfer coefficient (i.e., the smaller the U-value, the larger the weight, indicating better energy-saving performance) or directly proportional to the heat transfer coefficient (i.e., the larger the U-value, the larger the weight, indicating a greater impact on energy consumption), depending on the evaluation objective. For the solar heat gain coefficient, the weight can be set to be directly proportional to the SHGC to reflect its contribution to solar radiation heat gain. Another implementation is that the weights can be adjusted according to preset energy-saving standards or design objectives. For example, components that do not meet energy-saving standards can be given higher weights to highlight the necessity for improvement.

[0079] In this embodiment, the training dataset of the semantic segmentation model is further proposed to include pixel-level multi-class annotations of transparent facade components. The annotation categories are associated with building type or construction year to infer the corresponding thermal performance parameters.

[0080] Specifically, the training dataset for a semantic segmentation model is a dataset used to train a deep learning model, containing input data (e.g., images of building facades) and their corresponding expected outputs (e.g., pixel-level category labels). Pixel-level multi-class labeling refers to assigning a specific category label to each pixel in the image, such as "standard window," "glass door," "glass curtain wall," or "wall," and these category labels are meticulously subdivided for transparent facade components. This labeling method enables the semantic segmentation model to learn the precise boundaries and specific types of transparent components. For example, manual labeling tools, such as Labelme and VGG Image Annotator (VIA), can be used to meticulously depict and classify the acquired building facade images pixel by pixel. For a window, its glass portion and frame portion can be labeled as different transparent component subcategories, or further subdivided according to their function, material, etc. Another approach is to use semi-automatic labeling tools, combined with a pre-trained model, for initial segmentation, followed by manual correction and refinement to improve labeling efficiency and accuracy.

[0081] It should be noted that the above-mentioned technical features can be combined with each other to form various embodiments not listed above, all of which are considered to be within the scope of this invention specification; and, for those skilled in the art, improvements or modifications can be made based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An automated method for detecting the window-to-wall ratio of buildings based on deep learning and 3DGS, characterized by: Includes the following steps: Step 1: Collect multi-view facade images of the target building using a drone; Step 2: Use the structure-of-motion reconstruction method to perform sparse 3D reconstruction on the acquired images to obtain camera intrinsic parameters, camera pose, and sparse 3D point cloud; Step 3: Based on the sparse 3D point cloud, camera intrinsic parameters and camera pose, a 3D scene representation is constructed using 3D Gaussian splashing technology, and then trained and optimized using a differentiable rasterizer renderer. Step 4: Extract the facade parameters of the target building, and use the three-dimensional Gaussian splash model to generate an orthographic projection image of the facade using the orthographic projection method, wherein the interference of foreground occlusions is eliminated by setting the rendering distance; Step 5: Use a deep learning object detection model to detect windows in the orthographic projection image and output the pixel coordinates of the window region; Step 6: Calculate the window-to-wall ratio of a single facade based on the window pixel area and the facade pixel area, and calculate the overall building window-to-wall ratio by combining the area weights of each facade. 2.The deep learning and 3DGS-based automatic detection method of building window-to-wall ratio according to claim 1, wherein, In step 1, the overlapping area of ​​adjacent images is greater than 30% when the drone takes pictures, and the camera intrinsic parameters, including focal length, optical center position and distortion parameters, are recorded. 3.The deep learning and 3DGS-based automatic detection method of building window-to-wall ratio according to claim 1, wherein, In step 2, the open-source software COLMAP is used for motion recovery structure reconstruction, and a sequential matching strategy is used for feature point matching for continuously captured image data. 4.The method of claim 1, wherein the method further comprises: determining a window-to-wall ratio of the building based on the 3D point cloud data. In step 3, the parameters of each Gaussian element include position, covariance matrix, opacity, and color coefficients represented by spherical harmonic functions. The rotation matrix is ​​parameterized by quaternions, and the covariance matrix is ​​decomposed into the product of the rotation matrix and the scaling matrix.

5. The automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to claim 1, characterized in that, In step 3, during initialization, each point in the sparse 3D point cloud output by the motion recovery structure is initialized as a 3D Gaussian element, with its position set to point cloud coordinates, its covariance set to an isotropic small sphere, its color set to the RGB value of the point cloud, and its opacity set to 0.

1.

6. The automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to claim 1, characterized in that, In step 3, the training optimization uses a weighted combination of L1 loss and structural similarity loss as the loss function, dynamically adjusts the number of Gaussians through an adaptive density control strategy, and sets different learning steps for position, scale, rotation, opacity and color coefficient.

7. The automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to claim 1, characterized in that, In step 4, the building facade extraction adopts an automatic method based on RANSAC plane segmentation and minimum bounding box, or a semi-automatic method through manual corner point calibration. The facade parameters include corner point coordinates, normal direction, width and height information, and area weight.

8. The automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to claim 1, characterized in that, In step 4, spatial filtering is performed before orthographic projection rendering. The in-plane boundary is widened by 2% according to the width and height of the facade, and a certain thickness range is left on the front and back sides in the normal direction to preserve the effective structure of the facade and suppress interference from irrelevant scenes.

9. The automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to claim 1, characterized in that, In step 5, the window detection model is based on the YOLOv11 architecture, pre-trained on the COCO dataset, and trained by transfer learning on the street view window dataset. At least one of the following constraints is introduced during the testing process: Enhanced testing: Repeat the detection process after flipping, enlarging, and shrinking the image; Abnormal bounding box filtering: Detection boxes are filtered based on aspect ratio, area size, and the difference between brightness and median value; Detection box clustering: For windows arranged in a regular pattern, the position of the detection boxes is corrected by a clustering algorithm, and the undetected positions are re-detected with low confidence.

10. The automated window-to-wall ratio detection method for buildings based on deep learning and 3DGS according to claim 1, characterized in that, In step 6, the formula for calculating the window-to-wall ratio of a single facade is: ； in, express WWR on the facade express Total pixel area of ​​the facade window region. express Total pixel area of ​​the facade wall area; The formula for calculating the overall window-to-wall ratio of a building is: ； in, for The area weight of the facade, where n is the total number of facades.

Images