Mask-RCNN instance segmentation model-based building facade hole opening rate intelligent batch measurement and calculation method

Abstract

The invention discloses a Mask-RCNN instance segmentation model-based building facade aperture ratio intelligent batch measurement and calculation method, and belongs to the technical field of urban and rural planning heritage protection. The method comprises the following steps: firstly, acquiring streetscape photos and video data of a block and preprocessing the streetscape photos and video data to construct a basic data set trained by a Mask-RCNN (Regional Convolutional Neural Network) model; next, the processed data set is input into a Mask-RCNN model for training, and an optimal model weight is obtained; and the trained model is used to carry out instance segmentation and labeling on the streetscape building style and feature elements including doors and windows. And performing same automatic perspective correction on the building facade image and the Mask mask image obtained by model reasoning through an OpenCV library of the Python platform. And finally, performing intelligent batch calculation by utilizing a computer vision technology in a Python programming language in combination with a geometric calculation method to obtain the hole opening rate of the building facade. Compared with the prior art, the method has the advantages that the efficiency and accuracy of building facade holing rate measurement and calculation are improved, and scientific basis and technical support are provided for block protection and building style and feature management and control in urban and rural planning.

Description

Technical Field

[0001] The invention relates to an intelligent batch calculation method for building facade opening rate based on a Mask-RCNN instance segmentation model, and belongs to the field of urban and rural planning heritage protection and technology. Background Art

[0002] With the acceleration of urbanization, the protection and restoration of historical blocks and historical sites are becoming increasingly important. These areas not only carry rich historical and cultural information, but also show unique architectural style. When protecting and renovating, it is the core task of urban and rural planning heritage protection to ensure that the newly built or renovated buildings are coordinated and unified with the original architectural style and features of the historical blocks and sites while continuing the original architectural style and features. However, in actual operation, whether it is identifying buildings that are inconsistent with the original style and features, or setting constraints on the appearance characteristics of new buildings to achieve effective style and features control, they are faced with challenges of accuracy, quantification and efficiency. Therefore, being able to accurately, quantitatively and quickly identify and measure the key style and features of buildings in historical blocks and sites is of great significance for scientifically promoting heritage protection and inheriting historical culture. In this process, the opening rate is an important indicator for measuring the architectural style and features, and its importance cannot be ignored.

[0003] The opening ratio refers to the ratio of the area of ​​opening elements such as doors and windows on the facade of a building to the area of ​​the entire facade. It is a key indicator that reflects the local style characteristics of the building, such as ventilation, lighting and structure. In the field of urban and rural planning heritage protection, the adjustment of the opening ratio is directly related to the changes in the appearance and structure of the building, which is extremely critical for evaluating the preservation status of the building and formulating restoration strategies. Therefore, in the protection of historical blocks and sites, the opening ratio is not only a common feature of the style of heritage buildings, but also an important criterion for evaluating whether the newly built buildings are coordinated with the original style. Accurately measuring the opening ratio of the building facade will help to promptly discover and prevent possible style damage, thereby effectively protecting the cultural heritage of historical blocks and sites.

[0004] However, traditional methods for calculating the opening ratio of building facades usually rely on manual field measurements and visual inspections, which are not only time-consuming and labor-intensive, but also limited by the limited human resources and the instability of subjective judgment. Especially when faced with large-scale historical blocks and sites, the efficiency and accuracy of these traditional methods are insufficient. In addition, manual measurement may cause unnecessary intrusion to heritage buildings, increasing the potential risks of heritage buildings. Therefore, there is an urgent need to develop an efficient, accurate, and non-invasive technology to batch and automatically identify and calculate the opening ratio of building facades, which is of great significance to improving the protection efficiency and quality of historical blocks and sites.

[0005] In recent years, the rapid development of deep learning and computer vision technology has provided new possibilities for solving this problem. By combining deep learning with image processing technology, it is possible to efficiently, accurately and non-invasively identify, measure and monitor the openings on building facades, thereby providing dynamic data support and scientific decision-making basis for the protection and management of the appearance of historical blocks and areas.

[0006] Therefore, the present invention proposes a method for intelligent batch calculation of the opening rate of building facades based on the Mask-RCNN instance segmentation model. The method uses the Mask-RCNN model to realize intelligent batch recognition of opening elements such as doors and windows on building facades, and uses OpenCV technology to perform automatic perspective correction of images, and analyzes the Mask mask image to intelligently batch calculate the opening rate. This method can not only accurately identify and calculate the opening rate of a large number of building facades, but also, due to its high efficiency, can support the continuous monitoring of the opening rate of block buildings, and provide strong technical support for the protection and management of historical blocks and areas. Summary of the invention

[0007] The present invention aims to solve the limitations of the above-mentioned existing building facade opening rate calculation method, and proposes an intelligent batch calculation method for building facade opening rate based on the Mask-RCNN instance segmentation model. The method can realize the efficient and accurate identification and calculation of the building facade opening rate, and provide scientific reference data for building protection, restoration and new construction projects. At the same time, the method can also assist in the dynamic monitoring of key elements of the building facades of the block, and provide technical support and decision-making reference for the management and control of the block style.

[0008] To achieve the above object, the technical solution adopted by the present invention is an intelligent batch calculation method for the opening rate of building facades based on the Mask-RCNN instance segmentation model. The implementation steps of the method are as follows:

[0009] Step (1) collects street view photos and video data of the block where the building facade opening rate is to be measured, automatically batch-shots the video at an appropriate frame rate, and pre-processes it in Photoshop software; the pre-processing includes adjusting the image resolution, cropping unnecessary edge parts, etc. After pre-processing, it is used as the basic data set for building the Mask-RCNN model training.

[0010] Step (2): Input the basic data set into the Mask-RCNN model for training. During the training process, use optimization algorithms such as Adam or SGD, update the weights through the back-propagation algorithm, and finally obtain the weights that perform best on the validation set as the optimal weights.

[0011] Step (3), based on the trained Mask-RCNN model, fast batch recognition is performed on street view photos and video screenshots, and instance segmentation and annotation of the opening elements on the building facades are performed.

[0012] Step (4), using OpenCV technology to select street view photos or video screenshots of the building facades whose opening ratio needs to be calculated, and the corresponding Mask image, i.e., Mask picture, and perform the same automatic perspective correction.

[0013] Step (5), use OpenCV technology to automatically correct the white pixels of the Mask image, eliminate irregular contours, and improve the accuracy of the calculation results.

[0014] Step (6), based on the perspective transformed image and the corresponding Mask image, automatically and intelligently calculate the ratio of white pixels in the Mask image to the total image area, and finally obtain the measurement result of the opening rate.

[0015] The steps for automatically batch-snapshotting videos at an appropriate frame rate in step (1) above are as follows:

[0016] (a) Based on the OpenCV-Python library, the video file is read through the "cv2.VideoCapture" function;

[0017] (b) Obtain information such as the frame rate (FPS) and total number of frames of the video file, and set a fixed interval (e.g., capture an image every 5 seconds);

[0018] (c) Traverse the video frames and check whether the current frame number is the predetermined screenshot frame number. If so, save the frame as an image through the "cv2.imwrite" function; if not, skip the current frame number until traversing to the predetermined screenshot frame number.

[0019] The steps of training based on the Mask-RCNN model in the above steps (2) and (3) and quickly batch-recognizing street view photos and video screenshots are as follows:

[0020] (a) Dataset preparation: Based on the customized building facade opening feature classification, the basic dataset is annotated by Labelme;

[0021] (b) Mask-RCNN model training: First, modify the data configuration file, including data category, data path, training rounds, etc.; secondly, modify the number of categories in the Mask-RCNN model configuration file; then input the training weights, data configuration file, model configuration file path and training rounds into the Mask-RCNN model for training, and enable tensorbord to view the training parameters.

[0022] The dataset is divided into a training set and a validation set. The Mask-RCNN model is trained on the training set and the performance is evaluated on the validation set. Indicators (such as mAP) are monitored on the validation set. If there is no improvement for several consecutive epochs, training is stopped early to prevent overfitting. At the end of each epoch, the current optimal weights (usually the weights that perform best on the validation set) are recorded. After training is completed, the optimal weights of the Mask-RCNN model are obtained;

[0023] (c) Mask-RCNN model reasoning: The street view dataset to be identified and the optimal weight path obtained through training are input into the Mask-RCNN model for model reasoning, and finally the recognition of building facade opening elements such as doors and windows is completed, and the corresponding Mask images are exported at the same time.

[0024] In the above step (4), the steps of using OpenCV technology to automatically correct the perspective of street view photos or video screenshots and mask images are as follows:

[0025] (a) Read the image by calling the “cv2.imread” function;

[0026] (b) Select four corner points to construct the building facade area of ​​the block where the opening rate needs to be calculated: call the custom event processing function "click_event" by clicking the mouse, record the current click coordinates (x, y), repeat four times, and add the selected four point coordinates to a point coordinate list.

[0027] (c) Perspective correction: First, the image to be corrected stored in a specific path is read through the "cv2.imread" function in OpenCV; secondly, the height and width of the corrected image are calculated based on the set four-point correction area, and the four corner points of the transformed image are set accordingly;

[0028] The perspective transformation matrix is ​​calculated by the "cv2.getPerspectiveTransform" function in OpenCV; finally, the "cv2.warpPerspective" function of OpenCV is applied to automatically generate the corrected image based on the perspective transformation matrix and the size of the new image.

[0029] The steps for automatically correcting the white pixels of the Mask image in the above step (5) are as follows:

[0030] (a) Read the Mask image through the "cv2.imread" function in OpenCV;

[0031] (b) Detect the contours of white pixels in the Mask image using the “cv2.findContours” function, and create an all-zero array “rect_img” of the same size as the input image and with all pixel values ​​set to zero;

[0032] (c) Calculate the minimum enclosing rectangle of each contour and draw it on rect_img, and finally automatically output the corrected image.

[0033] The steps for automatically and intelligently calculating the opening rate in the above step (6) are as follows:

[0034] (a) Using computer vision technology in the Python programming language combined with geometric calculation methods, the total area of ​​white pixels in the modified Mask image and its proportion to the entire image area are automatically identified and calculated to obtain the opening ratio of the building facade. First, use OpenCV to read the Mask image, and use the "np.sum()" function in NumPy to calculate the total number of white pixels (value 255) in the mask image and the total number of pixels in the image (usually the product of the width and height of the image). Finally, divide the number of white pixels by the total number of pixels to obtain the proportion of white area.

[0035] (b) The calculated opening ratio values ​​are subjected to intelligent cluster analysis to identify and divide the opening ratio value ranges of the building facades in the block.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] 1. Through the fusion application of the Mask-RCNN model in deep learning and image processing technology, it is possible to achieve accurate, efficient, batch and intelligent recognition of the opening rate of building facades, so as to quickly summarize and analyze the facade features of building groups;

[0038] 2. Significantly improved the efficiency and accuracy of identifying and measuring the opening rate of building facades, optimizing the limitations of traditional methods in this field;

[0039] 3. Using street view photos and video datasets, combined with intelligent recognition technologies such as Python-OpenCV, non-contact measurement can be achieved, effectively avoiding the intrusion and damage that may be caused when conducting on-site measurements of heritage buildings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a flowchart of the batch intelligent estimation method for the opening rate of building facades based on the Mask-RCNN instance segmentation model.

[0041] Figure 2 This is a diagram of labeling pictures using Labelme.

[0042] Figure 3 Example photos for batch preprocessing and model inference of street view images.

[0043] Figure 4 Schematic diagram of the inference results of the Mask-RCNN instance segmentation model.

[0044] Figure 5 Schematic diagram of Mask image.

[0045] Figure 6 This is a schematic diagram of building facade perspective correction.

[0046] Figure 7 Schematic diagram of the automatic correction of the Mask image and the calculation results of the opening rate. DETAILED DESCRIPTION

[0047] The present invention is further illustrated below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. After reading the present invention, modifications to various equivalent forms of the present invention by those skilled in the art all fall within the scope defined by the appended claims of the present application.

[0048] The method for batch intelligent calculation of the opening rate of building facades based on the Mask-RCNN instance segmentation model proposed in the present invention selects the streetscape image datasets of blocks and sections as the analysis basis, trains the Mask-RCNN model and obtains the optimal weights, thereby carrying out reasoning to realize the intelligent batch recognition of opening elements such as doors and windows in the building facades; further, combining the Mask mask image obtained based on reasoning with the image processing technology, the area of ​​the facade opening elements and its proportion in the total facade area are automatically and intelligently calculated, thereby significantly improving the efficiency of identifying and calculating the opening rate of building facades, and ensuring the accuracy of the calculation results, assisting in the in-depth understanding and effective control of the characteristics of the opening rate of block building facades.

[0049] The technical solution of the present invention is further described below in conjunction with the case of the Dashilan Historical and Cultural District in Beijing. Figure 1 As shown, the specific method is as follows:

[0050] (1) Batch collect street view photos and video data of Beijing Dashilan Historical and Cultural District, automatically batch screenshot the video at 5-second intervals, and combine the screenshots and street view photos into a street view picture set. Subsequently, the street view picture set is imported into Photoshop software for preprocessing, including adjusting the image resolution, cropping unnecessary edge parts, etc., and the pictures are selected according to the following principles to construct the basic training data set:

[0051] ① Most of the building facades in the streets and alleys should be clearly visible to avoid analysis affected by severe obstruction or blurred images.

[0052] ②The perspective distortion of the photo should be within a reasonable range and not too extreme.

[0053] (2) After annotating the dataset with Labelme, import it into the Mask-RCNN model for training to obtain the optimal weights.

[0054] (2.1) Using Labelme to label data and prepare datasets

[0055] (2.1.1) Create a new folder named mydata. In this folder, create a new folder named pic to store the street view pictures of Dashilan; create a new labelme_json to store the json folder obtained by Labelme annotation, which contains the original street view image, segmentation mask image, label visualization image, label and category configuration files, and custom annotation category name files; create a new json folder and cv2_mask folder to store the json files obtained by annotation in the Mask segmentation mask image. The custom annotation categories in this case are: door 1, door 2, window 1, window 2. Among them, door 1 is a wooden door, door 2 is a glass door, window1 is an aluminum window, and window 2 is a wooden window.

[0056] (2.1.2) Start Labelme in the mydata folder, open the pic folder, and initialize the custom feature classification.

[0057] (2.1.3) Based on the defined landscape element classification, use the Create Polygon tool to label the pictures in the pic folder one by one, and save the resulting json folder to the corresponding folder.

[0058] (2.1.4) Create a new folder named Mask-RCNN as the working directory for Mask-RCNN model training and inference, and store the official pre-trained weights mask_rcnn_coco.h5 file required for model training in the Mask-RCNN directory. Then, place the mydata folder in the Mask-RCNN directory. At this point, the data preparation for Mask-RCNN model training is completed.

[0059] (2.2) Mask-RCNN model training

[0060] (2.2.1) Use Pycharm to open the Mask-RCNN folder and modify the data and model configuration files respectively. Open the train_test.py file and modify the values ​​of the number of categories and image size. At the same time, add category labels and names in the load_shapes module. At this point, the corresponding parameter modifications for Mask-RCNN model training are completed.

[0061] (2.2.2) Complete the basic settings for training in the train_test.py file, set "dataset_root_path" to "mydata / "; set "img_floder" to dataset_root_path+"pic"; set "mask_floder" to dataset_root_path+"cv2_mask". Finally, set the training rounds for the head branch and all layers of fine-tuning to 200 rounds. At this point, the model configuration of Mask-RCNN is completed. Finally, run the train_test.py file to start Mask-RCNN model training.

[0062] (2.2.3) After the training is completed, the mask_rcnn_shapes.h5 weight file of each round will be generated in the logs directory.

[0063] (2.3) Mask-RCNN model reasoning

[0064] (2.3.1) Save the image set to be inferred in the images folder under the Mask-RCNN directory. Open the test_model.py file in Pycharm and enter class_names = ['BG', 'window 1', 'window 2', 'door 1', 'door 2'] in the COCO category name module.

[0065] (2.3.2) Run the test_model.py file to start the Mask-RCNN model reasoning, and finally complete the instance segmentation of the building facade opening elements. At the same time, export the corresponding Mask image. The reasoning results are stored in the results folder under the Mask-RCNN / images directory.

[0066] (3) Automatic perspective correction of images using OpenCV-Python technology

[0067] (3.1) Define the "draw_yolo_boxes" function in the OpenCV library to read the street view image inference results and the corresponding Mask images in the Mask-RCNN directory images / results. Figure 4 (Snipaste_2023-12-08_09-29-07.jpg) is used as an example to demonstrate the subsequent perspective correction and opening rate calculation process.

[0068] (3.2) Automatically correct image perspective

[0069] (3.2.1) Based on the OpenCV and NumPy libraries in the Python platform, read Snipaste_2023-12-08_09-29-07.jpg and select the four corner points of the building facade in the street view. Based on the above four corner points, the linear algebra function of the NumPy library is used to automatically calculate the Euclidean distance between two pairs of corner points to determine the width and height of the corrected image.

[0070] (3.2.2) According to the calculated new width and height, set the positions of the corresponding four corner points on the new image to provide a target frame for perspective transformation.

[0071] (3.2.3) Call the “cv2.getPerspectiveTransform” function to automatically calculate the perspective transformation matrix “M”. Then the statement perspective transformation matrix “M” calls the “cv2.warpPerspective” function to apply the perspective transformation and obtain the transformed image “perspective_transformation”, which is saved in the images / results / transformation results folder under the Mask-RCNN directory.

[0072] (3.2.4) Define the “apply_perspective_transform(boxes,M)” function to apply the same perspective transform to the corresponding Mask image (Snipaste_2023-12-08_09-29-07_combined_mask.png) and automatically get the “corrected_perspective_mask” result image.

[0073] (3.2.5) Use the “cv2.findContours” function to detect the contours of the white pixels in the “corrected_perspective_mask” image, and create an all-zero array “rect_img” of the same size as the input image and with all pixel values ​​set to zero; then calculate the minimum enclosing rectangle of each contour and draw it on rect_img, and finally output the corrected image “corrected_perspective_mask_rect_corrected.png”.

[0074] (4) Automatic intelligent calculation of the opening ratio of the building facade: Through the basic methods of computer vision and geometric calculation in Python, the area of ​​the white pixels in the "corrected_perspective_mask_rect_corrected.png" obtained in step 3.2.5 is automatically identified, and the opening ratio value of the building facade is obtained by geometric calculation.

[0075] (5) The initialization configuration of the present invention is now completed, including the training and recognition of the Mask-RCNN model and various settings for image processing. After the Mask-RCNN model training is completed, street view photos and video screenshots of blocks and lots are directly imported, and after modifying the relevant file paths, intelligent batch recognition and measurement can be completed without tedious work.

Claims

1. An intelligent batch calculation method for building facade opening rate based on Mask-RCNN instance segmentation model, characterized in that: The implementation steps of this method are as follows: Step (1), collecting street view photos and video data of the block where the building facade opening rate is to be measured, automatically batch-capturing the video according to the number of frames, and pre-processing it in Photoshop software; the pre-processing includes adjusting the image resolution and cropping unnecessary edge parts, and the pre-processed data is used as the basic data set for constructing the Mask-RCNN model training; Step (2), input the basic data set into the Mask-RCNN model for training. During the training process, use the Adam or SGD optimization algorithm, update the weights through the back propagation algorithm, and finally obtain the weights that perform best on the validation set as the optimal weights; Step (3), based on the trained Mask-RCNN model, fast batch recognition is performed on street view photos and video screenshots, and instance segmentation and annotation of opening elements on building facades are performed; Step (4), using OpenCV technology to select a street view photo or video screenshot of the building facade whose opening rate needs to be calculated, and the corresponding Mask image, i.e., Mask picture, and perform the same automatic perspective correction; Step (5), using OpenCV technology to automatically correct the white pixels of the Mask image, eliminate irregular contours, and improve the accuracy of the calculation results; Step (6), based on the perspective transformed image and the corresponding Mask image, automatically and intelligently calculate the ratio of white pixels in the Mask image to the total image area, and finally obtain the measurement result of the opening rate.

2. The intelligent batch calculation method for building facade opening rate based on Mask-RCNN instance segmentation model according to claim 1 is characterized in that: The steps for automatically batch-capturing videos by frame number in step (1) are as follows: (a) Based on the OpenCV-Python library, read the video file through the "cv2.VideoCapture" function; (b) Obtain information such as the frame rate FPS and total number of frames of the video file, and set a fixed interval; (c) traverse the video frames and check whether the current frame number is the predetermined screenshot frame number; if so, save the frame as a picture through the "cv2.imwrite" function; if not, skip the current frame number until traversing to the predetermined screenshot frame number.

3. The intelligent batch calculation method for building facade opening rate based on Mask-RCNN instance segmentation model according to claim 1 is characterized in that: The steps of training based on the Mask-RCNN model in the above steps (2) and (3) and quickly batch-recognizing street view photos and video screenshots are as follows: (a) Dataset preparation: Based on the customized building facade opening feature classification, the basic dataset is annotated by Labelme; (b) Mask-RCNN model training: First, modify the data configuration file, including data category, data path, and training rounds; second, modify the number of categories in the Mask-RCNN model configuration file; then input the training weights, data configuration file, model configuration file path, and training rounds into the Mask-RCNN model for training, and enable tensorbord to view the training parameters; The dataset is divided into a training set and a validation set. The Mask-RCNN model is trained on the training set and its performance is evaluated on the validation set. The indicators are monitored on the validation set. If there is no improvement after several consecutive epochs, the training is stopped in advance to prevent overfitting. At the end of each epoch, the current optimal weight, i.e., the weight with the best performance on the validation set, is recorded. After the training is completed, the optimal weight of the Mask-RCNN model is obtained. (c) Mask-RCNN model reasoning: The street view dataset to be identified and the optimal weight path obtained through training are input into the Mask-RCNN model for model reasoning, and the identification of the door and window opening elements of the building facade is finally completed, and the corresponding Mask image is exported at the same time.

4. The intelligent batch calculation method for building facade opening rate based on Mask-RCNN instance segmentation model according to claim 1 is characterized in that: In the above step (4), the steps of using OpenCV technology to automatically correct the perspective of street view photos or video screenshots and mask images are as follows: (a) Read the image by calling the "cv2.imread" function; (b) Select four corner points to construct the building facade area of ​​the block where the opening rate needs to be calculated: by calling the custom event processing function "click_event", record the coordinates (x, y) of the current click, repeat four times, and add the coordinates of the selected four points to a point coordinate list; (c) Perspective correction: First, the image to be corrected stored in a specific path is read through the "cv2.imread" function in OpenCV; second, the height and width of the corrected image are calculated based on the set four-point correction area, and the four corner points of the transformed image are set accordingly; The perspective transformation matrix is ​​calculated by the "cv2.getPerspectiveTransform" function in OpenCV; finally, the "cv2.warpPerspective" function of OpenCV is applied to automatically generate the corrected image based on the perspective transformation matrix and the size of the new image.

5. The intelligent batch calculation method for building facade opening rate based on Mask-RCNN instance segmentation model according to claim 1 is characterized in that: The steps for automatically correcting the white pixels of the Mask image in the above step (5) are as follows: (a) Read the Mask image through the "cv2.imread" function in OpenCV; (b) Use the "cv2.findContours" function to detect the contours of white pixels in the Mask image and create an all-zero array "rect_img" with the same size as the input image and all pixel values ​​are zero; (c) Calculate the minimum enclosing rectangle of each contour and draw it on rect_img, and finally automatically output the corrected image.

6. The intelligent batch calculation method for building facade opening rate based on Mask-RCNN instance segmentation model according to claim 1 is characterized in that: The steps for automatically and intelligently calculating the opening rate in the above step (6) are as follows: (a) Using computer vision technology in the Python programming language combined with geometric calculation methods, the total area of ​​white pixels in the corrected Mask image and its proportion to the entire image area are automatically identified and calculated to obtain the opening rate value of the building facade; First, the Mask image is read using OpenCV, and the total number of white pixels in the mask image and the total number of pixels in the image are calculated using the "np.sum()" function in NumPy. Finally, the number of white pixels is divided by the total number of pixels to obtain the proportion of white area; (b) The calculated opening ratio values ​​are subjected to intelligent cluster analysis to identify and divide the opening ratio value ranges of the building facades in the block.

Images