Automatic extraction method of ancient city battlements

By adding edge enhancement modules in the Mask-RCNN framework and using connectivity and overlap for noise removal and aggregation, the problems of low edge segmentation accuracy and difficulty in fragment extraction in the three-dimensional model are solved, and efficient automatic extraction of ancient city battlements is achieved.

CN114511582BActive Publication Date: 2024-11-26WUHAN UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111622565.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-28
Publication Date
2024-11-26
Estimated Expiration
2041-12-28

AI Technical Summary

Technical Problem

When the prior art automatically recognizes and extracts objects of interest from three-dimensional models of cultural relics buildings, it faces the problems of low edge segmentation accuracy and difficulty in extracting three-dimensional object fragments, especially when complex backgrounds and fewer pixels at the edge.

Method used

By adding edge enhancement modules in the Mask-RCNN framework, the image gradient is extracted as pixel weight using the 5×5sobel operator, the loss function of the mask prediction branch is improved, the two-dimensional image segmentation accuracy is improved, and the three-dimensional bounding box overlap is combined with connectivity and 3D bounding box overlap. , remove noise fragments and aggregate three-dimensional object fragments to realize automatic extraction of ancient city battlements.

Benefits of technology

It improves the accuracy of object edge segmentation, reduces manual intervention, saves manpower and material resources, improves the efficiency of battlements extraction, and realizes automated recognition and extraction from two-dimensional images to three-dimensional models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114511582B_ABST
    Figure CN114511582B_ABST
Patent Text Reader

Abstract

The present invention relates to an automatic extraction method for battlements of an ancient city. An improved MaskRCNN algorithm with edge enhancement is proposed to improve the segmentation accuracy of 2D objects, and the segmentation results are projected onto a three-dimensional model to obtain three-dimensional object fragments. Noise fragments generated by inaccurate segmentation results are eliminated according to connectivity, and three-dimensional object fragments from different perspectives are aggregated according to overlapping relationships in geometric positions to establish a complete three-dimensional object. Based on the MaskRCNN framework, the present invention adds an edge enhancement module to improve the segmentation accuracy of two-dimensional images; no manual intervention is required for the recognition and extraction of battlements, which saves manpower and material resources and improves the efficiency of battlement extraction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a method for automatically extracting battlements of an ancient city. Background Art

[0002] UAV oblique photogrammetry technology has many advantages such as efficient data collection, low cost, and high data accuracy. It is often used in the digitization of large cultural relics. However, the three-dimensional models produced by oblique photogrammetry do not have semantic information, and adjacent objects in space often stick together, which brings difficulties to further data use and expert analysis. How to automatically identify and extract objects of interest from the three-dimensional models of cultural relics is of great significance for the recording, protection and restoration of cultural relics.

[0003] At present, 2D target detection and instance segmentation technology have made some progress. Compared with traditional machine learning methods that manually design features based on experience, deep learning uses large neural networks to automatically extract features, which improves the accuracy of recognition. Limited by the amount of data and the complexity of the scene, it is difficult to directly identify and extract on the 3D model, and the actual application effect is not ideal. Therefore, it is considered to use the instance segmentation results of the 2D image to guide the automatic extraction of the target object from the 3D model. This method is less difficult and more operational than directly completing the target extraction from the 3D model, but there are still two major technical problems: 1) How to improve the accuracy of object edge segmentation. The edge of the object is usually surrounded by a more complex and chaotic background, which is very easy to misclassify; in addition, the number of edge pixels is far less than the number of internal pixels, and the model is easily dominated by internal pixels. 2) How to extract a complete object from 3D object fragments. A single image can only observe a part of the object. If the segmentation result is directly projected onto the model, it will form 3D object fragments. It is necessary to aggregate the multi-view segmentation results to obtain a complete 3D target object. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an automatic extraction method for ancient city battlements. By accurately segmenting the edges of objects in an image, using connectivity and overlap to remove and aggregate fragments, the ancient city battlements can be automatically extracted from a three-dimensional model.

[0005] In order to achieve the above object, the technical solution provided by the present invention is an automatic extraction method of ancient city battlements, comprising the following steps:

[0006] Step 1: Generate a 3D model of the scene using oblique photogrammetry technology, use data annotation tools to create training and test sets for 2D image battlement extraction, and use data augmentation methods to enhance the diversity of the training data set;

[0007] Step 2: Use the training set images to train the improved Mask-RCNN instance segmentation model with edge enhancement module;

[0008] Step 3, use the trained instance segmentation model to perform instance segmentation on the images in the test set to obtain the mask of the battlements on each test set image;

[0009] Step 4, 3D object extraction, includes the following steps:

[0010] Step 4.1, projecting the mask result obtained by segmentation in step 3 onto the three-dimensional model by using the ray intersection method;

[0011] Step 4.2, remove the noise fragments generated by inaccurate two-dimensional segmentation results according to connectivity;

[0012] In step 4.3, the 3D model fragments generated by mask projection are aggregated using the fragment overlap and the 3D bounding box overlap of the fragments, and finally the automatic extraction of the battlement object is completed.

[0013] Moreover, in order to improve the segmentation accuracy of the object edge in step 2, based on the classic MaskRCNN framework, an edge enhancement extension module is added to the loss function Loss of the mask prediction branch. mask An improvement is made by using a 5×5 sobel operator to perform a convolution operation on the label image during training, extracting the image gradient and obtaining the pixel weight. Then, a weighted binary cross entropy loss function is used in the mask prediction branch to give edge pixels a larger weight, so as to strengthen the learning of edge pixels during training.

[0014] The weighted binary cross entropy loss function is calculated as follows:

[0015]

[0016] in:

[0017]

[0018] In the formula, Loss mask (y,p) represents the loss function, N represents the total number of pixels, y i represents the label value of the i-th pixel, p i is the probability predicted by the network classification branch, W i is the pixel weight obtained by convolution operation, is a 5×5 sobel operator, I i represents the label image and ☉ is the convolution symbol.

[0019] Gradients contain rich edge information, indicating the feature complexity of pixels at different positions. The edge enhancement extension module uses gradient values ​​as pixel weights, which can strengthen various areas of the object to varying degrees, thereby improving the accuracy of object segmentation.

[0020] Moreover, due to the inaccuracy of the two-dimensional segmentation results in step 4.2, noise fragments exist around the real object. These noise fragments are separated from the object and have fewer triangles than the object. The fragments under the same perspective are grouped together, and all fragments are divided according to connectivity, and the fragments with the most triangles in each group are retained as the target object, thereby removing the noise fragments.

[0021] Moreover, the overlap between two fragments is defined in step 4.3.1 as the ratio of the number of common triangles in the two fragments to the total number of triangles. When the overlap is greater than T1, the two fragments are regarded as the same object and merged. The merge operation is performed on all fragments on the same side of the three-dimensional object, and the process is iterated until no fragments are merged.

[0022] Moreover, the degree of overlap of the three-dimensional bounding boxes of two fragments is defined in step 4.3.2 as the ratio of the intersection volume of the two bounding boxes to the sum of their volumes. For the fragments on both sides of the battlement wall, the common triangle faces exist on the top of the battlement. The three-dimensional bounding boxes of the fragments are generated according to the three-dimensional coordinates of the fragments. The fragments whose three-dimensional bounding box overlap is greater than T2 are aggregated into one object, and the fragment merging operation is iteratively performed until no fragments are merged.

[0023] Compared with the prior art, the present invention has the following advantages:

[0024] 1) Based on the MaskRCNN framework, an edge enhancement module is added to improve the accuracy of 2D image segmentation; 2) No human intervention is required for the recognition and extraction of battlements, which saves manpower and material resources and improves the efficiency of battlement extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The technical flow chart of an embodiment of the present invention.

[0026] Figure 2 Schematic diagram of the Mask-RCNN network structure according to an embodiment of the present invention.

[0027] Figure 3 Schematic diagram of the improved Mask-RCNN workflow according to an embodiment of the present invention, wherein Figure 3 (a) is the workflow of Mask-RCNN. Figure 3 (b) is the added edge enhancement module.

[0028] Figure 4 Schematic diagram of edge feature enhancement according to an embodiment of the present invention, wherein Figure 4 (a) is the label image, Figure 4 (b) is the pixel weight map.

[0029] Figure 5A schematic diagram of a noise removal process according to an embodiment of the present invention.

[0030] Figure 6 Schematic diagram of the fragment aggregation process according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The present invention provides an automatic extraction method for the battlements of an ancient city. By proposing an improved MaskRCNN algorithm with edge enhancement, the segmentation accuracy of 2D objects is improved, and the segmentation results are projected onto a three-dimensional model to obtain three-dimensional object fragments. Noise fragments added due to inaccurate segmentation results are eliminated according to connectivity, and three-dimensional object fragments from different perspectives are aggregated according to overlapping relationships in geometric positions to establish a complete three-dimensional object.

[0032] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0033] like Figure 1 As shown, the process of the embodiment of the present invention includes the following steps:

[0034] Step 1: Use oblique photogrammetry technology to generate a three-dimensional model of the scene, use data annotation tools to create training sets and test sets for two-dimensional image battlement extraction, and use data augmentation methods to enhance the diversity of the training data set.

[0035] Step 2: Use the training set images to train the improved Mask-RCNN instance segmentation model with edge enhancement module.

[0036] Mask-RCNN is a typical convolutional neural network, and its workflow is as follows Figure 2 As shown. First, the image to be detected is passed through the feature extraction network for feature extraction; then the candidate region is generated by the Region Proposal Network (RPN). Since the generated candidate regions have a high degree of overlap, the Non-Maximum Suppression (NMS) algorithm is used to suppress them; then the remaining candidate regions after the suppression process are pooled by ROIAlign; finally, the candidate regions are classified by the Softmax classifier, the coordinates of the detection border are obtained by the multi-task loss border regression algorithm, and the mask is generated by the Fully Convolutional Networks (FCN).

[0037] In order to improve the segmentation accuracy of object edges, this paper adds an edge enhancement extension module based on the classic MaskRCNN framework to improve the loss function Loss of the mask prediction branch. maskImprovements are made to give edge pixels greater weights to strengthen the learning of edge pixels during training. The improved Mask-RCNN workflow is shown in the figure below: Figure 3 shown.

[0038] During the training process, a 5×5 sobel operator is used to perform a convolution operation on the label image to extract the image gradient and obtain the pixel weights. Then, a weighted binary cross-entropy (BCE) loss function is used in the mask prediction branch.

[0039] The weighted binary cross entropy loss function is calculated as follows:

[0040]

[0041] in:

[0042]

[0043] In the formula, Loss mask (y,p) represents the loss function, N represents the total number of pixels, y i represents the label value of the i-th pixel, p i is the probability predicted by the network classification branch, W i is the pixel weight obtained by convolution operation, is a 5×5 sobel operator, I i represents the label image and ☉ is the convolution symbol.

[0044] Gradients contain rich edge information, indicating the complexity of the features of pixels at different locations. The edge enhancement extension module uses gradient values ​​as pixel weights to enhance the various regions of an object to varying degrees, thereby improving the accuracy of object segmentation. Figure 4 As shown in (b), the value of the weight map W starts to decrease from the edge of the object. The weight of edge pixels is greater than that of internal area pixels. At the same time, the weight of edge pixels located on sharp edges is larger than that of flat edge pixels.

[0045] Step 3: Use the trained instance segmentation model to perform instance segmentation on the images in the test set to obtain the mask of the battlements on each image.

[0046] Step 4, 3D object extraction, includes the following steps:

[0047] Step 4.1, projecting the mask result obtained by segmentation in step 3 onto the three-dimensional model through the ray intersection method.

[0048] Step 4.2, remove the noise fragments generated by inaccurate two-dimensional segmentation results based on connectivity.

[0049] A single image can only record part of the object's information, and direct projection will produce three-dimensional object fragments from different perspectives. And due to the inaccuracy of the two-dimensional segmentation results, there may be some noise around the real object. These noise fragments are usually separated from the object and have fewer triangles than the object. Therefore, the fragments under the same perspective are grouped together, and all the fragments are divided according to connectivity. The fragments with the most triangles in each group are retained as the target object, thereby removing the noise fragments and obtaining the following: Figure 5 Results shown.

[0050] In step 4.3, the 3D model fragments generated by mask projection are aggregated using the fragment overlap and the 3D bounding box overlap of the fragments, and finally the automatic extraction of the battlement object is completed.

[0051] Step 4.3.1, aggregate the fragments of the three-dimensional object on the same side using the fragment overlap.

[0052] Define the overlap of two fragments as the ratio of the number of common triangles in the two fragments to the total number of triangles. When the overlap is greater than 0.5, the two fragments are considered to be the same object and merged. Traverse all fragments on the same side of the 3D object to perform the merge operation, and iterate the process until no fragments are merged.

[0053] Step 4.3.2, aggregate the fragments of the 3D objects on the opposite sides using the overlap of the 3D bounding boxes of the fragments.

[0054] The overlap of the three-dimensional bounding boxes of two fragments is defined as the ratio of the intersection volume of the two bounding boxes to the sum of their volumes. For the fragments on both sides of the battlement wall, the common triangles exist on the top of the battlement. The three-dimensional bounding boxes of the fragments are generated according to the three-dimensional coordinates of the fragments. The fragments with a three-dimensional bounding box overlap greater than 0.5 are aggregated into one object, and the fragment merging operation is iteratively performed until no fragments are merged.

[0055] During specific implementation, the above process can be automatically operated using computer software technology.

[0056] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A method for automatically extracting battlements of an ancient city, characterized in that: The following steps are involved: Step 1: Generate a 3D model of the scene using oblique photogrammetry technology, use data annotation tools to create training and test sets for 2D image battlement extraction, and use data augmentation methods to enhance the diversity of the training data set; Step 2: Use the training set images to train the improved Mask-RCNN instance segmentation model with edge enhancement module; Step 3, use the trained instance segmentation model to perform instance segmentation on the images in the test set to obtain the mask of the battlements on each test set image; Step 4, 3D object extraction, includes the following steps: Step 4.1, projecting the mask result obtained by segmentation in step 3 onto the three-dimensional model by using the ray intersection method; Step 4.2, remove the noise fragments generated by inaccurate two-dimensional segmentation results according to connectivity; The fragments under the same view angle are grouped together, and all the fragments are divided according to connectivity, and the fragments with the most triangles in each group are retained as the target object, thereby removing the noise fragments; Step 4.3, using the overlap of fragments and the overlap of the three-dimensional bounding boxes of the fragments, the three-dimensional model fragments generated by the mask projection are aggregated, and finally the battlement object is automatically extracted; Step 4.3.1, using the fragment overlap to aggregate the fragments of the three-dimensional object on the same side; The overlap of two fragments is defined as the ratio of the number of common triangles in the two fragments to the total number of triangles. When the overlap is greater than T1, the two fragments are considered to be the same object and merged. All fragments on the same side of the three-dimensional object are traversed to perform the merge operation. The process is iterated until no fragments are merged. T1 is the set threshold. Step 4.3.2, using the overlap of the 3D bounding boxes of the fragments to aggregate the fragments of the 3D objects on the opposite sides; The 3D bounding box overlap of two fragments is defined as the ratio of the intersection volume of the two bounding boxes to the sum of their volumes. For the fragments on both sides of the battlement wall, the common triangles exist on the top of the battlement. The 3D bounding boxes of the fragments are generated according to the 3D coordinates of the fragments. The fragments whose 3D bounding box overlap is greater than T2 are aggregated into one object. The fragment merging operation is iteratively performed until no fragments are merged. T2 is the set threshold.

2. The method for automatically extracting battlements of an ancient city as claimed in claim 1, characterized in that: In step 2, Mask-RCNN is a typical convolutional neural network. First, the image to be detected is subjected to feature extraction through a feature extraction network, and then a candidate region is generated through a region proposal network. Since the generated candidate regions have a high degree of overlap, they are suppressed using a non-maximum suppression algorithm. Then, ROIAlign pooling is performed on the remaining candidate regions after the suppression process. Finally, the candidate regions are classified through a Softmax classifier, the coordinates of the detection border are obtained using a multi-task loss border regression algorithm, and a mask is generated using a fully convolutional segmentation network.

3. The automatic extraction method of ancient city battlements according to claim 2, characterized in that: In step 2, in order to improve the segmentation accuracy of the object edge, based on the classic MaskRCNN framework, an edge enhancement extension module is added to the loss function Loss of the mask prediction branch. mask To make improvements, firstly, convolution operation is performed on the label image during training to extract the image gradient and obtain the pixel weights. Then, a weighted binary cross entropy loss function is used in the mask prediction branch to give edge pixels a larger weight to strengthen the learning of edge pixels during training. The weighted binary cross entropy loss function is calculated as follows: In the formula, Loss mask (y,p) represents the loss function, N represents the total number of pixels, W i is the pixel weight obtained by convolution operation, y i represents the label value of the i-th pixel, p i is the probability predicted by the network classification branch.

4. The automatic extraction method of ancient city battlements according to claim 3, characterized in that: The convolution operation in step 2 uses a 5×5sobel operator, and the convolution operation calculation formula is as follows: Where W i is the weight of the i-th pixel obtained by the convolution operation, is a 5×5 sobel operator, I i represents the label image, ☉ is the convolution symbol; Gradients contain rich edge information, indicating the feature complexity of pixels at different positions. The edge enhancement extension module uses gradient values ​​as pixel weights, which can strengthen various areas of the object to varying degrees, thereby improving the accuracy of object segmentation.