Digital pattern reference gallery construction method for multi-style fusion building restoration

Abstract

The invention discloses a digital pattern reference image library construction method for multi-style fusion building restoration. The method comprises the following steps: constructing a reference image library consisting of a plurality of single-style building component reference images; acquiring a plurality of images of the same building component, performing preprocessing to extract contour information, selecting one of the images with the most complete contour information, calculating the similarity between the selected image and the corresponding reference image in the reference image library, and matching the reference image with the similarity higher than a similarity threshold value; comparing the contour information of the to-be-repaired building component image with the reference image to obtain a defect position; fusing and reconstructing the plurality of to-be-repaired building component images with different defect positions to obtain a fused image; the pictures meeting the requirements are added into a reference picture library through manual approval; and if the defect exceeds the defect threshold, matching the reference images with the same age, the same region and the same main style to perform fusion reconstruction, performing manual approval again, and adding the fused image meeting the requirement into the reference image library.

Description

[0001] Divisional application This application is a divisional application of the Chinese invention patent application [Application Number: 2024108241777] [Invention Name: A method and device for repairing damaged buildings of multiple styles, and storage medium] filed on June 24, 2024. Technical Field

[0002] The present invention belongs to the technical field of building digital image restoration, and in particular relates to a method for constructing a digital pattern reference library for multi-style fusion building restoration. Background Art

[0003] Fusion architecture is a form of architecture that combines elements of many different styles. This architectural style usually combines multiple architectural design elements to create a unique and individual architectural form. Fusion architecture can integrate the characteristics of different cultures, eras and styles in design, showing diversity and innovation.

[0004] For example, Shanghai-style architecture is a typical multi-style fusion architecture, which usually refers to the Shikumen and Lane buildings with local characteristics that emerged in Shanghai in the first half of the 20th century. These buildings represent part of Shanghai's history and culture and have important historical, cultural and artistic values.

[0005] Many multi-style fusion buildings, including Shanghai-style buildings, are currently facing certain difficulties in repair and protection, such as demolition and reconstruction, lack of maintenance and restoration, planning conflicts, disaster damage and other issues. In order to better protect and inherit the cultural heritage of style-fusion buildings and protect the style and characteristics of style-fusion buildings, innovative use of database technology is used to digitally preserve the decorative features of Shanghai-style buildings to facilitate the subsequent restoration and repair process.

[0006] In the actual use of the architectural digital image database, it was found that due to the particularity of multi-style fusion buildings, the short period of popularity and the small area, the ones that can survive are even rarer. In addition, most of the surviving ancient buildings are poorly repaired and are rather dilapidated, making it difficult to find complete ones for reference. As a result, the existing restoration work needs to rely on the restorers' own understanding of the building and their own cultural heritage, resulting in low efficiency and lack of consistency in restoration quality.

[0007] Chinese patent application CN202010352962.9 discloses a multi-directional repair system and method for damaged ancient building columns. The multi-directional repair system for damaged ancient building columns includes an image acquisition module, an image analysis module, a central control module, a repair site determination module, a damage degree measurement module, a three-dimensional modeling module, a repair simulation module, a repair plan generation module, a feasibility analysis module, and a display module; the multi-directional repair method for damaged ancient building columns includes collecting and analyzing images of ancient building columns; determining the repair site and its area and depth; establishing a three-dimensional model of the column to simulate the repair operation; generating a repair plan and analyzing the feasibility.

[0008] In the above-mentioned prior art, a method of repairing by collecting architectural components of the ancient building to be repaired and determining the repair site and the degree of damage is mentioned. However, for the existing multi-style fusion buildings, which are extremely rare, it is difficult to find reference objects or / and reference images for repair. The repair can only be carried out based on the restorer's own understanding of the building. The final repair result is related to the restoration personnel's literacy and lacks objectivity.

[0009] Most of the existing ancient buildings that integrate multiple styles are severely damaged, and it is almost impossible to have complete building components preserved, making them impossible to be directly used as a reference for restoration. Summary of the invention

[0010] The purpose of the present invention is to provide a method and device for repairing damaged buildings in multiple styles, and a computer-readable storage medium, which partially solve or alleviate the above-mentioned deficiencies in the prior art. The digital image of the damaged building in multiple styles to be repaired is repaired by taking the complete image of the building components in the building in multiple styles or the baseline image of the building components in a single style as a reference, thereby providing a reference for building repair, thereby improving the efficiency and quality of repair.

[0011] In order to solve the above-mentioned technical problems, the present invention specifically adopts the following technical solutions: A method for repairing damaged buildings of multiple styles, comprising: Decomposing the building image to be restored, and obtaining a separate building component image of each building component in the building image to be restored; Preprocessing each building component image to extract contour information of the building component in the building component image; Comparing the contour information of the building component with the reference image of the building component in a pre-built reference image library to obtain the defect position and degree of the defect of the building component image; Matching the complete image of the building component of the multi-style fusion building in the pre-built reference library or the reference image of the building component in the reference image library according to the defect position and degree of the defect of the building component image; Referring to the complete image of the building component of the multi-style fusion building in the matched reference library or the reference image of the building component in the reference image library, the image of the building component is repaired to obtain a repaired image of the building component; The architectural image to be restored is restored using the architectural component restoration image to obtain a restored architectural image.

[0012] As an improvement, the step of matching the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library according to the defect position and degree of the defect of the building component image includes: The defect state of the building component image is obtained by using the defect position and defect degree of the building component image; Determining whether the defect state is within a preset first threshold interval, a second threshold interval, or a third threshold interval; When the defect state is within the first threshold interval, the style of the building component image is judged to obtain the main style of the building component image, and the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library is matched according to the style of the building component image; When the defect state is within the second threshold interval, the complete image of the building components of the multi-style fusion building in the reference library is matched; When the defect state is within the third threshold range, a reference image of the building component in the reference image library is matched.

[0013] As an improvement, the steps of obtaining the defective state of the building component image include: Defect state = a*defect location score + b*defect degree score, where a and b are defect coefficients.

[0014] As an improvement, the manual review degree is calculated according to the defect state of each building component image in the building image to be restored. If the manual review degree exceeds the manual review threshold, the restored building image is submitted to manual review. In the case where the manual review is not passed, a reference of the image of the building component whose defect state is within the third threshold interval is replaced for secondary repair to obtain a secondary repaired image of the building component; the repaired image of the building component is replaced with the secondary repaired image of the building component, and then a secondary restoration is performed to obtain a secondary restored building image; and the secondary restored building image is subjected to a secondary manual review; If the second manual review fails, the reference of the building component image in the defective state within the second threshold interval is replaced for secondary repair; the repaired image of the building component is replaced with the second repaired image of the building component, and then the building image to be repaired is restored three times to obtain the three restored building images; the three restored building images are manually reviewed three times; In the case of failing three manual reviews, the reference of the building component image in the defective state within the second threshold interval and the third threshold interval is replaced simultaneously for secondary restoration; the repaired image of the building component is replaced with the secondary repaired image of the building component, and then the building image to be repaired is restored four times to obtain the four restored building images; the four restored building images are subjected to four manual reviews.

[0015] As an improvement, the steps to replace the reference include: In the case where the original reference is a complete image of a building component of a multi-style fusion building in a reference library, a reference image of the building component in a reference image library is used as a reference; In the case where the original reference is the benchmark image of the building component in the standard image library, the complete image of the building component of the multi-style fusion building in the reference library is used for reference.

[0016] As an improvement, if the image fails to pass four manual reviews, architectural images of the same era, region and main style are obtained as references for restoration based on the main style of the architectural image to be restored.

[0017] The step of calculating the manual appraisal degree according to the defect state of each building component image in the building image to be repaired includes: The building components are divided into several categories according to their influence on the overall style of the building, and the manual review coefficient is set for each category of building component images; Calculate the average number of defective states of each type of building component image, and adjust the average number by the manual review coefficient of the building component image of that type to obtain the manual review degree of the single type of building component image; The sum of the manual evaluation degrees of all single-category building component images is taken as the manual evaluation degree of the building image to be restored.

[0018] The present invention also provides a repair device for damaged buildings of multiple styles, comprising: A decomposition module, used for decomposing the building image to be restored, and obtaining a separate building component image of each building component in the building image to be restored; A preprocessing module preprocesses each building component image and extracts contour information of the building component in the building component image; A defect acquisition module is used to compare the contour information of the building component with the reference image of the building component in the reference image library to obtain the defect position and degree of the building component image; A matching module is used to match the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library according to the defect position and degree of the defect of the building component image; A building component image restoration module is used to repair the building component image by referring to the complete image of the building component of the multi-style fusion building in the matched reference library or the reference image of the building component in the reference image library to obtain a restored image of the building component; The building image restoration module uses the building component restoration image to restore the building image to be restored to obtain the restored building image.

[0019] As an improvement, the matching module specifically includes: A defect state acquisition unit, used to acquire the defect state of the building component image by using the defect position and defect degree of the building component image; A judging unit, used for judging whether the defect state of the building component image is within a first threshold interval, a second threshold interval, or a third threshold area; a first reference image acquisition unit, configured to, when the judgment unit judges that the defect state of the building component image is within a first threshold interval, judge the main style of the building component image to acquire the style of the building component image, and match the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library according to the style of the building component image; A second reference image acquisition unit is configured to match a complete image of a building component of a multi-style fusion building in a reference image library when the judgment unit judges that the defect state of the building component image is within a second threshold interval; The third reference image acquisition unit is used to match a reference image of a main style of the building component in a reference image library when the judgment unit judges that the defect state of the building component image is within a third threshold range.

[0020] The present invention also provides a storage medium, in which a computer program is stored; when the computer program is executed, the above-mentioned method for restoring digital images of multi-style fusion buildings can be implemented.

[0021] The present invention also provides a computer system, comprising a memory and a processor; the memory stores a computer program; when the computer program is executed by the processor, the above-mentioned method for restoring digital images of multi-style fusion buildings can be implemented.

[0022] The present invention is beneficial in that: The present invention constructs in advance a complete image of building components containing multi-style fusion, and a single-style reference image library including the main style and the foreign style. After decomposing the digital image of the building to be repaired, each building component image is compared with the main style reference image to obtain its defect situation. Then, according to the defect situation, a suitable image (complete image or reference image) is matched as a reference (image) for repair.

[0023] After repairing all the defective images of building components, the images of building components are stitched together to form a restored building image. And according to the original degree of defect of the building image, it is judged whether it needs to be manually reviewed. Since each building contains hundreds of building components, and the repaired images of some building components may be repaired based on the complete images of multiple styles, while the repaired images of other building components may be repaired based on the reference images of a single style, in order to avoid the possibility of abrupt or unreasonable styles in the final stitched building image, it is necessary for experts to conduct manual review. For example, if the building itself has a small degree of defect, after the defective part is repaired, even if the style of the defective part in the final stitched building image is inconsistent with the original style of the building, the impact is not great, so no manual review is required. However, if the building itself has a large degree of defect, if the style of the defective part in the architectural image stitched after automatic repair is inconsistent with the architectural style, it will seriously affect the quality of the building restoration, so experts are required to intervene for manual review.

[0024] For architectural images that do not meet the requirements after manual review, secondary, tertiary, quaternary or even quintuple restorations are performed by adjusting the restoration strategy of the building components, making full use of the existing reference resources to make the restored architectural images as consistent with the original style as possible, thereby improving the efficiency and quality of the restoration.

[0025] The repair of building components in the present invention is performed with reference to the complete image or the reference image of the building component, eliminating the dependence of manual repair on the quality of the repairer, thereby greatly improving the consistency of the repair quality.

[0026] When automatic repair fails to meet the requirements, manual intervention for review can ensure the quality and efficiency of the repair and reduce the labor intensity of manual work. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, each element or part is not necessarily drawn according to the actual scale. Obviously, the drawings described below are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without paying creative labor.

[0028] Figure 1 This is a flow chart of Embodiment 1 of the present invention; Figure 2 This is a flow chart of Embodiment 2 of the present invention; Figure 3AThis is a functional module diagram of Embodiment 3 of the present invention; Figure 3B This is a functional module diagram of Embodiment 4 of the present invention; Figure 4 and Figure 5 Images of damaged buildings of various styles collected on-site. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0030] Herein, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings by themselves. Therefore, "module", "component" or "unit" can be used mixedly.

[0031] In this document, the terms "upper", "lower", "inner", "outer", "front", "back", "one end", "the other end" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0032] In this document, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection, a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0033] Herein "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] Herein, "plurality" means two or more than two, ie, it includes two, three, four, five, etc.

[0035] In this article, "multi-style fusion architecture" refers to buildings that integrate multiple architectural styles. Generally speaking, multi-style fusion architecture is formed by an existing main style and some foreign styles. In a multi-style building, it is possible that some building components are the main style, while other building components are foreign styles. It is also possible that a certain building component is the main style combined with some foreign styles. And the fusion is relatively random, and the location and degree of fusion are based on the personal preferences of the craftsmen, and there is no universal regularity. Accordingly, once any of the building components is damaged, the building is also damaged, which is also called a damaged multi-style building.

[0036] In this article, "reference image" refers to an image of a single style and complete standard building component. For example, an image of only a main style (e.g., a style of local architectural characteristics) and complete standard building components, or an image of only an exotic style (e.g., a style of foreign or foreign architecture) and complete standard building components. Usually, for building components of a single style, there is only one reference image of each style. Specifically, it can be obtained from various architecture-related teaching materials or existing materials.

[0037] In this article, "complete image" refers to a complete image obtained by stitching (or fusion reconstruction) multiple images of the same building component with different defect locations or the same building component. The building component is a fusion of multiple styles, for example, a main style is fused with at least one foreign style. The main style refers to the style that most of the structures or outlines of the building component that fused multiple styles adopt, while a small part of the structure or outline adopts another style.

[0038] Example 1: Figure 1 As shown, the present invention provides a method for constructing a digital pattern reference library for multi-style fusion architectural restoration, and the specific steps include: S11 builds a benchmark image library consisting of several benchmark images of building components of a single style.

[0039] The building components described in the present invention include basic elements of buildings such as doors, beams, walls, and columns. Single-style buildings, such as Hui-style buildings, have distinct characteristics and have been popular for a long time, so there are many existing buildings and images. Accordingly, a corresponding benchmark image library can be established according to each style.

[0040] In this implementation, the purpose of constructing a reference image library is to serve as a reference for the basic structure of the later restoration of architectural component drawings with multiple styles, which will be introduced in detail later and will not be repeated here.

[0041] Of course, in order to be closer to the image of the damaged building to be repaired and integrated with multiple styles (or called multi-style fusion building), a single-style building similar to the multi-style fusion building can be selected. For example, the main style of some multi-style fusion buildings is Huizhou architecture, and some other elements are integrated. Therefore, in this step, the architectural component baseline image of Huizhou architecture can be selected.

[0042] In addition, for the convenience of subsequent use, the reference image in this embodiment includes a 2D reference image and a 3D point cloud reference image; the 2D reference image of the reference image includes 2D reference images at several angles. Since there are many existing physical objects and images of single-style buildings, it is also convenient to collect 3D point clouds and multi-angle 2D images.

[0043] S12 obtains a plurality of images of the same building component to be repaired in the same multi-style fusion building; pre-processes the images of the building component to be repaired, and extracts contour information of the building component in the images of the building component to be repaired.

[0044] There are several ways to obtain architectural component drawings of multi-style fusion buildings to be restored: First, the building itself is preserved, but a certain architectural component is missing. In this case, on-site collection can be carried out. Figure 4 and Figure 5 When the images of the building components to be restored in the multi-style fusion building are obtained by on-site acquisition, 2D images and 3D point cloud images of the building components at several angles are collected.

[0045] The second is that the building has been completely destroyed and disappeared, with only some pictures or videos remaining. In this case, it is necessary to decompose the various building components as much as possible based on the remaining information to provide assistance for subsequent integration and reconstruction.

[0046] In this embodiment, a building component, such as a column, is used as an example. In this step, multiple images of a certain multi-style fusion building column can be obtained from the above channels (of course, the building components in these images are all defective). Of course, the more the number, the more information, and the greater the probability of successfully fusion and reconstruction into a complete image.

[0047] After obtaining the images of the building components to be repaired in the multi-style fusion buildings, these images may contain various background information and other information unrelated to the building components. Therefore, it is necessary to preprocess these images of the building components to be repaired, mainly to extract their contour information to obtain the building component images of the damaged building components themselves for subsequent comparison.

[0048] In addition, the age and region of the building components in the image of the building component to be restored can also be manually annotated.

[0049] S13 calculates the similarity between the image of the building component to be repaired and the reference image corresponding to the building component in the reference image library, and matches the reference image whose similarity is higher than the similarity threshold.

[0050] Since the image of the building component to be repaired does not have a complete object or image for reference, it is difficult to know the specific location and extent of its damage. The benchmark image library actually provides a rough reference. For example, as a column, even if the style is different, the general structure is still similar, only some decorative patterns, sculptures, etc. are different. Therefore, compared with the benchmark image, the approximate defect situation can still be obtained. Preferably, a similarity comparison is performed based on the above-mentioned contour information. If there are multiple images with a similarity higher than the preset similarity threshold, the one with the highest similarity is used as the benchmark image for comparison. Preferably, the one with the most complete contour information is selected from several images of the building component to be repaired to calculate the similarity with each building component benchmark image.

[0051] In this implementation, since the reference image library includes reference images of various single-style building components, for example, reference images of columns in the Hui style, reference images of other single-style columns that are quite different from the Hui style, and reference images of columns in other styles similar to the Hui style, the main style of the multi-style building component image to be repaired can be manually annotated in advance, and the reference image that matches the main style of the building component image to be repaired can be matched in the reference images with a similarity higher than the similarity threshold, that is, the reference image of the main style of the building component. Of course, in some embodiments, AI can also be selected for annotation, but its accuracy is not as good as the manual standard.

[0052] For example, the building component to be repaired is a fusion of the Hui style and other architectural elements, so during preprocessing, its main style can be manually annotated. Then, in this implementation, under the premise that the similarity meets the requirements, a reference image with the same main style can be selected, so that the reference image is closer to the appearance of the building component to be repaired.

[0053] In order to make the similarity calculation more accurate and thus find a reference image that is closer to the image of the building component to be repaired, in this implementation, the reference image should collect 3D point cloud images and multi-angle 2D images as much as possible.

[0054] In addition, calculating the similarity between images is a prior art and will not be described in detail in the present invention. For example: When calculating the similarity between 2D images, commonly used methods include: Perceptual Hashing: Maps images to hash values ​​of fixed length and evaluates the similarity of two images by comparing their hash values.

[0055] Structural Similarity Index (SSIM): It measures the similarity between two images by comparing the brightness, contrast, and structural information of the images. It is usually used to evaluate the quality of images.

[0056] Histogram Similarity: Compares the color histograms of images and evaluates the similarity of images by calculating the distance between histograms.

[0057] For similarity calculation between 3D images, commonly used methods include: 3D shape matching: Extract features from 3D shapes and then compare the similarity between two 3D shapes through feature matching.

[0058] 3D mesh comparison: 3D mesh models are registered or aligned, and then the similarity between 3D models is evaluated by comparing the differences between mesh coordinates or vertices.

[0059] 3D point cloud registration: Two 3D point clouds are registered and then their similarity is evaluated by measuring the difference between the point cloud transformations.

[0060] S14 compares the contour information of the image of the building component to be repaired with the matched reference image of the building component to obtain the defect position of the building component to be repaired.

[0061] After selecting a suitable reference image, the comparison can be performed.

[0062] When the image of the building component to be repaired is a 3D point cloud, the defect of the building component to be repaired can be obtained by directly subtracting the 3D point cloud image of the building component to be repaired from the 3D point cloud reference image.

[0063] When the image of the building component to be repaired is a 2D image, the image of the building component to be repaired and the reference image are first segmented, and the segmented building components are compared, and the missing part is the defect of the building component.

[0064] S15 fuses and reconstructs a plurality of images of the building components to be repaired having different defect positions to form a fused image of the building components.

[0065] The difficulty in the prior art is actually that there is no complete building component image for reference, and the problem to be solved by the present invention is actually to combine existing defective images of a certain building component in an attempt to construct a complete image of the building component (that is, a fused image).

[0066] Since the defect positions of each image have been found in the previous step, theoretically, a complete image of the building component can be obtained by stitching together multiple images with different defect positions. For example, the base of the column is missing in image A, and part of the relief of the column is missing in image B, so a complete image of the column can be obtained by stitching the two together. Image A and image B can be images of the same building component on different buildings, or images of the same building component on the same building. They can also be images of the same building component with different defect positions and / or defect degrees collected at different times.

[0067] Of course, since there are multiple images, it is possible that the defect position of two images is the same. Therefore, when fusing and reconstructing, the image with the smaller degree of defect is selected, that is, the degree of defect of the image of the building component to be repaired is obtained. When fusing and reconstructing multiple images of the building component to be repaired with different defect positions, the image of the building component to be repaired with a degree of defect less than the degree threshold is selected.

[0068] S16 manually reviews the fused images of building components and adds the fused images that meet the requirements into the reference library.

[0069] In the previous step, a fused image was obtained by fusing and reconstructing multiple images of the building components to be restored. Although the images of the same building components of the same multi-style fusion building are used, the multi-style fusion building elements are relatively complex. Even in the same style, there will be differences between different buildings. Therefore, in this step, the fused image needs to be reviewed to determine whether it conforms to the style.

[0070] Identifying a style is actually a subjective judgment that is difficult to be completed by AI. Therefore, in this step, it is recommended to use manual review to determine whether the image meets the requirements based on the expert's deep understanding of the architectural style.

[0071] Any images that meet the requirements can be added to the database and used as a reference for subsequent restoration of the electronic image of the multi-style fusion building or the building itself.

[0072] When the defect of the fused image exceeds the defect threshold, S17 matches the reference image of the same era, region and main style for fusion reconstruction, and submits the fused and reconstructed fused image for manual review, and adds the fused image that meets the requirements into the reference library.

[0073] It is possible that even if multiple images to be restored are fused, they cannot be pieced together into a complete image of the building component, so further fusion and reconstruction are required.

[0074] In step S12, the age and region of the building components to be repaired are marked. When the defect of the fused image exceeds the defect threshold, for example, 5%, a reference image of the same age, region and main style is found from the reference image library for fusion and reconstruction. Of course, the fused image reconstructed by this method is more likely to be unsatisfactory, so it also needs to be manually reviewed and added to the reference library.

[0075] Embodiment 2: The so-called multi-style fusion building refers to a building that integrates multiple architectural styles. Generally speaking, a multi-style fusion building is formed by an existing main style and some foreign styles. In a multi-style building, it is possible that some building components are of the main style, while other building components are of foreign styles, that is, building components of a single style. It is also possible that a certain building component is a main style that integrates some foreign styles to form a multi-style building component. Since the randomness of multi-style fusion is relatively large, the location and degree of fusion are based on the personal preferences of the craftsmen, and there is no universal regularity, which makes it difficult to repair a multi-style fusion building after it is damaged.

[0076] In response to the above technical issues, such as Figure 2 As shown, the present invention provides a method for repairing damaged buildings of multiple styles (i.e., damaged buildings with multiple styles integrated), which repairs the digital images of damaged buildings of multiple styles based on the digital pattern reference library constructed in Example 1, and also uses the reference library constructed by the method provided in Example 1. The specific steps of the present invention include: S1 decomposes the building image to be restored to obtain a separate building component image of each building component in the building image to be restored.

[0077] In some embodiments, after obtaining an image of a multi-style fusion building to be restored, it needs to be decomposed into images of building components.

[0078] There are several ways to obtain images of multi-style fusion buildings to be restored: First, the building itself is preserved, but some of its building components are missing. In this case, on-site collection can be performed. When obtaining images of building components of multi-style fusion buildings to be restored through on-site collection, 2D images of building components and 3D point cloud images from several angles are collected.

[0079] Second, the building has been completely destroyed and disappeared, leaving only some pictures or images. In this case, it is necessary to decompose each building component as much as possible based on the remaining data. The decomposition method can use an artificial intelligence model to perform instance segmentation, decomposing each building component to form a separate building component image. Instance segmentation is a prior art and will not be described in detail in the present invention.

[0080] In addition, the era and region of the multi-style fusion buildings in the building images to be restored can also be manually marked.

[0081] S2 preprocesses each building component image to extract contour information of the building component in the building component image.

[0082] Similar to Example 1, the purpose of acquiring the contour information of the building component in the building component image is to obtain the defect status of the building component. The specific method is shown in Example 1 and will not be described in detail in this example.

[0083] S3 compares the outline information of the building component with the main style reference image of the building component in the reference image library to obtain the defect position and degree of the building component image.

[0084] It is understandable that the style of the reference image of the building component is consistent with the main style of the multi-style fusion building to be repaired, so that the defect of the building component image can be restored as accurately as possible. Therefore, when the main style of the building is known, a reference image with the same main style as the building (that is, a reference image of the main style of the building component) can be matched from the reference image library, and then compared to determine the location and degree of the defect. Since there is only one reference image of each style of building component in the reference library, once the name of the damaged building component and the main style of the corresponding building are known (for example, manually marked or entered), the system can match it with a reference image with the same main style of its building in the reference library.

[0085] When the image of the building component is a 3D point cloud, the defect of the building component to be repaired can be obtained by directly subtracting the 3D point cloud image of the building component from the 3D point cloud reference image.

[0086] When the building component image is a 2D image, the building component image and the reference image are first segmented, and the segmented building components are compared. The missing part is the defect of the building component.

[0087] For a multi-style fusion building, its building components may be complete or incomplete. According to this step, it is possible to determine whether the building component image is defective and what the defect is. Of course, for a complete building component image, the next part of the building component image restoration work can be omitted. Of course, the complete building component image can also be added to the gallery as a reference image for restoration.

[0088] S4 matches the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library according to the defect position and degree of the defect of the building component image.

[0089] Since the situation of style fusion in multi-style fusion buildings is relatively complicated, when it is possible to determine whether the building component is the main style, the main style fused with foreign styles, or purely foreign styles, the style of the building component is determined first. Then, according to the style of the building component, it is determined whether to use the complete image (multi-style fusion) for restoration or the reference image (single style, including the main style and foreign styles) for restoration.

[0090] In addition, when the damage of a building component exceeds a certain degree, it is difficult to determine which of the above styles it belongs to. Therefore, more specifically, the present embodiment also includes the following steps: S41 obtains the defect state of the building component image by using the defect position and defect degree of the building component image.

[0091] Specifically, the step of obtaining the defect state of the building component image includes: Defect state = a*defect position score + b*defect degree score, where a and b are defect coefficients. For example, taking the 100-point system as an example, if the defect position is at the corner, the defect position score is 5; if the defect position is at the edge, the defect position score is 10; if the defect position is in the middle, the defect position score is 15, and then the number of defects at each position is multiplied by the corresponding score, and so on. If the defect area is less than 5%, the defect degree score is 50; if the defect area is between 5% and 10%, the defect degree score is 70; if the defect area is above 10%, the defect degree score is 100, and so on. The specific value of the defect coefficient can be adjusted according to the actual situation, for example, a is 35% and b is 75%.

[0092] The lower the final defect score, the higher the integrity of the building component.

[0093] S42, when the defect state is within the first threshold interval, determines the style of the building component image to obtain the style of the building component image, and matches the complete image of the building component of the multi-style fusion building in the reference library or the benchmark image corresponding to the building component in the benchmark image library according to the style of the building component image.

[0094] When the defect state is within the first threshold interval, it means that the defect of the building component can be judged by style. Specifically, the judgment is made by the style type that is manually annotated in advance (for example, if it is a single-style image, the corresponding style can be annotated; if it is a fusion of multiple styles, the multiple styles that are fused can be annotated). If it is judged to be a single style, it is enough to match the reference image of the single style (that is, the same style); if it is judged to be a fusion of multiple styles, it is enough to match the corresponding complete image of the fusion of multiple styles.

[0095] S43 matches the complete image of the building component of the multi-style fusion building in the reference library when the defect state is within the second threshold interval.

[0096] When the defect state is within the second threshold interval, it means that the style of the building component can no longer be determined. In this case, the probability of meeting the requirements after repair using a complete image with multiple styles fused is higher, so the corresponding complete image with multiple styles fused is matched. Specifically, since there are many ways to fuse multiple styles, even if multiple styles are manually annotated in advance, it is possible to match multiple complete images. Therefore, the complete image with the highest similarity can be matched from the multiple matched complete images. For example, the similarity between the contour information and the contour information of the corresponding area in the complete image is calculated, and a complete image with the highest similarity is matched.

[0097] S44: When the defect state is within the third threshold range, matching the main style reference image of the building component in the reference image library. When the defect state is within the third threshold range, it means that the style of the building component can no longer be determined and the degree of defect is very serious. In this case, the probability of success is highest when the main style benchmark image is used for repair.

[0098] It can be understood that, in this embodiment, the first threshold interval < the second threshold interval < the third threshold interval.

[0099] S5 refers to the complete image of the building component of the multi-style fusion building in the matched reference library or the benchmark image of the building component in the benchmark image library, and repairs the building component image to obtain the building component repair image.

[0100] When a reference image is obtained, the building components can be repaired according to the reference image. When repairing, the position can be filled according to the reference image.

[0101] S6 uses the architectural component restoration image to restore the architectural image to be restored to obtain a restored architectural image.

[0102] After obtaining the restored images of all building components (of course, the original complete building component images can be used directly), all the building component images are assembled to obtain the final restored building image.

[0103] In fact, it is unknown whether the restored architectural image meets the requirements. It is possible that the style of the restored architectural image will be very abrupt and not meet the requirements. Therefore, after obtaining the restored architectural image, further processing is required.

[0104] S7 calculates the manual review degree according to the defect state of each building component image in the building image to be restored. If the manual review degree exceeds the manual review threshold, the restored building image is submitted to manual review.

[0105] The defect state represents the degree of defect of building components. If the degree of defect of the entire building is low, there is more confidence in the repair of the above steps. Therefore, it can be directly considered that the restoration is successful and no subsequent steps are required.

[0106] However, when the degree of damage to the entire building exceeds a certain threshold, manual review will be required, and experts will analyze whether the repair work is qualified.

[0107] In this step, the method of calculating the manual appraisal degree according to the defect state of each building component image in the building image to be repaired includes: S71 divides architectural components into several categories according to their influence on the overall style of the building, and sets a manual review coefficient for each category of architectural component images.

[0108] For example, the external, more conspicuous building components are divided into one category, the less conspicuous ones into another category, and the internal, less conspicuous ones into another category. The first category is given a higher weight; the second category has the second highest weight, and the third category has the lowest weight. The manual review coefficient is adaptively set according to the weight of the type.

[0109] S72 calculates the average number of defect states of each type of building component image, and adjusts the average number using the manual review coefficient of the building component image of that type to obtain the manual review degree of the single type of building component image.

[0110] For example, there are 5 types of the first category of building components. After obtaining the average number of defect states of this type of building components and multiplying it by the manual evaluation coefficient, the manual evaluation degree of this type of building components can be obtained.

[0111] S73 takes the sum of the manual review degrees of all single-type building component images as the manual review degree of the building image to be restored.

[0112] Continuing with the previous example, by summing up the manual review degrees of the three types of building components, we can get the manual review degree of the entire building.

[0113] Finally, the manual review degree is compared with the manual review threshold to determine whether the restored building image still needs manual review.

[0114] S8, in the case where the manual review has not been passed, replace the reference of the image of the building component in the defective state within the third threshold interval to perform secondary repair to obtain a secondary repaired image of the building component; use the secondary repaired image of the building component to replace the repaired image of the building component, and then perform secondary restoration of the building image to obtain a secondary restored building image; and perform secondary manual review on the secondary restored building image.

[0115] After manual review by experts, it was found that the restored architectural image style was abrupt and did not meet the requirements, indicating that there was a problem with the restored architectural component image. At this time, the restoration strategy needs to be adjusted.

[0116] The most likely problem is the image of the building component whose defective state is within the third threshold interval. In the original strategy, the image of the building component whose defective state is within the third threshold interval is referenced by the reference image of the main style, but it does not meet the requirements. Therefore, in this step, the complete image of multi-style fusion is used for restoration to form a secondary restoration image of the building component.

[0117] The images of building components whose defects are within the first and second threshold ranges are repaired according to the original strategy without changing.

[0118] Then the image is stitched and restored again to obtain the second restored image of the building. The second restored image of the building needs to be manually reviewed to see if it is qualified.

[0119] S9, in the case where the second manual review fails, replaces the reference of the building component image in the defective state within the second threshold interval for secondary repair; uses the secondary repaired image of the building component to replace the repaired image of the building component, and then performs three restorations on the building image to be repaired to obtain the three restored building images; and performs three manual reviews on the three restored building images.

[0120] After manual review by experts, it was considered that the style of the architectural image after the second restoration was still abrupt and did not meet the requirements. It was possible that there was a problem with the repair of the defective architectural components within the second threshold range.

[0121] In the original strategy, the image of the building component with a defect state within the second threshold interval is referenced by the complete image of multi-style fusion, but it does not meet the requirements. Therefore, in this step, the reference image of the main style is used for restoration to form a secondary restoration image of the building component.

[0122] The images of building components whose defective states are within the first and third threshold intervals are repaired according to the original strategy without changing.

[0123] Then the image is stitched together and restored again to obtain the three restored images of the building. The three restored images of the building need to be manually reviewed to see if they are qualified.

[0124] S10, in the case that the three manual reviews have not been passed, the reference of the building component image in the defective state within the second threshold interval and the third threshold interval is replaced simultaneously for secondary restoration; the repaired image of the building component is replaced with the secondary repaired image of the building component, and then the building image to be repaired is restored four times to obtain the four restored building images; the four restored building images are subjected to four manual reviews.

[0125] After manual review by experts, it was considered that the architectural images after three restorations still had an abrupt style and did not meet the requirements. It was possible that there were problems with the restoration of the building components in the second and third threshold ranges.

[0126] In the original strategy, the image of the building component whose defective state is within the third threshold interval is referenced by the reference image of the main style; the image of the building component whose defective state is within the second threshold interval is referenced by the complete image of multi-style fusion, but it does not meet the requirements. Therefore, in this step, the image of the building component whose defective state is within the second threshold interval is repaired by using the reference image of the main style to form a secondary repaired image of the building component. The image of the building component whose defective state is within the third threshold interval is repaired by using the complete image of multi-style fusion to form a secondary repaired image of the building component.

[0127] The images of building components whose defective states are within the first threshold range are repaired according to the original strategy without changing.

[0128] Then the image is stitched together and restored again to obtain the fourth restored image of the building. And the fourth restored image of the building needs to be manually reviewed to see if it is qualified.

[0129] In the case that S11 fails to pass four manual reviews, it obtains architectural images of the same era, the same region and the same main style as the main style of the architectural image to be restored as reference for restoration.

[0130] If all four restorations fail to pass manual review, the next best option is to obtain architectural images of the same era, same region and with the same main style as the main style of the building image to be restored as a reference for restoration.

[0131] It is foreseeable that the results of this repair will still need to be submitted to manual review. If the manual review fails again, it means that automatic repair is hopeless and the repair work can only be taken over by manual personnel.

[0132] Example 3: Figure 3A As shown, the present invention also provides a repair device for damaged buildings of multiple styles, which comprises: A decomposition module, used for decomposing the building image to be restored, and obtaining a separate building component image of each building component in the building image to be restored; A preprocessing module preprocesses each building component image and extracts contour information of the building component in the building component image; A defect acquisition module is used to compare the contour information of the building component with the reference image of the building component in the reference image library to obtain the defect position and degree of the building component image; A matching module is used to match the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library according to the defect position and degree of the defect of the building component image; A building component image restoration module is used to repair the building component image by referring to the complete image of the building component of the multi-style fusion building in the matched reference library or the reference image of the building component in the reference image library to obtain a restored image of the building component; The building image restoration module uses the building component restoration image to restore the building image to be restored to obtain the restored building image.

[0133] In some embodiments, the matching module specifically includes: A defect state acquisition unit, used to acquire the defect state of the building component image by using the defect position and defect degree of the building component image; A judging unit, used for judging whether the defect state of the building component image is within a first threshold interval, a second threshold interval, or a third threshold area; a first reference image acquisition unit, configured to, when the judgment unit judges that the defect state of the building component image is within a first threshold interval, judge the main style of the building component image to acquire the style of the building component image, and match the complete image of the building component of the multi-style fusion building in the reference library or the reference image of the building component in the reference image library according to the style of the building component image; A second reference image acquisition unit is configured to match a complete image of a building component of a multi-style fusion building in a reference image library when the judgment unit judges that the defect state of the building component image is within a second threshold interval; The third reference image acquisition unit is used to match a reference image of a main style of the building component in a reference image library when the judgment unit judges that the defect state of the building component image is within a third threshold range.

[0134] Example 4: Figure 3B As shown, the present invention also provides a repair device for damaged building components of multiple styles, which provides repair reference sample images for building component repair by constructing a digital pattern reference library for multi-style fusion building repair. Specifically, it includes: A reference image library construction module is used to construct a reference image library consisting of several reference images of building components of a single style; The contour information extraction module obtains a plurality of images of the same building component to be restored of the same multi-style fusion building; pre-processes the image of the building component to be restored, and extracts the contour information of the building component in the image of the building component to be restored; A similarity calculation module is used to calculate the similarity between the image of the building component to be repaired and the reference image of the building component in the reference image library, and match the reference image whose similarity is higher than the similarity threshold; A defect position comparison module is used to compare the contour information of the image of the building component to be repaired with the reference image of the building component to obtain the defect position of the image of the building component to be repaired; A fusion and reconstruction module is used to fuse and reconstruct multiple images of building components to be repaired with different defect positions to form a fused image of the building component; The reference library construction module manually reviews the fused images of building components and adds the fused images that meet the requirements into the reference library.

[0135] In some embodiments, the device also includes: a secondary reconstruction module, which is used to manually mark the age and region of the building components in the image of the building component to be repaired; when the defect of the fused image exceeds the defect threshold, match the reference image of the same age, region and main style for fusion reconstruction, and submit the fused and reconstructed fused image for manual review, and add the fused image that meets the requirements to the reference library.

[0136] Embodiment 5: The present invention also provides a computer-readable storage medium, wherein the storage medium stores a computer program; when the computer program is executed, the above-mentioned method for repairing damaged buildings or building components of multiple styles can be implemented.

[0137] Embodiment 6: The present invention also provides a computer system, comprising a memory and a processor; a computer program is stored in the memory; when the computer program is executed by the processor, the above-mentioned method for repairing damaged buildings or building components of multiple styles can be implemented.

[0138] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a computer terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0140] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A method for constructing a digital pattern reference library for multi-style fusion architectural restoration, characterized in that: Includes steps: S11 builds a benchmark image library consisting of several single-style building component benchmark images; S12 obtains a plurality of images of the same building component to be restored in the same multi-style fusion building; and preprocessing the image of the building component to be repaired, extracting contour information of the building component in the image of the building component to be repaired; S13: selecting an image with the most complete contour information from the plurality of images of the building component, calculating the similarity between the image and the reference image corresponding to the building component in the reference image library, and matching the reference images whose similarity is higher than a similarity threshold; S14 compares the contour information of the image of the building component to be repaired with the matched reference image of the building component to obtain the defect position of the building component to be repaired; S15: fusing and reconstructing a plurality of images of building components to be repaired having different defect positions to form a fused image of the building components; S16 manually reviews the fused images of building components and adds the fused images that meet the requirements into the reference library; When the defect of the fused image exceeds the defect threshold, S17 matches the reference image of the same era, region and main style for fusion reconstruction, and submits the fused and reconstructed fused image for manual review, and adds the fused image that meets the requirements into the reference library.

2. The method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to claim 1, characterized in that: The image of the building component to be repaired is a 2D image or / and a 3D image, and the building component reference image includes a 2D reference image and a 3D point cloud reference image; the 2D reference image of the building component reference image includes 2D reference images at several angles.

3. The method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to claim 2 is characterized in that: When acquiring images of building components to be restored of a multi-style fusion building through on-site acquisition, 2D images of the building components at several angles and 3D point cloud images are collected.

4. The method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to claim 2 is characterized in that: The degree of defect of the image of the building component to be repaired is obtained, and when a plurality of images of the building component to be repaired with different defect positions are fused and reconstructed, the image of the building component to be repaired with a degree of defect less than a degree threshold is selected.

5. The method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to claim 2 is characterized in that: In step S14, when the image of the building component to be repaired is a 3D point cloud, the defect of the building component to be repaired can be obtained by directly subtracting the 3D point cloud image of the building component to be repaired from the 3D point cloud reference image.

6. The method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to claim 2 is characterized in that: In step S14, when the image of the building component to be repaired is a 2D image, the image of the building component to be repaired and the reference image are first segmented by instance, and the segmented building components are compared, and the missing part is the defect of the building component.

7. The method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to claim 2 is characterized in that: In step S15, if there are two images with the same defect position, the image of the building component to be repaired with a defect degree less than a degree threshold is selected.

8. A method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to any one of claims 1 to 7, characterized in that: The building components include doors, beams, walls and columns.

9. A method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to any one of claims 1 to 7, characterized in that: The single style includes Hui style.

10. A method for constructing a digital pattern reference library for multi-style fusion architectural restoration according to any one of claims 1 to 7, characterized in that: The method also includes the step of manually marking the age and region of the building components in the image of the building components to be repaired.

Images