Method, apparatus, device and storage medium for generating three-dimensional building model

Abstract

One or more embodiments of this specification provide a method, apparatus, device, and storage medium for generating a three-dimensional building model. The method includes: obtaining component diagrams of each component included in a two-dimensional building image; matching the component diagrams with three-dimensional component models in a three-dimensional asset library to determine the three-dimensional component models corresponding to the component diagrams; obtaining a structure diagram corresponding to the two-dimensional building image, where the structure diagram indicates the depth information of each component and the positional relationship between components; and assembling the three-dimensional component models according to the structure diagram to obtain a three-dimensional building model. The embodiments of this specification can automatically generate a corresponding high-precision three-dimensional building model based on a two-dimensional building image, realizing the efficient generation of three-dimensional materials corresponding to two-dimensional materials.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of three-dimensional graphics processing technology, and particularly to a method, apparatus, device, and storage medium for generating a three-dimensional building model. Background Art

[0002] Current building model construction software usually allows users to build a building model by calling three-dimensional components from a three-dimensional asset library.

[0003] However, for buildings with complex structures, this model construction method is very time-consuming, and it is impossible to effectively construct buildings with unknown building structure details. With the increasing demand for three-dimensional materials, this construction method can no longer meet the requirements. Summary of the Invention

[0004] In view of this, one or more embodiments of this specification provide a method, apparatus, device, and storage medium for generating a three-dimensional building model.

[0005] To achieve the above object, one or more embodiments of this specification provide the following technical solutions:

[0006] According to the first aspect of one or more embodiments of this specification, a method for generating a three-dimensional building model is proposed, including:

[0007] Obtain part diagrams of each part included in the two-dimensional building image;

[0008] Match the part diagram with three-dimensional part models in the three-dimensional asset library to determine the three-dimensional part model corresponding to the part diagram;

[0009] Obtain the structure diagram corresponding to the two-dimensional building image, where the structure diagram indicates the depth information of each part and the positional relationship between parts;

[0010] Assemble the three-dimensional part models according to the structure diagram to obtain a three-dimensional building model.

[0011] In some embodiments, the obtaining part diagrams of each part included in the two-dimensional building image includes:

[0012] Obtain the part detection result of the two-dimensional building image, where the part detection result includes the attribute information of each part and the detection frame of each part;

[0013] Obtain the part segmentation result of the two-dimensional building image;

[0014] According to the part segmentation result within the detection frame, obtain the part diagram of the part, where the part diagram has the attribute information of the part.

[0015] In some embodiments, obtaining the component segmentation result of the two-dimensional building image includes:

[0016] Inputting the two-dimensional building image into an entity segmentation network to obtain a mask for each component;

[0017] Mapping the mask of each component to the two-dimensional building image to obtain the component segmentation result.

[0018] In some embodiments, obtaining the structure diagram corresponding to the two-dimensional building image includes:

[0019] Determining the position of each component in the two-dimensional building image according to the position information of the detection frame of each component;

[0020] Determining the positional relationship between components according to the positions of the components;

[0021] Inputting the component diagram of each component into a depth detection network to obtain the depth information of each component;

[0022] Obtaining the structure diagram corresponding to the two-dimensional building image according to the depth information of each component and the positional relationship between components.

[0023] In some embodiments, matching the component diagram with the three-dimensional component model in the three-dimensional asset library includes:

[0024] Obtaining the two-dimensional rendering diagrams of the three-dimensional component model from multiple perspectives, where at least one two-dimensional rendering diagram is obtained for each perspective;

[0025] Matching the component diagram with the obtained multiple two-dimensional rendering diagrams respectively;

[0026] Taking the highest matching degree in the matching results as the matching degree between the component diagram and the three-dimensional component model.

[0027] In some embodiments, taking the highest matching degree in the matching results as the matching degree between the component diagram and the three-dimensional component model includes:

[0028] Obtaining the number of feature point matches between the component diagram and each two-dimensional rendering diagram;

[0029] Determining the matching degree between the component diagram and the three-dimensional component model according to the maximum value among multiple matching numbers.

[0030] In some embodiments, the three-dimensional component model in the three-dimensional asset library has an attribute label, and the component diagram has the attribute information of the component. Matching the component diagram with the three-dimensional component model in the three-dimensional asset library includes:

[0031] Determine the parts to be matched in the three-dimensional asset library that have the same type of tags according to the attribute information of the part diagrams;

[0032] Match the part diagram with the three-dimensional part model of the part to be matched. In some embodiments, the attribute information includes one or more of category information, construction period information, construction location information, and construction cultural background information.

[0033] In some embodiments, the category information is obtained according to the part detection result of the two-dimensional building image or according to user input.

[0034] One or more of the construction period information, the construction location information, and the construction cultural background information are obtained according to the information associated with the two-dimensional building image or according to user input.

[0035] According to the second aspect of one or more embodiments of this specification, a device for generating a three-dimensional building model is proposed, including:

[0036] A first acquisition unit for acquiring part diagrams of each part included in the two-dimensional building image;

[0037] A second acquisition unit for matching the part diagram with the three-dimensional part model in the three-dimensional asset library to determine the three-dimensional part model corresponding to the part diagram;

[0038] A matching unit for acquiring a structure diagram corresponding to the two-dimensional building image, where the structure diagram indicates the depth information of each part and the positional relationship between the parts;

[0039] A construction unit for assembling the three-dimensional part models according to the structure diagram to obtain a three-dimensional building model.

[0040] In some embodiments, the first acquisition unit is specifically configured to:

[0041] Acquire the part detection result of the two-dimensional building image, where the part detection result includes the attribute information of each part and the detection frame of each part;

[0042] Acquire the part segmentation result of the two-dimensional building image;

[0043] According to the part segmentation result within the detection frame, obtain the part diagram of the part, and the part diagram has the attribute information of the part.

[0044] In some embodiments, when the first acquisition unit is used to acquire the part segmentation result of the two-dimensional building image, it is specifically configured to:

[0045] Input the two-dimensional building image into an entity segmentation network to obtain masks of each part;

[0046] Map the masks of the respective components to the two-dimensional building image to obtain the component segmentation result.

[0047] In some embodiments, when the first acquisition unit is used to acquire the structure diagram corresponding to the two-dimensional building image, it is specifically configured to:

[0048] Determine the positions of the respective components in the two-dimensional building image according to the position information of the detection frames of the respective components;

[0049] Determine the positional relationship between the components according to the positions of the respective components;

[0050] Input the component diagrams of the respective components into the depth detection network to obtain the depth information of the respective components;

[0051] Obtain the structure diagram corresponding to the two-dimensional building image according to the depth information of the respective components and the positional relationship between the components.

[0052] In some embodiments, the matching unit is specifically configured to:

[0053] Obtain the two-dimensional rendering diagrams of the three-dimensional component model from multiple perspectives, where at least one two-dimensional rendering diagram is obtained for each perspective;

[0054] Match the component diagram with the obtained multiple two-dimensional rendering diagrams respectively;

[0055] Use the highest matching degree in the matching results as the matching degree between the component diagram and the three-dimensional component model.

[0056] In some embodiments, when the matching unit is used to use the highest matching degree in the matching results as the matching degree between the component diagram and the three-dimensional component model, it is specifically configured to:

[0057] Obtain the number of feature point matches between the component diagram and each two-dimensional rendering diagram;

[0058] Determine the matching degree between the component diagram and the three-dimensional component model according to the maximum value among the multiple matching numbers.

[0059] In some embodiments, the three-dimensional component models in the three-dimensional asset library have attribute tags, and the matching unit is specifically configured to:

[0060] Determine the to-be-matched components with the same type of tags in the three-dimensional asset library according to the attribute information of the component diagram;

[0061] Match the component diagram with the three-dimensional component models of the to-be-matched components.

[0062] In some embodiments, the attribute information includes one or more of category information, building period information, building location information, and building cultural background information.

[0063] In some embodiments, the category information is obtained based on the component detection result of the two-dimensional building image or based on user input.

[0064] One or more of the building period information, the building location information, and the building cultural background information are obtained based on the information associated with the two-dimensional building image or based on user input.

[0065] According to a third aspect of one or more embodiments of the present specification, an electronic device is provided, including:

[0066] A processor;

[0067] A memory for storing instructions executable by the processor;

[0068] Wherein, the processor realizes the method proposed in one or more embodiments of the present specification by running the executable instructions.

[0069] According to a fourth aspect of one or more embodiments of the present specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method proposed in one or more embodiments of the present specification are realized.

[0070] In the embodiments of the present specification, component diagrams of each component included in the two-dimensional building image are obtained, and the component diagrams are matched with three-dimensional component models in the three-dimensional asset library to determine the three-dimensional component models corresponding to the component diagrams; a structure diagram corresponding to the two-dimensional building image is obtained, and the structure diagram indicates the positional relationship between each component and the depth information of each component. The three-dimensional component models matched by each component are assembled according to the structure diagram to obtain a three-dimensional building model. The embodiments of the present specification can automatically generate a corresponding high-precision three-dimensional building model according to the two-dimensional building image, realizing the efficient generation of three-dimensional materials corresponding to two-dimensional materials. Description of the Drawings

[0071] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present disclosure;

[0072] Figure 2 A flowchart of a method for generating a three-dimensional building model provided by an exemplary embodiment;

[0073] Figure 3 A flowchart of a method for obtaining a component diagram provided by an exemplary embodiment;

[0074] Figure 4It is a flowchart of a method for obtaining a structural diagram corresponding to a two-dimensional building image provided by an exemplary embodiment;

[0075] Figures 5A to 5D It is a schematic diagram of a multi-view rendering method provided by an exemplary embodiment;

[0076] Figure 6 It is a schematic diagram of a method for generating a three-dimensional building model provided by an exemplary embodiment;

[0077] Figure 7 It is a block diagram of a device for generating a three-dimensional building model provided by an exemplary embodiment;

[0078] Figure 8 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Detailed implementation manners

[0079] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with one or more embodiments of this specification. On the contrary, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0080] It should be noted that: In other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0081] One or more embodiments of this specification provide a method for generating a three-dimensional building model. This method can be executed by an electronic device such as a terminal device or a server. The terminal device can be a fixed terminal or a mobile terminal, such as a mobile phone, a tablet computer, a game console, a desktop computer, an advertising machine, an all-in-one machine, a vehicle-mounted terminal, etc. The server includes a local server or a cloud server, etc. This method can be implemented by a processor calling computer-readable instructions stored in a memory. In one example, it can be executed by software or an application program installed in the electronic device. The three-dimensional building model generated based on the method described in the embodiment can be applied to various application scenarios including virtual shooting, games, etc. For example, in a virtual shooting scenario, a display screen can be rendered on the screen based on the three-dimensional building model as a shooting background and photographed together with the foreground in the real world to obtain the required shooting screen.

[0082] The above virtual shooting can be simply understood as follows: by displaying the background on the display screen, the actor stands in front of the screen and performs, and is filmed to replace the real scene shooting.

[0083] Figure 1 A schematic diagram showing an application scenario according to an embodiment of the present disclosure is as Figure 1 shown. The virtual shooting system includes three physical screens (021, 022, 023). The mobile terminal 01 can execute the method for generating a three-dimensional building model according to the embodiment of the present disclosure. Communication connections can be established between the mobile terminal 01 and the three physical screens (021, 022, 023) respectively. A communication connection can be established between the mobile terminal 01 and the computing device 03. The mobile terminal 01 can control each physical screen to display an identification graphic array and send multiple screen images obtained by image acquisition of each physical screen to the computing device 03. The computing device 03 can generate and display a screen model according to the multiple screen images sent by the mobile terminal 01.

[0084] Among them, the physical screens used in the virtual shooting system can be of types such as LED display screens and liquid crystal display screens, and can be of structures such as curved screens or flat screens. It should be understood that those skilled in the art can customize the types, quantities, sizes, resolutions, etc. of the physical screens in the virtual shooting system according to actual needs, and the embodiments of the present disclosure do not limit this. It should be understood that the embodiments of the present disclosure do not limit the communication connection methods between devices.

[0085] Figure 2 is a flowchart of a method for generating a three-dimensional building model shown in an exemplary embodiment of this specification. The method includes steps 201 to 204.

[0086] In step 201, component diagrams of each component included in the two-dimensional building image are obtained.

[0087] In the embodiment of the present disclosure, the two-dimensional building image refers to an image including a building, including photos taken of the building entity, and also including drawn pictures of the building, etc.

[0088] By performing object detection on the two-dimensional building, each component included in the two-dimensional building image can be determined. Among them, a component is a basic unit constituting a building module, including doors, windows, columns, walls, steps, railings, roofs, etc.

[0089] For each component determined by the object detection, a component diagram can be obtained according to the image area corresponding to the detection frame in the two-dimensional building image, or the image area corresponding to the detection frame can be cropped to obtain individual component diagrams of each component.

[0090] In step 202, the component diagram is matched with the 3D component models in the 3D asset library to determine the 3D component model corresponding to the component diagram.

[0091] In the embodiments of the present disclosure, a 3D asset library can be set up in advance. In this 3D asset library, 3D models of multiple components are stored, and each 3D component model corresponds to a component. Among them, each category of components can include multiple styles of models. For example, under the category of doors, there are 3D models of multiple styles of doors, and under the category of windows, there are 3D models of multiple styles of windows, and so on.

[0092] In step 203, the structure diagram corresponding to the 2D building image is obtained, and the structure diagram indicates the depth information of each component and the positional relationship between the components.

[0093] In some embodiments, the structure diagram may further include the attribute information of each component.

[0094] In step 204, the 3D component models are assembled according to the structure diagram to obtain a 3D building model.

[0095] According to the depth information of each component and the positional relationship between the components indicated by the structure diagram, the positions of the 3D component models corresponding to these components in the 3D space can be determined, and the 3D component models are deployed according to this position to realize the construction of the 3D building model.

[0096] In the embodiments of this specification, the component diagrams of each component included in the 2D building image are obtained, and the component diagrams are matched with the 3D component models in the 3D asset library to determine the 3D component models corresponding to the component diagrams; the structure diagram corresponding to the 2D building image is obtained, and the structure diagram indicates the positional relationship between each component and the depth information of each component. The 3D component models matched by each component are assembled according to the structure diagram to obtain a 3D building model. The embodiments of this specification can automatically generate a corresponding high-precision 3D building model according to the 2D building image, realizing the efficient generation of 3D materials corresponding to 2D materials.

[0097] Figure 3 It is a flowchart of a method for obtaining a component diagram shown in an exemplary embodiment of this specification. This method includes steps 301 to 303.

[0098] In step 301, the component detection result of the 2D building image is obtained.

[0099] In the embodiments of the present disclosure, the 2D building image can be input into a pre-trained object detection network to obtain the component detection result. The component detection result includes the attribute information of each component and the detection frame of each component.

[0100] The object detection network can be pre-trained using sample component diagrams annotated with the attributes of building components.

[0101] For building component detection scenarios with different attributes, the object detection network can be trained by increasing the annotated data of building components with that attribute, so that the object detection network can detect the types of building components contained in the two-dimensional building image.

[0102] Among them, the attribute information includes one or more of category information, building period information, building location information, and building cultural background information.

[0103] Taking the detection scenario of ancient Chinese building components as an example, through the category annotation data of ancient building components, such as cornices, brackets, and spandrels, the trained object detection network can detect various types of ancient building components contained in the two-dimensional building and output the category detection results.

[0104] Since components of the same category in ancient buildings may have different structural features in different building periods, different regions, and different cultural backgrounds.

[0105] For example, brackets in different dynasties in China usually have different structural features, including different styles or different sizes. For example, inclined brackets began to appear in the brackets of the Liao Dynasty. By training the object detection network with the annotated data of ancient building components with building period information, the trained object detection network can output the building period information of ancient building components.

[0106] Since China has a vast territory, buildings in different regions may have different styles or factions, such as Huizhou style, Fujian style, Beijing style, Suzhou style, etc. Components of the same category in different factions may have different structural features. Therefore, the faction of ancient buildings can be used as the regional annotation information of building components, and by training the object detection network with the annotation data, the trained object detection network can output the regional information of ancient building components.

[0107] Under different cultural backgrounds, for example, for different religions and ethnic groups, components of the same category in buildings may have different structural features. For example, the styles of components of the same category may be different in Buddhist buildings and Taoist buildings. Therefore, by training the object detection network with the annotated data of ancient building components with cultural background information, the trained object detection network can output cultural background information such as religion and ethnicity of ancient building components.

[0108] Since building components with different attributes may vary in size. For example, decorative components such as que ti are only one-tenth or even one-hundredth of the size of wall components; and the size of dougong also changed in different dynasties. Therefore, the object detection network applied in the embodiments of the present disclosure has the ability to detect objects of different scales.

[0109] In one example, multiple network layers can be used to extract features from the input two-dimensional building image respectively, obtaining feature maps of different scales, and performing an upsampling operation on the shallower feature maps to make their sizes the same as those of the deeper feature maps for feature map fusion, and performing object detection based on the fused features.

[0110] By fusing feature maps of different scales, the object detection ability of the object detection network for objects of different scales is improved, and the accuracy of object detection is also improved.

[0111] The detection results obtained using the object detection network include the detection frames of the detected components and indicate the attribute information of the components in the detection frames. That is, in the detection image output by the object detection network, the detection frames of the components are included, and the attribute information of the included components is labeled in the detection frames.

[0112] In step 302, obtain the component segmentation result of the two-dimensional building image.

[0113] In the embodiments of the present disclosure, an entity segmentation network can be used to obtain the component segmentation result of the two-dimensional building image.

[0114] In one example, the two-dimensional building image can be input into the entity segmentation network to obtain the masks of the components; map the masks of the components to the two-dimensional building image to obtain the component segmentation result.

[0115] In step 303, obtain the component diagram of the component according to the component segmentation result within the detection frame.

[0116] Since in step 301, all the components included in the two-dimensional building image have been detected and the attribute information of the components has been obtained, the attribute information of each segmented component can be determined according to the corresponding position of the detection frame in the component segmentation result, so that an entity segmentation result with attribute information can be obtained. According to the component segmentation result within the detection frame, the component diagram of the component can be obtained.

[0117] In one example, the image area outside the segmentation result within the detection frame can be set as the background area, for example, setting the pixel value to 255 or 0, that is, clearing the image outside the component and only retaining the image of the component itself.

[0118] In the embodiments of the present disclosure, a component diagram is obtained by combining the component detection result and the component segmentation result. Compared with obtaining the component diagram based on the component detection result, the accuracy of the component diagram can be improved, thereby improving the accuracy of three-dimensional asset matching based on the component diagram.

[0119] In some embodiments, before obtaining the component diagrams of the respective components included in the two-dimensional building image, the image may be corrected, including at least one of distortion correction or skew correction of the image, to ensure the accuracy of subsequent detection.

[0120] Figure 4 FIG. is a flowchart of a method for obtaining a structure diagram corresponding to a two-dimensional building image shown in an exemplary embodiment of this specification. The method includes steps 401 to 404.

[0121] In step 401, according to the position information of the detection frames of the respective components, the positions of the respective components in the two-dimensional building image are determined.

[0122] The component detection result of the two-dimensional building image includes the detection frames of the respective components. For each component, the position of the component in the image can be determined according to the position information of the detection frame where the component is located. For example, in the case where the detection frame is a rectangular detection frame, the position of the center point of the rectangular detection frame can be used as the position of the component.

[0123] In step 402, according to the positions of the respective components, the positional relationship between the components is determined.

[0124] In step 403, the component diagrams of the respective components are input into a depth detection network to obtain the depth information of the respective components.

[0125] In the embodiments of the present disclosure, a pre-trained depth estimation network can be used to estimate the depth of each component from the two-dimensional building image.

[0126] In step 404, according to the depth information of the respective components and the positional relationship between the components, the structure diagram corresponding to the two-dimensional building image is obtained.

[0127] Since the two-dimensional building image may be an image taken from a certain angle, the obtained component diagram is correspondingly taken from that angle. Therefore, the component diagram may not be able to show all the features or important features of the component, and thus cannot be accurately matched with the three-dimensional component model in the three-dimensional asset library.

[0128] In view of this, the embodiments of this specification propose a method for matching a component diagram with a three-dimensional component model.

[0129] First, two-dimensional renderings of the three-dimensional component model are respectively obtained from multiple perspectives, where at least one two-dimensional rendering is obtained for each perspective.

[0130] Figure 5A Shows multiple perspectives adopted in the multi - perspective rendering method, including the top - view perspective, 45 - degree perspective, and 0 - degree perspective.

[0131] From the top - view perspective, a two - dimensional rendering can be obtained, as Figure 5B shown, where the model marked with "." at the top is the model seen from the top - view perspective, and a two - dimensional rendering of this model is obtained from the top - view perspective.

[0132] From the 45 - degree perspective, four two - dimensional renderings can be obtained, as Figure 5C shown, where the model marked with "." at the top is the model seen from the 45 - degree perspective, and a two - dimensional rendering of each model is obtained from the 45 - degree perspective.

[0133] From the 0 - degree perspective, eight two - dimensional renderings can be obtained, as Figure 5D shown, where the model marked with "." at the top is the model seen from the 0 - degree perspective, and a two - dimensional rendering of each model is obtained from the 0 - degree perspective.

[0134] Those skilled in the art should understand that the above - selected perspectives and the number of two - dimensional renderings obtained at each perspective are only examples and are not intended to be limiting.

[0135] Next, the component diagram is respectively matched with the obtained multiple two - dimensional renderings. Among them, the multiple two - dimensional renderings can be all the two - dimensional renderings obtained from each perspective.

[0136] Finally, the highest matching degree in the matching results is used as the matching degree between the component diagram and the three - dimensional component model.

[0137] Specifically, the component diagram is respectively matched with each two - dimensional rendering to obtain the matching degree between the component diagram and each two - dimensional rendering, and the highest matching degree (maximum value) among all the matching degrees is used as the matching degree between the component diagram and the three - dimensional component model.

[0138] In this way, it can be found from which perspective the three - dimensional component model is observed and is most similar to the component diagram, and based on the similarity between the two - dimensional rendering of this perspective and the component diagram, the matching degree between the component diagram and the three - dimensional component model is evaluated.

[0139] In one example, the component diagram can be matched with the feature points of each two - dimensional rendering, and the number of feature - point matches between the component diagram and each two - dimensional rendering can be obtained; and based on the maximum value among the multiple numbers of feature - point matches, the matching degree between the component diagram and the three - dimensional component model is determined. That is, the two - dimensional rendering with the largest number of feature - point matches with the component diagram can be selected to evaluate the matching degree between the component diagram and the three - dimensional component model.

[0140] By performing multi - perspective rendering on the 3D component model, multiple 2D rendering images are obtained for evaluating the matching degree between the component drawing and the 3D component model, which can improve the accuracy of matching between the component drawing and the 3D component model.

[0141] In some embodiments, the 3D component models in the 3D asset library have attribute tags for indicating the attribute information of the 3D component models, including one or more of category information, construction period information, construction location information, and architectural cultural background information. In this case, first, according to the attribute information of the component drawing, the to - be - matched components with the same type of tags in the 3D asset library are determined, and then the component drawing is matched with the 3D component models of the to - be - matched components.

[0142] Among them, the category information can be obtained according to the component detection result of the 2D architectural image or according to user input; one or more of the construction period information, the construction location information, and the architectural cultural background information are obtained according to the information associated with the 2D architectural image or according to user input. Among them, the information associated with the 2D architectural image includes the attached description document of the image or the description information marked on the image.

[0143] By adding unique attributes of ancient buildings such as construction period information, construction location information, and architectural cultural background information as matching conditions, and adding corresponding tags to the 3D component models in the 3D asset library, the matching time can be further shortened and the matching accuracy can be improved.

[0144] By the above - mentioned method, the range of matching between the component drawing and the 3D component model can be narrowed, the amount of calculation can be reduced, and the matching efficiency can be improved.

[0145] Figure 6 It is a schematic diagram of a method for generating a 3D building model provided by an exemplary embodiment.

[0146] As Figure 6 shown, the to - be - processed 2D architectural image 601 is corrected. This correction process can be distortion correction or skew correction, or both distortion correction and skew correction can be performed simultaneously to obtain the corrected image 602.

[0147] Target detection is performed on the image 602, and entity segmentation is performed on the image 602. Combining the target detection result and the entity segmentation result, a component segmentation result 603 with attribute information can be obtained. Among them, the attribute information includes at least the category information of the component, and may also include one or more of the construction period information, the construction location information, and the architectural cultural background information.

[0148] According to the component segmentation result 603, the component diagrams of each component included in the image 602 can be obtained. Figure 6 Among them, 6041 is the component diagram of one of the components. By matching the component diagram with the 3D component model in the 3D asset library, the 3D component model corresponding to the component diagram can be determined. Taking the component diagram 6041 as an example, the 3D component model matched in the 3D asset library is 6042.

[0149] On the other hand, according to the component segmentation result 603, the structure diagram 605 corresponding to the image 602 can be obtained, and the structure diagram indicates the depth information of each component and the positional relationship between the components.

[0150] Assemble the 3D component models matched for each component according to the structure diagram 605. Figure 6 606 in it shows the process diagram of the assembly, and finally the 3D component model 607 is obtained.

[0151] See Figure 7 , Figure 7 A 3D building model generation device provided by an exemplary embodiment includes:

[0152] A first acquisition unit 701, configured to acquire the component diagrams of each component included in the two-dimensional building image;

[0153] A second acquisition unit 702, configured to acquire the structure diagram corresponding to the two-dimensional building image, and the structure diagram indicates the positional relationship between each component and the depth information of each component;

[0154] A matching unit 703, configured to match the component diagram with the 3D component model in the asset library to determine the 3D component model corresponding to the component diagram;

[0155] A construction unit 704, configured to assemble the 3D component models according to the structure diagram to obtain a 3D building model.

[0156] In some embodiments, the first acquisition unit is specifically configured to:

[0157] Acquire the component detection result of the two-dimensional building image, where the component detection result includes the attribute information of each component and the detection frame of each component;

[0158] Acquire the component segmentation result of the two-dimensional building image;

[0159] According to the component segmentation result within the detection frame, obtain the component diagram of the component, and the component diagram has the attribute information of the component.

[0160] In some embodiments, when the first acquisition unit is used to acquire the component segmentation result of the two-dimensional building image, it is specifically configured to:

[0161] Input the two-dimensional building image into the entity segmentation network to obtain the masks of each component;

[0162] Map the masks of each component to the two-dimensional building image to obtain the component segmentation result.

[0163] In some embodiments, when the first acquisition unit is used to acquire the structure diagram corresponding to the two-dimensional building image, it is specifically used for:

[0164] Determine the positions of each component in the two-dimensional building image according to the position information of the detection frames of each component;

[0165] Determine the positional relationship between components according to the positions of each component;

[0166] Input the component diagrams of each component into the depth detection network to obtain the depth information of each component;

[0167] Obtain the structure diagram corresponding to the two-dimensional building image according to the depth information of each component and the positional relationship between components.

[0168] In some embodiments, the matching unit is specifically used for:

[0169] Obtain the two-dimensional rendering diagrams of the three-dimensional component model from multiple perspectives, where at least one two-dimensional rendering diagram is obtained for each perspective;

[0170] Match the component diagram with the obtained multiple two-dimensional rendering diagrams respectively;

[0171] Take the highest matching degree in the matching results as the matching degree between the component diagram and the three-dimensional component model.

[0172] In some embodiments, when the matching unit is used to take the highest matching degree in the matching results as the matching degree between the component diagram and the three-dimensional component model, it is specifically used for:

[0173] Obtain the number of feature point matches between the component diagram and each two-dimensional rendering diagram;

[0174] Determine the matching degree between the component diagram and the three-dimensional component model according to the maximum value among multiple matching numbers.

[0175] In some embodiments, the three-dimensional component models in the three-dimensional asset library have attribute tags, and the matching unit is specifically used for:

[0176] Determine the to-be-matched components with the same type of tags in the three-dimensional asset library according to the attribute information of the component diagram;

[0177] Match the component diagram with the three-dimensional component model of the component to be matched.

[0178] In some embodiments, the attribute information includes one or more of category information, construction period information, construction location information, and construction cultural background information.

[0179] In some embodiments, the category information is obtained according to the component detection result of the two-dimensional building image or according to user input.

[0180] One or more of the construction period information, the construction location information, and the construction cultural background information are obtained according to the information associated with the two-dimensional building image or according to user input.

[0181] Figure 8 It is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Please refer to Figure 8 , at the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810. Of course, it may also include other hardware required by other services. One or more embodiments of this specification can be implemented in a software manner. For example, the processor 802 reads the corresponding computer program from the non-volatile memory 810 into the memory 808 and then runs it. Of course, in addition to the software implementation manner, one or more embodiments of this specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware. That is to say, the execution subject of the following processing flow is not limited to each logic unit, and can also be hardware or a logic device.

[0182] The systems, devices, modules, or units illustrated in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any several of these devices.

[0183] In a typical configuration, a computer includes one or more processors (CPUs), an input / output interface, a network interface, and a memory.

[0184] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.

[0185] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0186] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0187] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0188] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0189] It should be understood that although the terms first, second, third, etc. may be used in one or more embodiments of this specification to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".

[0190] The above are only the preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of one or more embodiments of this specification shall be included within the scope protected by one or more embodiments of this specification.

Claims

1. A method for generating a three-dimensional building model for ancient buildings, comprising: Obtaining part diagrams of each part included in a two-dimensional building image, where the part is a basic unit constituting a building module, and the part diagram has attribute information of the part, and the attribute information includes one or more of category information, building period information, building location information, and building cultural background information; The two-dimensional building image refers to an image containing a building; According to the attribute information of the part diagram, determining a to-be-matched part in a three-dimensional asset library with the same type of label, where the label indicates the attribute information of the three-dimensional part model in the three-dimensional asset library; Matching the part diagram with the three-dimensional part model of the to-be-matched part, including: Matching the part diagram with multiple two-dimensional rendering diagrams of the three-dimensional part model respectively to obtain a matching result for each two-dimensional rendering diagram, where the multiple two-dimensional rendering diagrams are two-dimensional rendering diagrams of the three-dimensional part model obtained from multiple perspectives, and at least one two-dimensional rendering diagram is obtained for each perspective; Taking the highest matching degree in the matching results as the matching degree between the part diagram and the three-dimensional part model, and determining the three-dimensional part model corresponding to the part diagram; Obtaining a structure diagram corresponding to the two-dimensional building image, where the structure diagram indicates the depth information of each part and the positional relationship between the parts; Assembling the three-dimensional part models according to the structure diagram to obtain a three-dimensional building model, including: According to the depth information of each part and the positional relationship between the parts indicated by the structure diagram, determining the positions of the three-dimensional part models corresponding to the respective parts in three-dimensional space, and deploying the three-dimensional part models according to the positions to construct a three-dimensional building model.

2. The method according to claim 1, where obtaining part diagrams of each part included in the two-dimensional building image includes: Obtaining a part detection result of the two-dimensional building image, where the part detection result includes the attribute information of each part and the detection frame of each part; Obtaining a part segmentation result of the two-dimensional building image; According to the part segmentation result within the detection frame, obtaining the part diagram of the part, and the part diagram has the attribute information of the part.

3. The method according to claim 2, where obtaining the structure diagram corresponding to the two-dimensional building image includes: Determining the positions of each part in the two-dimensional building image according to the position information of the detection frames of each part; Determining the positional relationship between the parts according to the positions of each part; Inputting the part diagrams of each part into a depth detection network to obtain the depth information of each part; According to the depth information of each part and the positional relationship between the parts, obtaining the structure diagram corresponding to the two-dimensional building image.

4. The method according to claim 1, where the category information is obtained according to the part detection result of the two-dimensional building image or according to user input, One or more of the building period information, the building location information, and the building cultural background information are obtained according to the information associated with the two-dimensional building image or according to user input.

5. A generating device for a three-dimensional building model of an ancient building, comprising: A first acquisition unit, configured to acquire part drawings of each part included in a two-dimensional building image, where the part is a basic unit constituting a building module, and the part drawing has attribute information of the part, and the attribute information includes one or more of category information, building period information, building location information, and building cultural background information; The two-dimensional building image refers to an image including a building; A second acquisition unit, configured to determine, according to the attribute information of the part drawing, a to-be-matched part in a three-dimensional asset library having the same type of label, where the label indicates the attribute information of a three-dimensional part model in the three-dimensional asset library; Matching the part drawing with the three-dimensional part model of the to-be-matched part includes: Matching the part drawing with multiple two-dimensional renderings of the three-dimensional part model respectively to obtain a matching result for each two-dimensional rendering, where the multiple two-dimensional renderings are two-dimensional renderings of the three-dimensional part model acquired from multiple perspectives, and at least one two-dimensional rendering is acquired from each perspective; taking the highest matching degree in the matching results as the matching degree between the part drawing and the three-dimensional part model, and determining the three-dimensional part model corresponding to the part drawing; A matching unit, configured to acquire a structure diagram corresponding to the two-dimensional building image, where the structure diagram indicates the depth information of each part and the positional relationship between the parts; A construction unit, configured to assemble the three-dimensional part models according to the structure diagram to obtain a three-dimensional building model, including: determining the positions of the three-dimensional part models corresponding to the respective parts in three-dimensional space according to the depth information of each part and the positional relationship between the parts indicated by the structure diagram, and deploying the three-dimensional part models according to the positions to construct a three-dimensional building model.

6. An electronic device, comprising: A processor; A memory for storing processor-executable instructions; Wherein, the processor realizes the method according to any one of claims 1 to 5 by running the executable instructions.

7. A computer-readable storage medium, on which computer instructions are stored, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are realized.

Images