Semantic live-action three-dimensional modeling method for urban building

Abstract

The invention is applicable to the technical field of three-dimensional modeling, and provides a semantic live-action three-dimensional modeling method for urban buildings, which comprises the following steps: constructing a three-dimensional model of each building in a city according to OSM data of the city; determining a plurality of streetscape image sampling positions based on the road network data, and collecting streetscape image data in a plurality of directions at each streetscape image sampling position; obtaining building feature description information and a text information set of the streetscape image data, and determining a building function type corresponding to the streetscape image data according to the building feature description information, the text information set and a predefined building function description dictionary; and according to the determined building function type and the street scene image sampling position, assigning semantic description information to the three-dimensional model of each building in the city to obtain a semantic real scene three-dimensional model of each building in the city. According to the method, the building three-dimensional model with semantic description information can be constructed, so that the building three-dimensional model can meet the high-precision requirement of intelligent city application.

Description

Technical Field

[0001] The present application belongs to the technical field of three-dimensional modeling, and in particular to a semantic real-scene three-dimensional modeling method for urban buildings. Background Art

[0002] With the rapid advancement of global urbanization, urban building planning, management and construction face increasingly complex challenges, especially in dynamic and complex urban environments. Refined three-dimensional models of urban buildings have become an important data foundation for supporting the development of smart cities. Urban building three-dimensional models can not only intuitively display the structure and building distribution of urban space, but also provide data support for emergency response, urban planning, navigation services, environmental monitoring, etc. The construction of traditional three-dimensional models of urban buildings usually relies on expensive equipment and time-consuming mapping processes, such as LiDAR scanning, satellite remote sensing and aerial photogrammetry, which makes the construction and updating of urban models expensive and difficult to keep up with the rapid pace of urban changes. The rise of Internet crowd-source data provides a new direction for solving this problem. Crowd-source data refers to data spontaneously provided by Internet users or devices (such as mobile terminals and Internet of Things devices). This type of data has the advantages of large scale, low cost and real-time. Specifically, crowd-source data can include geographic location information and geometric structure information of geographic entities, such as the Open Street Map (OSM) crowd-source annotation data containing building geometric outlines and other information. These rich Internet data provide an efficient and dynamic data source for the construction and updating of three-dimensional models of urban buildings, thus breaking through the limitations of traditional surveying and mapping methods.

[0003] However, the three-dimensional building models currently generated using crowd-source data lack semantic description information, resulting in the three-dimensional building models being unable to meet the high-precision requirements of smart city applications. Summary of the invention

[0004] The embodiment of the present application provides a semantic real-scene three-dimensional modeling method for urban buildings, which can solve the problem that the three-dimensional model of the building cannot meet the high-precision requirements of smart city applications.

[0005] The present application provides a semantic real-scene 3D modeling method for urban buildings, including: Build a 3D model of each building in the city based on the city’s OSM data; Determine multiple street view image sampling locations based on the city's road network data, and collect street view image data in multiple directions at each street view image sampling location; For each street view image data, respectively, building feature description information and a text information set of the street view image data are obtained, and the building function type corresponding to the street view image data is determined according to the building feature description information, the text information set and a predefined building function description dictionary; the building feature description information is used to describe the relationship between the building and other entities in the street view image data, the text information set includes text information of the building facade in the street view image data, and the building function description dictionary includes function description information corresponding to multiple building function types; According to all the determined building function types and multiple street view image sampling locations, semantic description information is assigned to the three-dimensional model of each building in the city to obtain a semantic real-scene three-dimensional model of each building in the city; the semantic description information is used to describe the attributes of the building.

[0006] Optionally, multiple street view image sampling locations are determined based on the city's road network data, including: A plurality of street view image sampling locations are set on each road in the road network data; the distance between two adjacent street view image sampling locations on the same road is a preset distance.

[0007] Optionally, street view image data in multiple directions are collected at each street view image sampling location, including: For each street view image sampling position, the azimuth of the road where the street view image sampling position is located is calculated, and street view image data in four directions of front, back, left, and right taken at the street view image sampling position is obtained from an Internet crawler based on the azimuth.

[0008] Optionally, obtain building feature description information and text information set of street view image data, including: The street view image data is analyzed and processed using a graphic-text multimodal model to obtain building feature description information of the street view image data; The OCR technology is used to perform text detection and recognition on the street view image data to obtain a text information set of the street view image data.

[0009] Optionally, determining the building function type corresponding to the street view image data according to the building feature description information, the text information set and a predefined building function description dictionary includes: The building feature description information and the text information set are fused by string concatenation to obtain a fused text description; Calculate the similarity between the fused text description and each functional description information in the building functional description dictionary; The building function type corresponding to the function description information having the highest similarity to the fused text description is used as the building function type corresponding to the street view image data.

[0010] Optionally, according to all the determined building function types and the sampling locations of the multiple street view images, semantic description information is assigned to the three-dimensional model of each building in the city, including: For each street view image sampling location, an area in the city with the street view image sampling location as the center and a preset buffer distance as the radius is used as the buffer area of ​​the street view image sampling location, and the building function types corresponding to all street view image data corresponding to the street view image sampling location are assigned as semantic description information to the three-dimensional models of all buildings in the buffer area.

[0011] Optionally, the semantic real scene 3D modeling method further includes: If there is a building in the city that is not located in any buffer area, the target buffer area closest to the building is determined from all buffer areas, and the building function type corresponding to the target buffer area is assigned to the three-dimensional model of the building as semantic description information.

[0012] Optionally, other entities include roads, trees, sidewalks, road signs, and vehicles.

[0013] Optionally, the semantic real scene 3D modeling method further includes: When the update cycle arrives, the OSM data of the city and the street view image data corresponding to each street view image sampling location are updated, and the semantic real scene 3D model of each building in the city is updated based on the updated OSM data and street view image data.

[0014] The above solution of the present application has the following beneficial effects: In an embodiment of the present application, a three-dimensional model of each building in the city is constructed based on Internet crowd-source data, and building feature description information and text information of the building facade are innovatively obtained based on street view image data. Then, the semantic description information of each building in the city is determined by combining the building feature description information and the text information of the building facade. By assigning the semantic description information to the three-dimensional model of the corresponding building, a three-dimensional model of the building with semantic description information can be obtained, so that the three-dimensional model of the building can meet the high-precision requirements of smart city applications.

[0015] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A flowchart of a semantic real-scene 3D modeling method for urban buildings provided in one embodiment of the present application; Figure 2 A schematic diagram of a three-dimensional model of various buildings in a city provided by an embodiment of the present application; Figure 3 A schematic diagram of a buffer area in a city provided in one embodiment of the present application; Figure 4 A schematic diagram of a semantic real-scene three-dimensional model of buildings in a city provided in one embodiment of the present application. DETAILED DESCRIPTION

[0018] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0019] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0020] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0021] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0022] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0024] In response to the problem that the current three-dimensional models of buildings cannot meet the high-precision requirements of smart city applications, an embodiment of the present application provides a semantic real-scene three-dimensional modeling method for urban buildings. By constructing a three-dimensional model of each building in the city based on Internet crowd-source data, innovatively obtaining building feature description information and text information of the building facade based on street view image data, and then determining the semantic description information of each building in the city in combination with the building feature description information and the text information of the building facade, and by assigning the semantic description information to the three-dimensional model of the corresponding building, a three-dimensional model of the building with semantic description information can be obtained, so that the three-dimensional model of the building can meet the high-precision requirements of smart city applications.

[0025] The semantic real-scene 3D modeling method of urban buildings provided by the present application is exemplarily described below in conjunction with specific embodiments.

[0026] like Figure 1 As shown, the semantic real scene 3D modeling method of urban buildings provided in the embodiment of the present application includes the following steps: Step 11: construct a three-dimensional model of each building in the city based on the city's OSM data.

[0027] In related technologies, Open Street Map (OSM) data is widely used to construct three-dimensional white models of urban buildings (i.e., three-dimensional models of the above-mentioned buildings). OSM data provides open source geographic data on a global scale, especially detailed building outline data, which provides a basis for the rapid generation of three-dimensional geometric models of buildings. For example, Ma (2024) et al. constructed a three-dimensional model of the city using OSM data. The construction of three-dimensional models based on OSM data has the advantages of low cost and rapid response, but because the main composition of OSM data is two-dimensional building outline information, the directly generated three-dimensional model lacks semantic description information. Among them, the construction results of the three-dimensional models of each building in the city can be as follows: Figure 2 shown.

[0028] Specifically, building a three-dimensional model of a building based on OSM data is a complex process involving data extraction, preprocessing, geometric modeling, and model optimization. As a global open source map project, OSM provides a wealth of geographic information data, including basic information such as the location, shape, and height of buildings, which provides a rich data source for the construction of three-dimensional models of buildings. When building three-dimensional models of buildings in a city, it is first necessary to extract building-related information from the city's OSM data. OSM data is usually stored in XML format, which contains a large number of elements such as tags, nodes, and ways. For the construction of three-dimensional models of buildings, the main focus is on elements with the "building" tag, which usually contains information such as the building's geometry, height, and purpose.

[0029] The data extraction process is mainly implemented by writing Python (Python is a computer programming language) scripts and using specialized OSM data processing toolkits networkx and geopandas (networkx and geopandas are two different Python libraries used for network analysis and geospatial data processing, respectively). The extracted data needs to be preprocessed to ensure its quality and consistency. The preprocessing steps include data cleaning and format conversion. The purpose of data cleaning is to remove duplicate or erroneous data entries and ensure that the identifier (ID) of each building is unique. Format conversion is to convert OSM data into a format suitable for three-dimensional modeling. The sample building data obtained by OSM is shown in Table 1.

[0030] Table 1 Attribute fields of buildings obtained from OSM

[0031] The geographic entity in the above table can be a building, the unit of the number of floors is floor, and the unit of height is meter.

[0032] Constructing a 3D model of a building is divided into the following key steps: geometry determination, height information application, and visualization. Geometry determination is to determine the basic geometry of the building based on the geometry attributes in the OSM data. The height information application is to apply the building height information provided by the "building height" tag in the OSM data to the corresponding building geometry and stretch it to form a three-dimensional building model. The 3D model of a building is determined by the geometry and height of the building and can be defined by a mathematical formula. For example, the 3D model of a rectangular building can be defined by the coordinates of its four corner points: in, For the three-dimensional model of the building, , , , are the coordinates of the four corner points of the building.

[0033] Step 12: determine multiple street view image sampling locations based on the city's road network data, and collect street view image data in multiple directions at each street view image sampling location.

[0034] In some embodiments of the present application, multiple street view image sampling locations may be set on each road in the road network data, wherein the distance between two adjacent street view image sampling locations on the same road is a preset distance, which may be set according to actual conditions, for example, 50 meters.

[0035] After determining each street view image sampling location, the azimuth of the road where the street view image sampling location is located can be calculated for each street view image sampling location, and street view image data in four directions of front, back, left, and right taken at the street view image sampling location can be obtained from an Internet crawler based on the azimuth, so as to ensure that the collected street view image data can accurately reflect the actual viewing angle of the road.

[0036] In practical applications, the process of determining the sampling location of street view images is as follows: pre-process the city's road network data (which can be OSM road network data) to generate street view image sampling locations suitable for street view image data collection. First, select the Spherical Mercator coordinate system to project the road network data, converting it from the geographic coordinate system to the projection coordinate system. Next, select the main road types, such as main roads, secondary roads, and residential roads, while removing irrelevant categories such as trails and bicycle lanes, splitting multi-component features, interrupting all intersecting roads, and simplifying roads to reduce unnecessary complexity.

[0037] The generation of street view image sampling locations is completed by the Generate Points Along Line tool provided by the Arcpy package in Python (Arcpy package is a tool package in Python), and the sampling interval is 50 meters. The latitude and longitude coordinates of the sampling points are added to the point attribute table through the Add Coordinates tool for subsequent processing. Finally, a CSV file containing street view image sampling location information is obtained.

[0038] Use a CSV file containing street view image sampling location information (which can be X, Y coordinates) and use Python crawler technology to automatically collect street view image data from the Internet. The crawler program can send a request through the Baidu Street View Application Programming Interface (API) to obtain the street view image of the specified street view image sampling location. Collect street view image data in four directions, front, back, left and right, for each street view image sampling location. Specifically, for each street view image sampling location, calculate the azimuth of the road, and adjust the horizontal heading parameter in the street view API request based on the angle to ensure that the collected street view image data can accurately reflect the actual perspective of the road. If it is assumed is the set of sampling locations of street view images, is a road set, then the street view image sampling location With Road The relationship can be expressed as: in, is the sampling function, and interval is the sampling interval. For each street view image sampling position , street view image data from four directions are collected, which can be expressed as: in, is the sampling location in the street view image A collection of street view image data collected in four directions. , , and Represent the sampling locations of street view images Street view image data in front, behind, left and right of the camera.

[0039] It should be noted that street view image data is a kind of multi-source visual information data with a wide range of sources and frequent updates. Compared with remote sensing data and aerial images, street view image data has the advantages of low cost, high detail and real-time update. It is an important data source for obtaining building appearance details and extracting semantic information. Street view image data can capture the facade information of buildings, including house numbers, advertising signs, etc., to supplement the semantic description of the three-dimensional model of the building generated by OSM data. In addition, street view image data also provides rich environmental context information, which helps to improve the refinement of building models and the accuracy of scene understanding.

[0040] Specifically, street view image data can reflect the relative relationship between buildings and surrounding entities and their spatial arrangement characteristics. In addition to text information, these features include the distribution of buildings, their relationship with the surrounding environment, etc., which can provide important references for the semantic classification of buildings. For example, in residential areas, buildings are usually arranged compactly and at similar heights, forming a neat streetscape layout; around educational buildings such as schools, there may be obvious visual elements such as playgrounds and students with backpacks, which constitute the specific "scene" characteristics of the area, thereby helping to determine the functional purpose of the building.

[0041] It is understandable that street view image data and OSM data, as crowd-source data, can be obtained through mobile devices, social networks, IoT sensors and other channels. To facilitate data processing, after obtaining these data, they will be formatted, denoised and normalized to ensure the validity and consistency of the data, providing a solid foundation for the geometric and semantic construction of urban building models.

[0042] Step 13, for each street view image data, obtain the building feature description information and text information set of the street view image data, and determine the building function type corresponding to the street view image data according to the building feature description information, the text information set and a predefined building function description dictionary.

[0043] The above-mentioned building feature description information is used to describe the relationship between the building and other entities in the street view image data, and the other entities include roads, trees, sidewalks, road signs and vehicles, and the relationship is a spatial or functional relationship. Exemplarily, the building feature description information may be: there are multiple modern buildings beside the road, the buildings have many signs, there are multiple cars on the roadside, and pedestrians are dense.

[0044] In some embodiments of the present application, a graphic-text multimodal model (CLIP, Contrastive Language-Image Pre-Training) can be used to analyze and process street view image data to obtain building feature description information of the street view image data. Specifically, the graphic-text multimodal model deeply analyzes the street view image data, fully understands the semantic information of the building and its surrounding environment, and outputs the building feature description information of the street view image data.

[0045] The above-mentioned multimodal model includes an image encoder and text encoder Before using the image-text multimodal model to analyze and process street view image data, it is necessary to extract image-text pairs from a large-scale dataset collected on the Internet, denoted as ,in Represents an image, Represents the text description related to it. Using the principle of contrastive learning, a dual encoder model, namely the image encoder, is trained and text encoder , to learn to extract rich semantic features from data pairs.

[0046] The image encoder and text encoder minimize the contrast loss function Joint training is performed, which measures the positive sample pair With negative samples The distance difference between Represents negative sample text. Specifically, for each image-text pair, the image feature vector is calculated and text feature vector , and their matching degree is measured by cosine similarity. The contrast loss function can be expressed as: in, represents the cosine similarity function, is the temperature parameter used to control the smoothness of the probability distribution, B is the set of all image-text pairs in the batch, Negative sample text The feature vector of .

[0047] In the test phase, given a target image , using the trained image encoder Extract its feature vector Then, by calculating With all candidate text feature vectors The cosine similarity between them is used to retrieve the best matching text description. Finally, the overall semantic perception of the built environment can be expressed as: in, is predicted with The most relevant text description.

[0048] The above-mentioned text information set includes text information on the facades of buildings in the street view image data, that is, text information such as billboards, signs, house numbers, etc. in the street view image data. The text information such as billboards, signs, house numbers, etc. usually directly indicates the function or purpose of the building. The text information may include signs, names, functional descriptions, etc. (such as "bank", "school", "supermarket", etc.), providing direct clues about the purpose and characteristics of the building.

[0049] In some embodiments of the present application, optical character recognition (OCR) technology may be used to perform text detection and recognition on street view image data to obtain a text information set of the street view image data.

[0050] In practical applications, image processing and natural language processing techniques can be combined to identify the functional types of buildings in street view image data. OCR technology is used to detect and recognize text in street view image data to obtain a text information set. ,in Indicates the detected A text string, such as bank, school, etc. , The number of text strings.

[0051] The building function description dictionary includes function description information corresponding to various building function types, wherein the function description information is used to describe the function of the building, and the various building function types include commercial services, corporate enterprises, public services, residential service function types, etc.

[0052] In some embodiments of the present application, the specific implementation method of determining the building function type corresponding to the street view image data according to the building feature description information, the text information set and the predefined building function description dictionary is as follows: Step 13.1, the building feature description information and the text information set are fused by string concatenation to obtain a fused text description. The fused text description D' can be expressed as: , For fusion operation, Describes the building characteristics. A collection of text information.

[0053] Step 13.2, calculate the similarity between the fused text description and each functional description information in the building functional description dictionary.

[0054] Dictionary of functional descriptions of the above buildings It can be recorded as , Represents the functional description information corresponding to the j-th building function type in the dictionary, , is the number of building function types. In some embodiments of the present application, the fused text description can be calculated by the cosine similarity measurement method. and Similarity : Here, · represents the vector dot product, and ||·|| represents the Euclidean norm of the vector.

[0055] Step 13.3, taking the building function type corresponding to the function description information having the highest similarity to the fused text description as the building function type corresponding to the street view image data.

[0056] Exemplarily, for certain street view image data, the image-text multimodal model is used to analyze and process it, and the building feature description information obtained is: this building is a small company. The OCR technology is used to detect and recognize text, and the text information set obtained is: a certain limited company, a certain dealer. "This building is a small company" and "a certain limited company, a certain dealer" are fused, and then the similarity between the fused text description and each functional description information in the building functional description dictionary is calculated. Finally, it is determined that the building function type corresponding to the functional description information with the highest similarity is a company enterprise, and the company enterprise is used as the building function type of the street view image data.

[0057] It is worth mentioning that the key text information in the street view image data of buildings provides a key basis for inferring the functions of buildings (for example, text information such as billboards, signs, house numbers, etc. in street view image data usually directly indicates the function or purpose of the building (such as "bank", "school", "supermarket", etc.)), which can effectively improve the efficiency and accuracy of building function inference. The process first involves in-depth analysis of street view image data, using the open source CLIP model to generate an overall semantic description of the building, and at the same time, extracting text information from the same street view image through OCR technology. Finally, the description and text information are compared with the pre-defined building type text description template to infer the semantic function of the building.

[0058] Step 14, assigning semantic description information to the three-dimensional model of each building in the city according to all the determined building function types and multiple street view image sampling locations, and obtaining a semantic real-scene three-dimensional model of each building in the city.

[0059] The above semantic description information is used to describe the properties of buildings, such as shopping malls, banks, companies, etc. The above semantic real-scene 3D model refers to: based on the traditional urban building 3D geometric model (i.e. the above 3D model built based on OSM data), combined with the building attribute information from multi-source data, the model is added with semantic information, so that it can not only express the geometric structure of the building, but also reflect the function, type and other attributes of the building. The semantic real-scene 3D model has a wide range of application scenarios in smart city construction, such as traffic management, building energy consumption analysis, disaster emergency response, urban planning and simulation, etc.

[0060] In some embodiments of the present application, the above-mentioned step 14, according to all the determined building function types and multiple street view image sampling locations, a specific implementation method of assigning semantic description information to the three-dimensional model of each building in the city can be: for each street view image sampling location, respectively, an area in the city with the street view image sampling location as the center and a preset buffer distance (which can be set according to actual conditions, for example, set to 50 meters) as a radius is used as a buffer area of ​​the street view image sampling location, and the building function type corresponding to all street view image data corresponding to the street view image sampling location is assigned as semantic description information to the three-dimensional models of all buildings in the buffer area.

[0061] Of course, if there are buildings in the city that are not located in any buffer area, the target buffer area closest to the building will be determined from all buffer areas, and the building function type corresponding to the target buffer area will be assigned to the three-dimensional model of the building as semantic description information.

[0062] Through the buffer area method, the functional information of buildings extracted from street view image data can be effectively associated with predefined functional types, thereby realizing the automatic construction of semantic real scene 3D models. Among them, the results of the buffer area in the city can be as follows Figure 3 As shown, Figure 3 The OSM buildings in the figure are the 3D models constructed based on OSM data in the previous article. The semantic real scene 3D models of the buildings in the city can be shown as follows: Figure 4 shown.

[0063] In some embodiments of the present application, the real-time collection characteristics of street view image data can be used to dynamically update the model according to the changes in the appearance of urban buildings. For example, the replacement of shops, changes in building facade information or conversion of functions can be detected and identified through new street view image data, thereby updating the semantic description information in the model. Combining OCR and image recognition algorithms ensures the geometric update of the model and dynamic adjustment at the semantic level.

[0064] Specifically, when the update cycle arrives, the OSM data of the city and the street view image data corresponding to each street view image sampling position can be updated, and the semantic real scene 3D model of each building in the city can be updated based on the updated OSM data and street view image data. That is, when the update cycle arrives, it is equivalent to re-executing the above steps 11, 13 and 14 according to the updated OSM data and street view image data to obtain the updated semantic real scene 3D model.

[0065] It is worth mentioning that the semantic real-scene 3D modeling method of the present application constructs a 3D model of each building in the city based on Internet crowd-source data, and innovatively obtains building feature description information and text information of the building facade based on street view image data, and then determines the semantic description information of each building in the city in combination with the building feature description information and the text information of the building facade, and by assigning the semantic description information to the 3D model of the corresponding building, a 3D model of the building with semantic description information can be obtained, so that the 3D model of the building can meet the requirements of the functional attributes of the building in smart city applications, so that the model can more accurately reflect the actual use of the building, and enhance the usability and reference value of the model.

[0066] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A semantic real scene 3D modeling method for urban buildings, characterized in that: include: Constructing a three-dimensional model of each building in the city based on the OSM data of the city; Determine a plurality of street view image sampling locations based on the road network data of the city, and collect street view image data in multiple directions at each street view image sampling location; For each of the street view image data, respectively, obtain building feature description information and a text information set of the street view image data, and determine the building function type corresponding to the street view image data according to the building feature description information, the text information set and a predefined building function description dictionary; the building feature description information is used to describe the relationship between the building and other entities in the street view image data, the text information set includes text information of the building facade in the street view image data, and the building function description dictionary includes function description information corresponding to multiple building function types; According to all the determined building function types and multiple street view image sampling locations, semantic description information is assigned to the three-dimensional model of each building in the city to obtain a semantic real-scene three-dimensional model of each building in the city; the semantic description information is used to describe the attributes of the building.

2. The semantic real scene 3D modeling method according to claim 1, characterized in that: The determining of a plurality of street view image sampling locations based on the road network data of the city comprises: A plurality of street view image sampling positions are set on each road in the road network data; and a distance between two adjacent street view image sampling positions on the same road is a preset distance.

3. The semantic real scene 3D modeling method according to claim 2, characterized in that: The collecting of street view image data in multiple directions at each street view image sampling position includes: For each of the street view image sampling positions, the azimuth of the road where the street view image sampling position is located is calculated, and based on the azimuth, street view image data in four directions of front, back, left, and right taken at the street view image sampling position is obtained from an Internet crawler.

4. The semantic real scene 3D modeling method according to claim 1, characterized in that: The step of obtaining the building feature description information and text information set of the street view image data includes: Analyzing and processing the street view image data using a graphic-text multimodal model to obtain building feature description information of the street view image data; The OCR technology is used to perform text detection and recognition on the street view image data to obtain a text information set of the street view image data.

5. The semantic real scene 3D modeling method according to claim 1, characterized in that: The determining the building function type corresponding to the street view image data according to the building feature description information, the text information set and a predefined building function description dictionary includes: The building feature description information and the text information set are merged by string concatenation to obtain a merged text description; Calculating the similarity between the fused text description and each functional description information in the building functional description dictionary; The building function type corresponding to the function description information having the highest similarity to the fused text description is used as the building function type corresponding to the street view image data.

6. The semantic real scene 3D modeling method according to claim 1, characterized in that: The step of assigning semantic description information to the three-dimensional model of each building in the city according to the determined functional types of all buildings and the sampling locations of the plurality of street view images includes: For each street view image sampling position, an area in the city with the street view image sampling position as the center and a preset buffer distance as the radius is used as a buffer area of ​​the street view image sampling position, and building function types corresponding to all street view image data corresponding to the street view image sampling position are assigned as semantic description information to the three-dimensional models of all buildings in the buffer area.

7. The semantic real scene 3D modeling method according to claim 6, characterized in that: The semantic real scene 3D modeling method also includes: If there is a building in the city that is not located in any buffer area, the target buffer area closest to the building is determined from all buffer areas, and the building function type corresponding to the target buffer area is assigned to the three-dimensional model of the building as semantic description information.

8. The semantic real scene 3D modeling method according to claim 1, characterized in that: The other entities include roads, trees, sidewalks, road signs, and vehicles.

9. The semantic real scene 3D modeling method according to claim 1, characterized in that: The semantic real scene 3D modeling method also includes: When the update cycle arrives, the OSM data of the city and the street view image data corresponding to each street view image sampling position are updated, and the semantic real scene three-dimensional model of each building in the city is updated based on the updated OSM data and street view image data.

Images