A building classification method based on street view image multi-modal information

Abstract

The application discloses a kind of building classification methods based on street view image multi-modal information, the method is based on SVI global visual information, local visual information, sign text and various modal information, adopts graph neural network to realize multi-modal information reasoning, realizes city building single function automatic classification.This method first obtains text information through sign text knowledge graph, then carries out multi-modal reasoning to text information and local vision, finally realizes building function classification by fusing global visual information.The present application overcomes the problem of insufficient utilization of SVI information in the prior art by constructing sign text knowledge graph and multi-modal reasoning graph and fusing SVI multi-modal information.The sufficient utilization and effective fusion of SVI multi-modal information effectively improve the accuracy of building classification.

Description

Technical Field

[0001] This invention relates to the field of building classification, and more particularly to a building classification method based on multimodal information from street view images. Background Technology

[0002] Buildings are the foundation for citizens' lives, work, and recreation, and are a fundamental component of urban planning, urban management, and risk assessment. Identifying and recording building types is of great reference value for disaster management, improving urban functions, raising urban living standards, better formulating urban construction plans, and promoting coordinated and sustainable urban development.

[0003] Traditional building classification methods rely heavily on field surveys, which are time-consuming, labor-intensive, and inefficient in updating information. With the development of deep learning technology, building classification methods based on remote sensing imagery have been proposed. However, due to the homogeneity of building shapes and textures, they cannot effectively distinguish building types. Compared to remote sensing images, street view images not only have a wide distribution but also contain fine-grained built-up environment information at the street scale, providing socio-economic attributes related to function. Existing urban function classification methods based on street view images mostly rely solely on scene features, resulting in low classification accuracy and an inability to distinguish similar types.

[0004] In addition to scene information, Street View Images (SVIs) also include text information provided by signage. Effectively fusing multimodal information from Street View Images to classify buildings by function helps improve the accuracy of automatic classification of urban building functions. Summary of the Invention

[0005] To address the shortcomings of existing building classification methods that suffer from low accuracy and incomplete information extraction from building images, this invention proposes a building classification method based on multimodal information from street view images. The method specifically includes the following steps:

[0006] S1: Obtain global visual features of SVI, local visual features of SVI, visual position data, sign text information, text position data, and building category text features;

[0007] S2: Classify the signboard text information to obtain signboard concept text data; input the signboard concept text data into a pre-trained natural language model to extract the concept text features of the signboard concept text data; input the building category text features and concept text features into a knowledge graph convolutional neural network to obtain SVI text features;

[0008] S3: Input the visual position data and text position data into a fully connected layer network to obtain visual position feature vectors and text position feature vectors;

[0009] S4: Input the SVI local visual features, visual position feature vector, SVI text features and text position feature vector into the multimodal reasoning graph convolutional neural network to be trained, and perform reasoning fusion on multiple graph convolutional layers to obtain multimodal reasoning features;

[0010] S5: After combining the multimodal reasoning features and the SVI global visual features, the results are output to the classifier for classification to obtain the building classification results.

[0011] The beneficial effects provided by this invention are: by constructing a signboard text knowledge graph and a multimodal reasoning graph, and integrating SVI multimodal information, the problem of insufficient utilization of the information contained in SVI in existing technologies is overcome. The full utilization and effective integration of SVI multimodal information effectively improves the accuracy of building classification. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0013] Figure 2 These are the overall implementation steps of the method. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0015] Please refer to Figure 1 , Figure 1 This is a schematic diagram of the method flow of the present invention.

[0016] This invention provides a building classification method based on multimodal information from street view images, the method comprising the following steps:

[0017] S1: Obtain global visual features of SVI, local visual features of SVI, visual position data, sign text information, text position data, and building category text features;

[0018] It should be noted that step S1 specifically involves:

[0019] S11: Input SVI into the pre-trained ResNet 152 network to obtain the global visual feature vector;

[0020] S12: Input SVI into the pre-trained Faster R-CNN network to obtain local visual features and local visual location data;

[0021] S13: Input SVI into the PaddleOCR basic model to extract the signboard text and obtain the signboard text data and signboard text location data;

[0022] S14: Input different building categories into the pre-trained Fasttext base model to obtain text features for each building category.

[0023] As one example, the ResNet 152 network was pre-trained on the ImageNet dataset; the Faster R-CNN network was pre-trained on the Visual Genome dataset.

[0024] S2: Classify the signboard text information to obtain signboard concept text data; input the signboard concept text data into a pre-trained natural language model to extract the concept text features of the signboard concept text data; input the building category text features and concept text features into a knowledge graph convolutional neural network to obtain SVI text features;

[0025] It should be noted that step S2 specifically involves:

[0026] S21: Based on expert prior knowledge, classify the signboard text data into signboard concepts to obtain signboard concept text data; input the signboard concept text data into the pre-trained Fasttext basic model to map it into the concept text features of the signboard concept text data;

[0027] As one example, the conceptual text features are 300-dimensional feature vectors.

[0028] S22: For each SVI, construct a knowledge graph for the concept text features and building category text features.

[0029] The knowledge graph The node v includes two types: building category node v b And the signboard concept node v t ; where the building category node v b Representing the SVI building category, a signboard concept node v t This represents a signboard text within an SVI (Signature VI).

[0030] The edges e in the graph G(V, e) include two types: one is the edge e between a building category node and a signboard concept node. ct One type is the edge e between signboard concept nodes. tt ;

[0031] S23: Transfer the knowledge graph The data is fed into a graph convolutional neural network, and after knowledge fusion, the SVI text features are obtained.

[0032] Step S23 is as follows:

[0033] For the knowledge graph in Represents a set of nodes. The relation matrix is ​​represented as follows:

[0034]

[0035] Where k i k j ∈V, Representing two fully connected layers, the knowledge graph... The input graph is a convolutional neural network, and the operation of a single convolutional layer is as follows:

[0036]

[0037] Where R is the correlation matrix, V is the node, and W is the node. g W is the learnable weight matrix in a graph convolutional neural network. r Let l be the residual weight matrix, and l be the number of layers in the graph convolutional neural network.

[0038] S3: Input the visual position data and text position data into a fully connected layer network to obtain visual position feature vectors and text position feature vectors;

[0039] It should be noted that step S3 specifically involves:

[0040] S31: Input the visual position data into a fully connected layer network and map it into a visual position feature vector;

[0041] As one embodiment, the visual position feature vector is a 128-dimensional feature vector;

[0042] S32: Input the text location data into a fully connected layer network and map it into a text location feature vector;

[0043] As one embodiment, the text position feature vector is a 128-dimensional feature vector;

[0044] S4: Input the SVI local visual features, visual position feature vector, SVI text features and text position feature vector into the multimodal reasoning graph convolutional neural network to be trained, and perform reasoning fusion on multiple graph convolutional layers to obtain multimodal reasoning features;

[0045] It should be noted that step S4 specifically involves:

[0046] S41: Input the SVI local visual features into the fully connected layer and map them to obtain a first local visual feature vector; connect the first local visual feature vector with the visual position feature vector to obtain a second local visual feature vector;

[0047] As one embodiment, the first local visual feature vector is a 1920-dimensional feature vector; the second local visual feature vector is a 2048-dimensional feature vector.

[0048] The SVI text features are input into a fully connected layer and mapped to obtain a first text feature vector. The first text feature vector is then connected to the text position feature vector to obtain a second text feature vector.

[0049] As one embodiment, the first text feature vector is a 1920-dimensional feature vector; the second text feature vector is a 2048-dimensional feature vector.

[0050] S42: For each SVI, construct a multimodal inference graph for the concept text features and building category text features.

[0051] S43: The inference diagram The data is fed into a graph convolutional neural network, and after multimodal feature fusion, the multimodal inference features are obtained.

[0052] S5: After combining the multimodal reasoning features and the SVI global visual features, the results are output to the classifier for classification to obtain the building classification results.

[0053] In this invention, the building classification results include: commercial, residential, industrial, retail, and office.

[0054] As one example, Figure 2 The overall implementation steps of the method are outlined below, including visual feature acquisition; text feature acquisition; multimodal inference; model optimization; and building type prediction. Table 1 presents the experimental results of the method. As shown in Table 1, a total of 799 street view images were analyzed, of which 407, 50, 44, 5, and 137 were correctly predicted as residential, retail, office, industrial, and commercial, respectively, accounting for 80.47% of the total street view images.

[0055] Table 1 Experimental Results

[0056]

[0057] The beneficial effects of this invention are: by constructing a signboard text knowledge graph and a multimodal reasoning graph, and integrating SVI multimodal information, it overcomes the problem of insufficient utilization of the information contained in SVI in existing technologies. The full utilization and effective integration of SVI multimodal information effectively improves the accuracy of building classification.

[0058] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A building classification method based on multimodal information from street view images, characterized in that: Includes the following steps: S1: Obtain global visual features of SVI, local visual features of SVI, visual position data, sign text information, text position data, and building category text features; Step S1 is as follows: S11: Input SVI into the pre-trained ResNet 152 network to obtain the global visual feature vector; S12: Input SVI into the pre-trained Faster R-CNN network to obtain local visual features and local visual location data; S13: Input SVI into the PaddleOCR basic model to extract the signboard text and obtain the signboard text data and signboard text location data; S14: Input different building categories into the pre-trained Fasttext base model to obtain text features for each building category; S2: Classify the signboard text information to obtain signboard concept text data; input the signboard concept text data into a pre-trained natural language model to extract the concept text features of the signboard concept text data; input the building category text features and concept text features into a knowledge graph convolutional neural network to obtain SVI text features; S3: Input the visual position data and text position data into a fully connected layer network to obtain visual position feature vectors and text position feature vectors; S4: Input the SVI local visual features, visual position feature vector, SVI text features and text position feature vector into the multimodal reasoning graph convolutional neural network to be trained, and perform reasoning fusion on multiple graph convolutional layers to obtain multimodal reasoning features; S5: After combining the multimodal reasoning features and the SVI global visual features, the results are output to the classifier for classification to obtain the building classification results.

2. The building classification method based on multimodal information of street view images as described in claim 1, characterized in that: Step S2 is as follows: S21: Based on expert prior knowledge, classify the signboard text data into signboard concepts to obtain signboard concept text data; input the signboard concept text data into the pre-trained Fasttext basic model to map it into the concept text features of the signboard concept text data; S22: For each SVI, construct a knowledge graph for the concept text features and building category text features. S23: Transfer the knowledge graph The data is fed into a graph convolutional neural network, and after knowledge fusion, the SVI text features are obtained.

3. The building classification method based on multimodal information of street view images as described in claim 2, characterized in that: The knowledge graph The node v includes two types: building category node v b And the signboard concept node v t ; where the building category node v b Representing the SVI building category, a signboard concept node v t This represents a signboard text within an SVI (Signature VI). The diagram Edge e includes two types: one is the edge e between building category nodes and signboard concept nodes. ct One type is the edge e between signboard concept nodes. tt .

4. The building classification method based on multimodal information of street view images as described in claim 2, characterized in that: The knowledge graph It is an undirected graph.

5. The building classification method based on multimodal information of street view images as described in claim 3, characterized in that: Step S23 is as follows: For the knowledge graph in Represents a set of nodes. The relation matrix is ​​represented as follows: Where k i ,k j ∈V, Representing two fully connected layers, the knowledge graph... The input graph is a convolutional neural network, and the operation of a single convolutional layer is as follows: Where R is the correlation matrix, V is the node, and W is the node. g W is the learnable weight matrix in a graph convolutional neural network. r Let l be the residual weight matrix, and l be the number of layers in the graph convolutional neural network.

6. The building classification method based on multimodal information of street view images as described in claim 1, characterized in that: Step S3 is as follows: S31: Input the visual position data into a fully connected layer network and map it into a visual position feature vector; S32: Input the text location data into a fully connected layer network and map it into a text location feature vector.

7. The building classification method based on multimodal information of street view images as described in claim 6, characterized in that: Step S4 is as follows: S41: Input the SVI local visual features into the fully connected layer and map them to obtain a first local visual feature vector; connect the first local visual feature vector with the visual position feature vector to obtain a second local visual feature vector; The SVI text features are input into a fully connected layer and mapped to obtain a first text feature vector. The first text feature vector is then connected to the text position feature vector to obtain a second text feature vector. S42: For each SVI, construct a multimodal inference graph for the concept text features and building category text features. S43: The inference diagram The data is fed into a graph convolutional neural network, and after multimodal feature fusion, the multimodal inference features are obtained.

8. The building classification method based on multimodal information of street view images as described in claim 7, characterized in that: The reasoning diagram The node v includes two types: local visual nodes V v and text node V t One of the local visual nodes V v A text node V represents the visual features and positional features contained within a border of an SVI. t The inference graph represents the text features contained within a border in an SVI and the positional features of the border; The edge e includes three types: one is the edge e between a local visual node and a text node. vt One type is the edge e between local visual nodes. vv One type is the edge e between text nodes. tt ; The reasoning diagram It is an undirected graph.

9. The building classification method based on multimodal information of street view images as described in claim 1, characterized in that: Step S5 is as follows: S51: After performing average pooling on the multimodal inference features, they are combined with the SVI global visual features through a fully connected layer to obtain a multimodal feature vector; S52: Output the multimodal feature vector to the classifier to obtain the building classification result.

Images