Urban functional area recognition method and system based on street view image target relationship perception network

Abstract

The present invention provides a method and system for identifying urban functional areas based on a street view image target relationship perception network, comprising: obtaining a street view image, and performing random horizontal flipping, random geometric transformation and random color jittering operations on the street view image to enhance training data; constructing a training data set according to the training data, inputting the street view image of the training set into a graph-based relationship perception network for training, and obtaining a trained urban functional area recognition model; performing spatial relationship perception on the street view image, and identifying the street view image according to the obtained urban functional area type; obtaining POI data within the scope of a study area, classifying the POI data, and obtaining an urban functional area classification based on the POI data; being able to simulate human recognition and reasoning processes, capture local spatial relationships and global semantic relationships in street view images, and enhance the capabilities of an urban functional area detector.

Description

Technical Field

[0001] The present invention relates to the field of urban functional area recognition, and in particular to an urban functional area recognition method and system based on a street view image target relationship perception network. Background Art

[0002] At present, the research on urban functional area recognition based on high-resolution remote sensing images mainly uses the low-level visual features of the objects, which cannot accurately identify different types of objects with similar textures and grayscales. For example, the spectral information from high-resolution remote sensing images is not enough to distinguish the same (land cover) material into different building categories. For example, a concrete building may belong to a school, a town hall, or a hotel. In addition, urban functional areas are usually composed of a series of objects with different materials and textures, and are enclosed within a certain boundary. For example, a hospital may contain parks, roads, and buildings within its boundaries. It is much more difficult to determine the functional area category of complex urban areas from a bird's-eye view, because we cannot determine from remote sensing images whether a building is a restaurant or a barber shop. Using remote sensing images for recognition only from a bird's-eye view lacks important building detail information, and it is difficult to obtain the correct building type. Therefore, it is difficult to use remote sensing images to identify the types of urban functional areas with very obvious economic and social characteristics.

[0003] With the rise of street view image services such as Google Street View (GSV), more and more street view images with accurate geolocation of different cities can be obtained. Visual models that perform well in common computer vision tasks, such as CNN, have recently been widely used to extract visual features from street view images for urban land use and urban functional area analysis. However, existing urban functional area classification based on street view images is keen on focusing on the overall visual features of the image, while ignoring the relationship between local visual features in the image. Therefore, when the entire image is used directly, the features of many identical visual elements (such as the sky and the ground) they contain will interfere with the distinction between different functional areas, while the most distinguishable visual elements (such as buildings) are not fully utilized. A common shortcoming of these methods is that they are essentially limited to ignoring the complex local spatial relationships between objects in the image. Ignoring the complex global semantic relationships between objects in the image is also one of the limitations of these methods. Summary of the invention

[0004] The present invention provides a method and system for identifying urban functional areas based on a street view image target relationship perception network, the purpose of which is to solve the problem of urban functional area type identification.

[0005] In order to achieve the above object, the present invention provides a method for identifying urban functional areas based on a street view image object relationship perception network, comprising:

[0006] Step 1, obtaining a street view image, and performing random horizontal flipping, random geometric transformation, and random color jittering operations on the street view image to enhance training data;

[0007] Step 2, constructing a training data set based on the training data, inputting the street view images of the training set into a graph-based relationship perception network for training, and obtaining a trained urban functional area recognition model;

[0008] Step 3, performing spatial relationship perception on the street view image, and identifying the street view image according to the obtained urban functional area type;

[0009] Step 4: Acquire POI data within the study area, classify the POI data, and obtain the urban functional area classification based on the POI data.

[0010] Wherein, the step 2 specifically includes:

[0011] 75% of the samples in each category of the street view images are randomly selected as training / validation sets, and the remaining 25% are used as test sets. The training / validation sets are randomly divided into training sets and validation sets in a ratio of 9:1. The street view images in the training set are input into a graph-based relational perception network for training to obtain a trained urban functional area recognition model.

[0012] The step of perceiving the spatial relationship of the street view image specifically includes:

[0013] Input the street view image into the pre-trained baseline model Faster R-CNN network to obtain the target box and classification weight of the building in the street view image;

[0014] The obtained classification weights are used to visually embed the building objects, and the paired spatial information is captured using the paired pseudo-coordinate function u(a,b), from which the Gaussian kernel WK with a learnable mean and covariance K is determined as the weight of the graph convolutional network;

[0015] The regional visual embedding is used as a node and the Gaussian kernel WK is used as a weight and input into the graph convolutional network to perceive the spatial relationship of street view images.

[0016] The step of identifying the street view image according to the obtained urban functional area type specifically includes:

[0017] Perform semantic relationship perception on street view images, embed the labels of street view images into words, and obtain the word embedding vector of the labels;

[0018] A normalized adjacency matrix of the graph convolutional network is obtained by calculating the number of collinearities of the labels in the dataset;

[0019] Input the word embedding vector and adjacency matrix into the graph convolutional neural network to obtain the graph embedding vector of the image label;

[0020] The semantic relationship features of the street view image are obtained by performing a dot multiplication between the image embedding vector and the visual feature vector obtained by the pre-trained baseline model Faster R-CNN.

[0021] Connecting the obtained spatial relationship features of the street view image with the semantic relationship features and the overall visual relationship features of the street view image;

[0022] The connected vector is input into the fully connected network, and finally softmax classification is performed to obtain the type of urban functional area.

[0023] Wherein, the step 4 specifically includes:

[0024] Obtain POI data within the study area from the OSM official website through crawling;

[0025] According to the urban functional area type, POI data is divided into residential functional area, commercial functional area, industrial functional area and public service functional area, which corresponds to the functional area classification of street view images;

[0026] By performing key data processing on POI data, the urban functional area type of POI is defined;

[0027] Obtain the corresponding road network data in the study area and cut the data according to the scope of the study area;

[0028] The urban functional area classification based on street view images obtained by the relationship perception network and the functional area mapping obtained by POI point data are combined, and the two-dimensional kernel density calculation is performed on the four types of functional area data to obtain the kernel density distribution of each functional area category;

[0029] According to the obtained kernel density distribution of each functional area category, the type of urban functional area in each area is determined by setting the kernel density threshold in each area.

[0030] The present invention also provides an urban functional area recognition system based on a street view image target relationship perception network, comprising:

[0031] A data acquisition module, used to acquire street view images, and perform random horizontal flipping, random geometric transformation and random color jittering operations on the street view images to enhance training data;

[0032] A model training module, used to construct a training data set according to the training data, input the street view images of the training set into a graph-based relationship perception network for training, and obtain a trained urban functional area recognition model;

[0033] An image recognition module, used to perceive the spatial relationship of the street view image and recognize the street view image according to the obtained urban functional area type;

[0034] The functional area classification module is used to obtain POI data within the study area, classify the POI data, and obtain the urban functional area classification based on the POI data.

[0035] The above scheme of the present invention has the following beneficial effects:

[0036] The urban functional area recognition method and system based on street view image target relationship perception network of the present invention is based on a graph-based relationship perception network framework, which can be used to capture local spatial relationships and global semantic relationships in street view images, and can obtain more building detail features, thereby obtaining more accurate building categories. Compared with the street view image urban functional area recognition method based on image-level CNN, it can simulate the human recognition and reasoning process, capture the local spatial relationships and global semantic relationships in street view images, and enhance the capabilities of the urban functional area detector.

[0037] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the method for identifying urban functional areas based on a street view image object relationship perception network of the present invention.

[0039] Figure 2 Visualization results of building detection.

[0040] Figure 3 Predict the proportion of building types and functional areas of buildings for Calgary.

[0041] Figure 4 It is a classification map of urban functional areas that combines POI data and street view data. DETAILED DESCRIPTION

[0042] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0044] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0045] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0046] like Figure 1 As shown, an embodiment of the present invention provides a method for identifying urban functional areas based on a street view image object relationship perception network, comprising:

[0047] Step 1: Preprocessing of street view images: mainly performing random horizontal flipping, random geometric transformation and random color jittering on street view images to enhance training data.

[0048] Step 2, build training data set and model training: randomly select 75% of the samples from each category of street view images as training / validation sets, and the remaining 25% as test sets. Then randomly divide the training / validation sets into training sets and validation sets in a ratio of 9:1. Input the street view images of the training set into the graph-based relational perception network for training, and obtain the trained urban functional area recognition model.

[0049] Step 3: The main steps of functional area recognition for street view images are as follows:

[0050] To perceive the spatial relationship of street view images, first, the street view images are input into the pre-trained baseline model Faster R-CNN network to obtain the target boxes and classification weights of the buildings in the street view images.

[0051] The obtained classification weights are used to perform visual embedding of building objects.

[0052] The paired spatial information is captured using a pairwise pseudo-coordinate function u(a,b), from which a Gaussian kernel WK with a learnable mean and covariance K is determined as the weight of the graph convolutional network.

[0053] The regional visual embedding is used as a node and the Gaussian kernel WK is used as a weight and input into the graph convolutional network to perceive the spatial relationship of street view images.

[0054] To perceive the semantic relationship of street view images, we first embed the labels of street view images and obtain the word embedding vectors of the labels.

[0055] A normalized adjacency matrix of the graph convolutional network is obtained by calculating the number of collinearities of the labels in the dataset.

[0056] The word embedding vector and adjacency matrix are input into the graph convolutional neural network to obtain the graph embedding vector of the image label.

[0057] The semantic relationship features of the street view image are obtained by performing a dot multiplication between the graph embedding vector and the visual feature vector obtained by the pre-trained baseline model Faster R-CNN.

[0058] The obtained spatial relationship features of the street view image are connected with the semantic relationship features and the overall visual relationship features of the street view image.

[0059] The connected vector is input into the fully connected network, and finally softmax classification is performed to obtain the type of urban functional area.

[0060] Step 4: The main steps for obtaining the urban functional area classification based on POI data are as follows:

[0061] The POI data within the study area was obtained from the OSM official website by crawling.

[0062] Taking into account the general awareness and functional significance of urban functions, the POI data are also divided into residential functional areas, commercial functional areas, industrial functional areas and public service functional areas, similar to the functional area classification of street view images, and directly used as urban functional area types.

[0063] By extracting, cleaning, checking for duplicates, transforming coordinates, defining projections and spatial positions and other key data processing, the urban functional area types of POIs and the relative quantity calculations are completed.

[0064] Download the corresponding road network data in the study area from the OpenStreetMap official website and crop the data according to the scope of the study area.

[0065] According to the POI sample points of the present invention, they are expanded in a two-dimensional plane, and then the functional areas are initially extracted according to the network coverage area ratio index Sij of various types of functional land.

[0066] The urban functional area classification based on street view images obtained by the relationship perception network and the functional area mapping obtained by POI point data are combined, and the two-dimensional kernel density calculation is performed on the four types of functional area data to obtain the kernel density distribution of each functional area category. Based on the obtained kernel density distribution of each functional area category, the urban functional area type in each area is determined by setting the kernel density threshold in each area.

[0067] The present invention also provides an urban functional area recognition system based on a street view image target relationship perception network, comprising:

[0068] A data acquisition module, used to acquire street view images, and perform random horizontal flipping, random geometric transformation and random color jittering operations on the street view images to enhance training data;

[0069] A model training module, used to construct a training data set according to the training data, input the street view images of the training set into a graph-based relationship perception network for training, and obtain a trained urban functional area recognition model;

[0070] An image recognition module, used to perceive the spatial relationship of the street view image and recognize the street view image according to the obtained urban functional area type;

[0071] The functional area classification module is used to obtain POI data within the study area, classify the POI data, and obtain the urban functional area classification based on the POI data.

[0072] The urban functional area recognition method and system based on street view image target relationship perception network of the present invention is based on a graph-based relationship perception network framework, which can be used to capture local spatial relationships and global semantic relationships in street view images, and can obtain more building detail features, thereby obtaining more accurate building categories. Compared with the street view image urban functional area recognition method based on image-level CNN, it can simulate the human recognition and reasoning process, capture the local spatial relationships and global semantic relationships in street view images, and enhance the capabilities of the urban functional area detector.

[0073] First embodiment

[0074] The following describes the specific implementation of the present invention by taking Calgary, Canada as a research area.

[0075] Step 1: Data preprocessing

[0076] 1) The BEAUTY dataset used in the present invention contains 19,070 street view images and 38,857 independent buildings.

[0077] 2) Randomly select 75% of the samples from each category as training / validation sets, and the remaining 25% as test sets. The training / validation sets are then randomly divided into training and validation sets in a ratio of 9:1. In order to reduce the impact of class imbalance and obtain better performance, a random minority oversampling strategy is used to rebalance the training samples. The public and industrial samples are enlarged by 2 times and 2.5 times, respectively. Since all training samples are randomly flipped horizontally in the data augmentation stage before training, only a few need to be randomly copied. The present invention also tries random geometric transformation and random color jittering to enhance the training data. However, considering the tasks of street view image classification and building detection, the performance is not as good as random horizontal flipping. Sample distribution after rebalancing. After balancing the classes of data augmentation, the total number of training images and individual buildings is 16,871 and 32,439, respectively.

[0078] Step 2. Baseline test for street view image classification: For the task of street view image classification, ResNet50 and ResNet101 are selected as candidate baseline models. We fine-tune the pre-trained model 5 on BEAUTY with a learning rate of 0.01, multiplied by 0.1 after every 10 epochs. Training was performed for 100 epochs using Adam as the optimizer. Although the training set has been rebalanced, the macro-average of each class of indicators, namely global precision (MP), global recall (MR), and global F1Score (M-F1), is used. As shown in Table I, ResNet50 performs better than ResNet101 in all macro-average indicators. Therefore, ResNet50 is selected as the baseline model for street view image classification on BEAUTY.

[0079] Table I Image classification accuracy

[0080]

[0081] Step 3. Baseline test for building detection: For the task of building detection, Faster R-CNN and Cascaded RCNN with ResNet50 and ResNet101 backbones are selected as candidate baseline models. The building detection accuracy is shown in Table II

[0082] Table II Building detection accuracy

[0083]

[0084] Step 4: Functional area recognition test based on relationship-aware network: After selecting Ca-101 as the default detector, the functional area recognition test is performed through two relationship-aware networks, namely the spatial relationship-aware network and the semantic relationship-aware network.

[0085] Table III Functional area recognition accuracy

[0086]

[0087] Step 5: Functional area identification based on POI data

[0088] 1) Obtain POI data within the study area from the OSM official website through crawling.

[0089] 2) Taking into account the general awareness and significance of urban functions, the POI data are divided into residential functional areas, commercial functional areas, industrial functional areas and public service functional areas, similar to the functional area classification of street view images, and directly used as urban functional area types.

[0090] 3) Through key data processing such as extraction, cleaning, duplication checking, coordinate transformation, definition of projection and spatial position of POI data, the urban functional area type definition of POI and the relative quantity calculation are completed.

[0091] 4) Download the corresponding road network data in the study area from the OpenStreetMap official website and trim the data according to the scope of the study area.

[0092] 5) According to the POI sample points of the present invention, they are expanded in a two-dimensional plane, and then the functional areas are initially extracted according to the network coverage area ratio index Sij of various functional land uses.

[0093] 6) Combine the urban functional area classification based on street view images obtained by the relationship perception network and the functional area mapping obtained by POI point data, perform two-dimensional kernel density calculation on the four types of functional area data, and obtain the kernel density distribution of each functional area category. Based on the obtained kernel density distribution of each functional area category, the urban functional area type in each area is determined by setting the kernel density threshold in each area.

[0094] Step 6: The urban functions obtained by combining POI data and street view image data are used to identify urban functional areas using the kernel density analysis method.

[0095] The experimental results of the present invention show that compared with the image-level CNN based model, the proposed method achieves a performance improvement of 12.8% in macro-precision and 12% in macro-recall.

[0096] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for identifying urban functional areas based on street view image object relationship perception network, characterized in that: include: Step 1, obtaining a street view image, and performing random horizontal flipping, random geometric transformation, and random color jittering operations on the street view image to enhance training data; Step 2, constructing a training data set based on the training data, inputting the street view images of the training set into a graph-based relationship perception network for training, and obtaining a trained urban functional area recognition model; Step 3: Perform spatial relationship perception on the street view image and input the street view image into the pre-trained baseline model FasterR-CNN network to obtain the target box and classification weight of the building in the street view image; The obtained classification weights are used to visually embed the building objects, and the paired spatial information is captured using the paired pseudo-coordinate function u(a, b), from which the Gaussian kernel WK with a learnable mean and covariance K is determined as the weight of the graph convolutional network; The regional visual embedding is used as a node and the Gaussian kernel WK is used as a weight, which is input into the graph convolutional network to perceive the spatial relationship of street view images. Identify the street view image according to the obtained urban functional area type, perceive the semantic relationship of the street view image, embed the label of the street view image into words, and obtain the word embedding vector of the label; A normalized adjacency matrix of the graph convolutional network is obtained by calculating the number of collinearities of the labels in the dataset; Input the word embedding vector and adjacency matrix into the graph convolutional neural network to obtain the graph embedding vector of the image label; The semantic relationship features of the street view image are obtained by performing a dot multiplication between the image embedding vector and the visual feature vector obtained by the pre-trained baseline model Faster R-CNN. Connecting the obtained spatial relationship features of the street view image with the semantic relationship features and the overall visual relationship features of the street view image; The connected vector is input into the fully connected network, and finally softmax classification is performed to obtain the type of urban functional area; Step 4: Acquire POI data within the study area, classify the POI data, and obtain the urban functional area classification based on the POI data.

2. The method for identifying urban functional areas based on street view image object relationship perception network according to claim 1 is characterized in that: The step 2 specifically includes: 75% of the samples in each category of the street view images are randomly selected as training / validation sets, and the remaining 25% are used as test sets. The training / validation sets are randomly divided into training sets and validation sets in a ratio of 9:

1. The street view images in the training set are input into a graph-based relational perception network for training to obtain a trained urban functional area recognition model.

3. The urban functional area recognition method based on street view image object relationship perception network according to claim 2 is characterized in that: The step 4 specifically includes: Obtain POI data within the study area from the OSM official website through crawling; According to the urban functional area type, POI data is divided into residential functional area, commercial functional area, industrial functional area and public service functional area, which corresponds to the functional area classification of street view images; By performing key data processing on POI data, the urban functional area type of POI is defined; Obtain the corresponding road network data in the study area and cut the data according to the scope of the study area; The urban functional area classification based on street view images obtained by the relationship perception network and the functional area mapping obtained by POI point data are combined, and the two-dimensional kernel density calculation is performed on the four types of functional area data to obtain the kernel density distribution of each functional area category; According to the obtained kernel density distribution of each functional area category, the type of urban functional area in each area is determined by setting the kernel density threshold in each area.

4. A system for identifying urban functional areas based on street view image object relationship perception network, characterized in that: include: A data acquisition module, used to acquire street view images, and perform random horizontal flipping, random geometric transformation and random color jittering operations on the street view images to enhance training data; A model training module, used to construct a training data set according to the training data, input the street view images of the training set into a graph-based relationship perception network for training, and obtain a trained urban functional area recognition model; The image recognition module is used to perceive the spatial relationship of street view images. The street view images are input into the pre-trained baseline model Faster R-CNN network to obtain the target boxes and classification weights of the buildings in the street view images. The obtained classification weights are used to visually embed the building objects, and the paired pseudo-coordinate function u(a, b) is used to capture the paired spatial information, and the Gaussian kernel WK with a learnable mean and covariance K is determined as the weight of the graph convolutional network. The regional visual embedding is used as a node and the Gaussian kernel WK is used as a weight, which is input into the graph convolutional network to perceive the spatial relationship of street view images. Identify the street view image according to the obtained urban functional area type, perceive the semantic relationship of the street view image, embed the label of the street view image into words, and obtain the word embedding vector of the label; A normalized adjacency matrix of the graph convolutional network is obtained by calculating the number of collinearities of the labels in the dataset; The word embedding vector and the adjacency matrix are input into the graph convolutional neural network to obtain the graph embedding vector of the image label; the graph embedding vector is dot-multiplied with the visual feature vector obtained by the pre-trained baseline model Faster R-CNN to obtain the semantic relationship features of the street view image; the spatial relationship features of the street view image obtained are connected with the semantic relationship features and the overall visual relationship features of the street view image; The connected vector is input into the fully connected network, and finally softmax classification is performed to obtain the type of urban functional area; The functional area classification module is used to obtain POI data within the study area, classify the POI data, and obtain the urban functional area classification based on the POI data.

Images