A method and system for identifying urban functional areas by integrating multimodal time series data
By fusing Sentinel-2 remote sensing images, road network data and POI data, and using Res-HANet, AtLSTM and BERT-RF models to extract features and fusion, the problems of irregular shapes, uneven sizes and missing data in urban functional area identification are solved. , improves the accuracy and objectivity of recognition.
Patent Information
- Application Number
- CN202410785210.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-18
- Publication Date
- 2025-01-03
- Estimated Expiration
- 2044-06-18
AI Technical Summary
The existing technology has problems such as time-consuming, inefficient, and strong subjectivity in urban functional area identification. In addition, deep learning models are difficult to deal with functional areas with irregular shapes and uneven sizes. They are apparently similar when recognizing remote sensing images but function. Different blocks are prone to confusion, and the absence of POI data or false information affects the integrity and objectivity of the identification.
By using the multimodal timing data fusion method, by obtaining Sentinel-2 remote sensing images, road network data and POI data, spatial features are extracted using the Res-HANet network model, temporal features are extracted by the AtLSTM model, and social functions semantic features are extracted by the BERT-RF model. And fuse it to achieve accurate identification of urban functional areas.
Improve the accuracy and understanding of dynamic changes in urban functional area recognition, overcome the limitations of irregular spatial data processing, enhance the identification of the uniqueness of functional area, make up for the shortcomings of insufficient single-modal data information, and improve the identification of Objectivity and integrity.
Smart Images
Figure CN118570551B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of remote sensing image processing, and in particular relates to a method and system for identifying urban functional areas by integrating multi-modal time series data. Background Art
[0002] As the main activity place of human society, cities have different functions, such as commercial land, residential land, industrial land, public service land, etc. Identifying urban functional areas can better understand the characteristics and functions of each area, so as to allocate and utilize resources in a targeted manner. Initially, the identification of urban functional areas mostly used qualitative methods such as statistical surveys and expert judgment, but these methods have problems such as time-consuming and labor-intensive, low efficiency, and strong subjectivity in the application process. Using high-resolution remote sensing image data, the spatial information of objects can be effectively extracted, and combined with deep learning models, the classification of urban functional areas can be basically realized.
[0003] However, general deep learning models only support input of regular square images, while urban functional areas are usually of arbitrary shapes. In addition, deep learning models usually require the same size of input images when performing batch processing, but the sizes of functional areas are often uneven. Setting a specific size to crop the functional area will fragment the functional area, making it difficult for the model to extract features from the complete image of each functional area.
[0004] On the other hand, remote sensing images can be confusing when identifying blocks that are similar in appearance but different in function, so it is a trend to introduce data sources with distinguishing capabilities. Point of Interest (POI) data carries text information that represents the function of blocks. Some studies have used density analysis to complete the division of urban functional areas. Kernel density analysis is a common method in traditional POI analysis methods. However, the selection of kernel functions and the determination of window size are often subjective, and the analysis assumes that the data is evenly distributed throughout the study area. However, in actual situations, the data often presents a non-uniform distribution, which may produce boundary effects. In addition, kernel density analysis usually focuses on the spatial distribution of geographic location data, while ignoring the rich semantic information provided by text descriptions. In addition, due to the rich temporal features embedded in mobile phone time series data, user behavior data, and Weibo data, the wide coverage, simple collection method, and accessibility to geographic coordinates make geographic social media data very beneficial for improving the timeliness of urban functional area classification. However, these data have the problem of missing data in certain regions and specific populations, and cannot provide rich information comparable to remote sensing image data. In addition, if there is missing or false information in the POI data, it will affect the integrity and objectivity of the identification of urban functional areas. Summary of the invention
[0005] In view of this, the present invention proposes a method and system for identifying urban functional areas by integrating multimodal time series data, so as to solve the problem of inaccurate identification of urban functional areas.
[0006] In a first aspect, the present invention discloses a method for identifying urban functional areas by fusing multimodal time series data, the method comprising:
[0007] Acquire multi-temporal Sentinel-2 remote sensing images of the study area and perform the first preprocessing to obtain remote sensing images of four seasons;
[0008] Obtain the road network data of the study area and perform a second preprocessing, and superimpose the road network data after the second preprocessing onto the remote sensing images of the four seasons respectively;
[0009] The spatial features are extracted from the single-season remote sensing images superimposed with road network data through the first neural network model, and the temporal features are extracted from the remote sensing images of four seasons through the second neural network model to obtain the spatiotemporal fusion features.
[0010] Acquire POI data of the research area and perform a third preprocessing, and extract social function semantic features from the POI data after the third preprocessing by using a third neural network model;
[0011] The spatiotemporal fusion features are integrated with the social function semantic features to obtain the classification results of urban functional areas.
[0012] On the basis of the above technical solution, preferably, the obtaining of the road network data of the study area and performing the second preprocessing specifically includes:
[0013] Crop the road network data, remove internal roads and invalid roads, and determine the primary, secondary and tertiary road networks;
[0014] Perform a topological check on the road network to obtain closed, non-overlapping functional area units.
[0015] On the basis of the above technical solution, preferably, the extracting spatial features from the single-season remote sensing image by the first neural network model specifically includes:
[0016] Generate a minimum bounding rectangle for each functional area unit, and expand the minimum bounding rectangle outward to obtain a square bounding rectangle;
[0017] The single-season remote sensing image is divided according to the square boundary rectangle to obtain a set of square image blocks and a set of mask images representing the boundaries of urban functional areas;
[0018] Resampling the square image blocks into square image blocks of different scales, and inputting the square image blocks into the first neural network model for feature extraction, and outputting feature maps of different scales;
[0019] The feature maps of different scales output by the first neural network model are respectively multiplied element-by-element with the mask image representing the boundary of the urban functional area to obtain the spatial characteristics of the single-season remote sensing image.
[0020] On the basis of the above technical solution, preferably, the first neural network model is a Res-HANet network model, comprising a first convolutional layer, a first pooling layer, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module connected in sequence;
[0021] Among them, the first feature extraction module includes 3 Res-HA modules with a step size of 1;
[0022] The second feature extraction module includes 1 Res-HA module with a step size of 2 and 3 Res-HA modules with a step size of 1;
[0023] The third feature extraction module includes 1 Res-HA module with a step size of 2 and 5 Res-HA modules with a step size of 1;
[0024] The fourth feature extraction module includes 1 Res-HA module with a step size of 2 and 2 Res-HA modules with a step size of 1;
[0025] The Res-HA module includes a second convolutional layer, an HGC module, a sCBAM module, a third convolutional layer and a splicing layer which are connected in sequence.
[0026] On the basis of the above technical solution, preferably, the expression of the Res-HA module is:
[0027] y=x+conv 1×1 (sCBAM(HGC(conv 1×1 (x))))
[0028] Where x and y represent the input and output of the Res-HA module respectively; conv 1×1 (·) represents a 1×1 convolution operation, HGC(·) represents the multi-scale convolution operation of the HGC module, and sCBAM(·) represents the attention mechanism operation of the sCBAM module;
[0029] The HGC module divides the input feature map into five sub-feature maps x1-x5 according to the channel, and performs convolution operations of different scales on each sub-feature map. The expressions of convolution operations of different scales are:
[0030]
[0031] Among them, y i Convolution is the feature map obtained by convolution operations of different scales.3×3 (·) represents a 3×3 convolution operation;
[0032] The feature maps y1 to y5 are concatenated to obtain a feature map containing multi-scale information and input into the sCBAM module;
[0033] The sCBAM module is a hybrid attention module that combines the channel attention mechanism and the spatial attention mechanism.
[0034] On the basis of the above technical solution, preferably, the extracting time features from the remote sensing images of the four seasons by the second neural network model to obtain the spatiotemporal fusion features specifically includes:
[0035] The spatial features of the four seasons extracted by the first neural network model are respectively input into the second neural network model in the order of spring, summer, autumn and winter, and the spatiotemporal fusion features are output; the second neural network model adopts the AtLSTM model.
[0036] On the basis of the above technical solution, preferably, the third preprocessing of the POI data superimposed with the road network data and extracting the social function semantic features from the third preprocessed POI data by the third neural network model specifically includes:
[0037] The POI data is cleaned and the data with public awareness below the preset threshold is eliminated; each POI data includes three attributes: name, category and longitude and latitude;
[0038] Connect the names of POI data belonging to the same functional area unit into text data;
[0039] Encode and segment the text data, and convert the text sequence after encoding and segmentation into a token sequence through a dictionary;
[0040] Input the token sequence corresponding to each functional area unit into the pre-trained BERT model and output the corresponding embedded semantic features;
[0041] The embedded semantic features are classified by the pre-trained random forest classifier, and the category probability of the urban functional area to which the corresponding functional area unit belongs is output;
[0042] The embedded semantic features output by the BERF model and the category probability output by the random forest classifier are combined as social function semantic features.
[0043] In a second aspect, the present invention discloses an urban functional area identification system integrating multimodal time series data, the system comprising:
[0044] Preprocessing module: used to obtain multi-temporal Sentinel-2 remote sensing images of the study area and perform the first preprocessing to obtain remote sensing images of four seasons; obtain the road network data of the study area and perform the second preprocessing, and superimpose the road network data after the second preprocessing on the remote sensing images of the four seasons respectively;
[0045] Spatial feature extraction module: used for extracting spatial features from a single-season remote sensing image superimposed with road network data through a first neural network model;
[0046] Temporal feature extraction module: used to extract temporal features from remote sensing images of four seasons through the second neural network model to obtain spatiotemporal fusion features;
[0047] Semantic feature extraction module: used to obtain POI data of the research area and perform the third preprocessing, and extract social function semantic features from the POI data after the third preprocessing through the third neural network model;
[0048] Multimodal feature fusion module: used to fuse spatiotemporal fusion features with social function semantic features to obtain the classification results of urban functional areas.
[0049] In a third aspect of the present invention, an electronic device is disclosed, comprising: at least one processor, at least one memory, a communication interface and a bus;
[0050] Wherein, the processor, memory, and communication interface communicate with each other via the bus;
[0051] The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to the first aspect of the present invention.
[0052] According to a fourth aspect of the present invention, a computer-readable storage medium is disclosed, wherein the computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method described in the first aspect of the present invention.
[0053] Compared with the prior art, the present invention has the following beneficial effects:
[0054] 1) The present invention extracts spatial features from single-season remote sensing images through a first neural network model, extracts temporal features from remote sensing images of four seasons through a second neural network model, obtains spatiotemporal fusion features, extracts social function semantic features from the third pre-processed POI data through a third neural network model, and finally performs multimodal feature fusion for urban functional area identification. Since the high spatiotemporal resolution data of Sentinel-2 images and the semantic information of POI data are integrated, and the time series analysis of remote sensing images of four seasons is combined, the urban functional area identification method integrating multimodal time series data of the present invention can make up for the defect of insufficient information in urban functional area classification research relying only on single-modal data, and enhance the understanding of the dynamic changes of urban functional areas and the accuracy of urban functional area identification.
[0055] 2) The present invention generates a minimum bounding rectangle for each functional area unit through the road network data of the study area, expands the minimum bounding rectangle outward to obtain a square bounding rectangle, and divides the single-season remote sensing image according to the square bounding rectangle to obtain a group of square image blocks and a group of mask images representing the boundaries of urban functional areas. Urban functional areas with irregular shapes and inconsistent sizes can be converted into regular square image blocks for multi-scale feature learning, thereby overcoming the limitations of existing urban functional area recognition technology in processing irregular spatial data.
[0056] 3) The present invention extracts spatial features from single-season remote sensing images by constructing a Res-HANet network model, and uses multiple Res-HA modules to be repeated in series to form a deep network structure. Each Res-HA module combines traditional convolution operations, multi-scale convolutions and attention mechanisms to achieve efficient encoding of input feature maps, which not only enhances the learning ability of multi-scale features, but also achieves adaptive emphasis on key information through the attention mechanism, effectively suppressing noise interference; in addition, by introducing a mask layer based on the boundary of functional areas, it is possible to focus on the effective features inside the urban functional areas, eliminate the interference of non-functional area backgrounds, make full use of contextual information, and enhance the recognition of the uniqueness of functional areas.
[0057] 4) The present invention utilizes the context understanding ability of the BERF model to extract embedded semantic features from the text description of POI data, and extracts probabilistic features through a random forest classifier, and combines the two as social function semantic features, further enhancing the expressive power of the features.
[0058] 5) Based on the spatial characteristics of remote sensing images of a single season, the present invention extracts temporal features by constructing a seasonal series AtLSTM model. In the AtLSTM model, the hidden state of each time step is weighted by an attention mechanism layer. The weight vector reflects the degree of attention paid by the model to information at different time steps, so that the model can automatically focus on the most informative time step in the sequence and effectively extract information in the time series. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0060] Figure 1 A flow chart of a method for identifying urban functional areas by fusing multimodal time series data provided by an embodiment of the present invention;
[0061] Figure 2 A framework diagram of a method for identifying urban functional areas by fusing multimodal time series data provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0062] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0063] See also Figure 1 The present invention discloses a method for identifying urban functional areas by integrating multimodal time series data, the method comprising:
[0064] S1. Obtain multi-temporal Sentinel-2 remote sensing images of the study area and perform the first preprocessing to obtain remote sensing images of four seasons.
[0065] The present invention collects remote sensing data of four seasons near a study area on a Google Earth Engine (GEE) platform and performs a first preprocessing.
[0066] The specific contents of the first preprocessing include: for single-season remote sensing images, select Sentinel-2L2A 10m and 20m resolution band data, and resample all remote sensing images to 10m. Use ENVI to stitch multiple remote sensing images to cover the entire study area. Finally, remote sensing images of the study area in spring, summer, autumn and winter are obtained.
[0067] S2. Acquire the road network data of the study area and perform a second preprocessing, and superimpose the road network data after the second preprocessing onto the remote sensing images of the four seasons respectively.
[0068] The road network data comes from Open Street Map (OSM). According to the scope of the study area, the OSM data related to the study area is cut out, and then the road network data is subjected to the second preprocessing.
[0069] The specific contents of the second preprocessing include: removing internal roads and invalid roads of the plots, determining the primary, secondary and tertiary road networks, and performing topological checks on the roads. For roads with hanging points, the initially unconnected roads are connected to ensure that each block (functional area unit) is in a closed state and there is no overlapping of plots.
[0070] Then the different levels of roads in the OSM data are processed as hierarchical buffers. The road network layer is superimposed on the remote sensing images of each season to obtain the final functional area unit.
[0071] S3. Extract spatial features from single-season remote sensing images superimposed with road network data through the first neural network model.
[0072] Step S3 specifically includes the following sub-steps:
[0073] S31. Generate a minimum bounding rectangle for each functional area unit, and expand the minimum bounding rectangle outward to obtain a minimum square bounding rectangle.
[0074] Specifically, each functional area unit is generally a polygonal area or an irregular area, so each functional area unit has a minimum circumscribed rectangle, that is, a minimum bounding rectangle, which is a minimum area rectangle that completely contains all pixels of the functional area. Then, by comparing the long side and the short side of the original rectangle, the shorter side is selected to expand southward or eastward until the two sides are equal in length, thereby forming a square bounding rectangle.
[0075] The present invention adjusts each minimum boundary rectangle into a square by the above method, so as to ensure that all segmented image blocks have a uniform shape.
[0076] S32. Divide the single-season remote sensing image according to the square boundary rectangle to obtain a group of square image blocks and a group of mask images representing the boundaries of urban functional areas.
[0077] The single-season remote sensing image is cropped according to the square boundary rectangle generated above to obtain a group of square image blocks. At the same time, according to the cropping process, a group of mask layers are created. This group of mask images accurately represents the boundaries of each urban functional area and corresponds one-to-one to the square image blocks.
[0078] The present invention can convert urban functional areas with irregular shapes and inconsistent sizes into regular square image blocks, which is convenient for subsequent feature learning through the first neural network model and overcomes the limitations of existing urban functional area recognition technology in processing irregular spatial data.
[0079] S33, constructing a first neural network model.
[0080] Figure 2 This is a framework diagram of a method for identifying urban functional areas by fusing multimodal time series data according to the present invention.
[0081] like Figure 2 As shown, the first neural network model of the present invention is a Res-HANet network model, comprising a first convolutional layer, a first pooling layer, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module connected in sequence;
[0082] Among them, the first feature extraction module includes 3 Res-HA modules with a step size of 1; the second feature extraction module includes 1 Res-HA module with a step size of 2 and 3 Res-HA modules with a step size of 1; the third feature extraction module includes 1 Res-HA module with a step size of 2 and 5 Res-HA modules with a step size of 1; the fourth feature extraction module includes 1 Res-HA module with a step size of 2 and 2 Res-HA modules with a step size of 1; the Res-HA module includes a second convolutional layer, an HGC module, a sCBAM module, a third convolutional layer and a splicing layer connected in sequence.
[0083] The expression of the Res-HA module is:
[0084] y=x+conv 1×1 (sCBAM(HGC(conv 1×1 (x))))
[0085] Where x and y represent the input and output of the Res-HA module respectively; conv 1×1 (·) represents a 1×1 convolution operation. HGC(·) represents the multi-scale convolution operation of the HGC module, and sCBAM(·) represents the attention mechanism operation of the sCBAM module.
[0086] In the Res-HA module, HGC(·) is used to divide the input feature map into five sub-feature maps x1~x5 according to the channel, and perform convolution operations of different scales on each sub-feature map. The expressions of convolution operations of different scales are:
[0087]
[0088] y i Convolution is the feature map obtained by convolution operations of different scales. 3×3 (·) represents a 3×3 convolution operation; the feature maps y1~y5 are concatenated to obtain a feature map containing multi-scale information and input it into the sCBAM module.
[0089] The sCBAM module combines the channel attention mechanism and the spatial attention mechanism to enhance the representation ability of the feature map. In the channel attention module, each channel is first average pooled to generate a one-dimensional feature vector that reflects the global distribution of each channel. The one-dimensional feature vector is activated by the Sigmoid function to output the channel attention map, which contains the importance weight of each channel. Finally, the channel attention map is multiplied with the input feature map. In the spatial attention module, the feature map is first average pooled and max pooled in the channel dimension to generate two feature maps, which are connected together. After that, a 7×7 convolution operation is performed on the connected feature map to mine spatial correlation, and then the spatial attention map is output through the Sigmoid function. Finally, the spatial attention map is multiplied with the input feature map of the spatial attention module to further optimize the features in the spatial dimension.
[0090] The core of the Res-HANet network model of the present invention lies in its unique Res-HA module, which is repeated in series to form a deep network structure. Each Res-HA module realizes efficient encoding of the input feature map by combining convolution operation, multi-scale convolution and attention mechanism. This design not only enhances the learning ability of multi-scale features, but also realizes adaptive emphasis on key information through the attention mechanism, effectively suppressing noise interference.
[0091] S34, resampling the square image blocks into square image blocks of different scales, and inputting them into the first neural network model for feature extraction respectively, and outputting feature maps of different scales.
[0092] The present invention extracts spatial features of different scales based on the above first neural network model.
[0093] All square image blocks are resampled to four different sizes (128×128, 256×256, 512×512, 800×800) and respectively input into the first neural network model for feature extraction. The purpose is to alleviate the scale effect problem by providing perspectives of different granularities, so as to facilitate the capture of diverse information from fine details to macro patterns through images of different sizes, and finally output feature maps of different scales.
[0094] S35. Multiply the feature maps of different scales output by the first neural network model by the mask images representing the boundaries of urban functional areas element by element to obtain the spatial characteristics of the single-season remote sensing image.
[0095] The present invention further filters the output feature map obtained by the Res-HANet network model through a mask layer, that is, the mask layer is placed after the Res-HANet network model, and uses the image information around the mask to help the network learn the contextual features of the functional area.
[0096] Specifically, the mask layer of the present invention is the mask image of step S32, which is composed of a matrix identical to the output feature map, and the matrix records the shape information of the urban functional area. If a pixel in the output feature map of the Res-HANet network model falls outside the mask image, the corresponding element value in the matrix is set to 0; if it is within the mask, the corresponding element value is set to 1. Feature maps of different scales are respectively multiplied element by element with the corresponding mask image to retain the spatial features of the functional area in the single-season remote sensing image and remove irrelevant features.
[0097] By introducing a mask layer based on the boundaries of functional areas, the present invention can focus on the effective features inside the urban functional areas and eliminate the interference of the background of non-functional areas. The use of the mask layer allows the network to make full use of contextual information during the learning process and enhance the recognition of the uniqueness of the functional areas. The spatial-spectral feature vector of the study area is obtained.
[0098] S4. The second neural network model is used to extract temporal features from the remote sensing images of the four seasons to obtain spatiotemporal fusion features.
[0099] like Figure 2As shown, the second neural network model of the present invention adopts the seasonal series AtLSTM model to extract temporal features from Sentinel-2 remote sensing image data of four seasons. Based on the standard LSTM, the AtLSTM model introduces an attention module to enhance the model's attention to the information of specific time steps in the sequence. The attention module includes a fully connected layer and a Softmax activation function, which is used to calculate the attention weight of the hidden state of each time step. By integrating the attention mechanism, the AtLSTM model not only retains the ability of LSTM to process long-term dependencies in sequence data, but also enhances the model's sensitivity and selectivity to information at specific time points in the sequence, which is particularly effective for processing tasks such as seasonal city classification with obvious time series characteristics.
[0100] like Figure 1 As shown, the spatial features of the four seasons extracted by the first neural network model are respectively input into the second neural network model in the order of spring, summer, autumn and winter, and the temporal features are extracted based on the spatial features to output the spatiotemporal fusion features.
[0101] S5. Acquire the POI data of the study area and perform a third preprocessing, and extract social function semantic features from the POI data after the third preprocessing through a third neural network model.
[0102] Remote sensing images, road network data and POI data can all be regarded as a modality. The fusion of multimodal data can make up for the defects of relying solely on single-modal data in the classification research of urban functional areas, and can understand the urban spatial pattern more comprehensively and accurately. Therefore, the present invention obtains the POI data of the study area and extracts social function semantic features from it.
[0103] The present invention adopts the BERT model to extract semantic features, but the feature extraction capability of a single network model is limited. The present invention further enhances the feature expression capability through a random forest classifier, and constructs a third neural network model, namely, a BERT-RF model, through the BERT model and the random forest classifier to extract richer social function semantic features from the third preprocessed POI data, such as Figure 2 shown.
[0104] Step S5 specifically includes the following sub-steps:
[0105] S51, converting the geographic coordinate system of the original POI data into the WGS-84 coordinate system, and then performing a third preprocessing on the POI data after the coordinate conversion.
[0106] The specific contents of the third preprocessing include:
[0107] (1) Clean the POI data and delete POI data with incorrect, duplicate, or abnormal information. Points with low public awareness, such as transportation stations, public toilets, and logistics and express delivery, should also be eliminated.
[0108] (2) Reclassify the POI data into five types of urban functional areas: residential land, commercial service facility land, industrial land, public management and public service land, and green space and square land. Finally, retain the POI data with three attributes: name, category, and latitude and longitude.
[0109] (3) The names of POI data belonging to the same functional area unit are concatenated into a text in TXT format, and the categories of urban functional areas are marked by combining visual inspection of remote sensing images.
[0110] S52. Randomly select 80% of the data labeled with the categories of urban functional areas from the TXT text as training samples and 20% as test samples to pre-train the BERT model.
[0111] In order to fully learn the data features, the training samples of the study area can be expanded by adjusting the order of POI data content. The text content of the functional area without POI data is set to "Null".
[0112] S53. Based on the above TXT text, the text content of the POI data of each functional area unit is input into the pre-trained BERT model for feature extraction.
[0113] First, the text data is encoded and segmented through the BERT model Tokenizer module, and each subword is mapped to the corresponding ID. Secondly, the vocab.txt file open sourced by Google is loaded to construct a dictionary. According to the dictionary, the text sequence after encoding and segmentation is converted into a token sequence, and the [CLS] and [SEP] symbols are added to the beginning and end of the token sequence respectively. The maximum length of the input text sequence is set to 300 characters. When the length of the text sequence exceeds the maximum length, it will be truncated. If it is less than the maximum length, it will be padded to the maximum length with zero values. For each token sequence, the attention_mask vector is 0 if it is equal to the PAD value, otherwise it is 1. Then, create a data loader to process the data in batches. Finally, load the BERT pre-trained model, select the pre-trained language model bert-base-chinese of the Hugging face library, input the encoded token sequence into the pre-trained BERT model for semantic feature extraction, and output the corresponding embedded semantic features.
[0114] The present invention utilizes the pre-trained BERT model to generate high-dimensional embedded semantic features, which can capture the rich semantic information in the description text of each functional area unit.
[0115] S54. Train a random forest classifier based on the embedded semantic features and the category label of each functional area unit to perform category prediction on the embedded semantic features.
[0116] The embedded semantic features are classified by the pre-trained random forest classifier, and the category probability of the urban functional area to which the corresponding functional area unit belongs is output.
[0117] S55. Combine the embedded semantic features output by the BERF model and the category probabilities output by the random forest classifier to obtain the social function semantic features representing the entire study area.
[0118] S6. Integrate the spatiotemporal fusion features with the social function semantic features to obtain the classification results of urban functional areas.
[0119] The spatiotemporal fusion features are connected with the social function semantic features through a fully connected layer, input into the Softmax function, and output the results of classifying urban functional areas.
[0120] The method for identifying urban functional areas by integrating multimodal time series data proposed in the present invention extracts spatial features from remote sensing images through a Res-HANet network model that integrates the HGC module and the sCBAM module; the seasonal series AtLSTM model is used to extract temporal features from the spatial feature data of the four seasons. The AtLSTM model introduces an attention module based on the standard LSTM to enhance the model's attention to specific time step information in the sequence. Then, the BERT-RF model is used to extract embedded semantic features and probabilistic features from POI data, which not only retains the contextual understanding ability of BERT, but also further enhances the feature expression ability through the random forest classifier, and combines embedded semantic features and probabilistic features into social function semantic features. Finally, the spatiotemporal fusion features and social function semantic features are fused to obtain the classification results of urban functional areas.
[0121] The urban functional area identification method that integrates multimodal time series data proposed in the present invention makes full use of Sentinel-2 data, POI data and road network data, which can make up for the lack of information in the urban functional area classification research relying solely on single-modal data, deepen the understanding of urban space, better optimize urban resource allocation, and achieve sustainable development of cities.
[0122] On the basis of the above method embodiments, the present invention further discloses an urban functional area identification system integrating multimodal time series data, the system comprising:
[0123] Preprocessing module: used to obtain multi-temporal Sentinel-2 remote sensing images of the study area and perform the first preprocessing to obtain remote sensing images of four seasons; obtain the road network data of the study area and perform the second preprocessing, and superimpose the road network data after the second preprocessing on the remote sensing images of the four seasons respectively;
[0124] Spatial feature extraction module: used for extracting spatial features from a single-season remote sensing image superimposed with road network data through a first neural network model;
[0125] Temporal feature extraction module: used to extract temporal features from remote sensing images of four seasons through the second neural network model to obtain spatiotemporal fusion features;
[0126] Semantic feature extraction module: used to obtain POI data of the research area and perform the third preprocessing, and extract social function semantic features from the POI data after the third preprocessing through the third neural network model;
[0127] Multimodal feature fusion module: used to fuse spatiotemporal fusion features with social function semantic features to obtain the classification results of urban functional areas.
[0128] The above system embodiments and method embodiments correspond one to one, and for a brief description of the system embodiments, please refer to the method embodiments.
[0129] The present invention also discloses an electronic device, comprising: at least one processor, at least one memory, a communication interface and a bus; wherein the processor, memory, and communication interface communicate with each other through the bus; the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the aforementioned method of the present invention.
[0130] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, wherein the computer instructions enable the computer to implement all or part of the steps of the method described in the embodiment of the present invention. The storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk, and other media that can store program codes.
[0131] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be distributed to multiple network units. A person skilled in the art may select some or all of the modules according to actual needs to achieve the purpose of the solution of this embodiment without creative effort.
[0132] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for identifying urban functional areas by integrating multimodal time series data, characterized in that: The method comprises: Acquire multi-temporal Sentinel-2 remote sensing images of the study area and perform the first preprocessing to obtain remote sensing images of four seasons; Obtain the road network data of the study area and perform a second preprocessing, and superimpose the road network data after the second preprocessing onto the remote sensing images of the four seasons respectively; The spatial features are extracted from the single-season remote sensing images superimposed with road network data through the first neural network model, and the temporal features are extracted from the remote sensing images of four seasons through the second neural network model to obtain the spatiotemporal fusion features. Acquire POI data of the research area and perform a third preprocessing, and extract social function semantic features from the POI data after the third preprocessing by using a third neural network model; The spatiotemporal fusion features are integrated with the social function semantic features to obtain the classification results of urban functional areas; The extracting of spatial features from the single-season remote sensing image superimposed with road network data by the first neural network model specifically includes: Generate a minimum bounding rectangle for each functional area unit, and expand the minimum bounding rectangle outward to obtain a square bounding rectangle; The single-season remote sensing image is divided according to the square boundary rectangle to obtain a set of square image blocks and a set of mask images representing the boundaries of urban functional areas; Resampling the square image blocks into square image blocks of different scales, and inputting the square image blocks into the first neural network model for feature extraction, and outputting feature maps of different scales; The feature maps of different scales output by the first neural network model are respectively multiplied element by element with the mask image representing the boundary of the urban functional area to obtain the spatial characteristics of the single-season remote sensing image; The first neural network model is a Res-HANet network model, comprising a first convolutional layer, a first pooling layer, a first feature extraction module, a second feature extraction module, a third feature extraction module and a fourth feature extraction module connected in sequence; Among them, the first feature extraction module includes 3 Res-HA modules with a step size of 1; The second feature extraction module includes 1 Res-HA module with a step size of 2 and 3 Res-HA modules with a step size of 1; The third feature extraction module includes 1 Res-HA module with a step size of 2 and 5 Res-HA modules with a step size of 1; The fourth feature extraction module includes 1 Res-HA module with a step size of 2 and 2 Res-HA modules with a step size of 1; The Res-HA module includes a second convolutional layer, an HGC module, an sCBAM module, a third convolutional layer and a splicing layer connected in sequence; The expression of the Res-HA module is: y=x+conv 1×1 (sCBAM(HGC(conv 1×1 (x)))) Where x and y represent the input and output of the Res-HA module respectively; conv 1×1 (·) represents a 1×1 convolution operation, HGC(·) represents the multi-scale convolution operation of the HGC module, and sCBAM(·) represents the attention mechanism operation of the sCBAM module; The HGC module divides the input feature map into five sub-feature maps x1-x5 according to the channel, and performs convolution operations of different scales on each sub-feature map. The expressions of convolution operations of different scales are: Among them, y i Convolution is the feature map obtained by convolution operations of different scales. 3×3 (·) represents a 3×3 convolution operation; The feature maps y1 to y5 are concatenated to obtain a feature map containing multi-scale information and input into the sCBAM module; The sCBAM module is a hybrid attention module that combines the channel attention mechanism and the spatial attention mechanism.
2. The method for identifying urban functional areas by fusing multimodal time series data according to claim 1 is characterized in that: The obtaining of the road network data of the study area and the second preprocessing specifically include: Crop the road network data, remove internal roads and invalid roads, and determine the primary, secondary and tertiary road networks; Perform a topological check on the road network to obtain closed, non-overlapping functional area units.
3. The urban functional area identification method based on fusion of multimodal time series data according to claim 1 is characterized in that: The second neural network model is used to extract time features from the remote sensing images of the four seasons to obtain the spatiotemporal fusion features, which specifically include: The spatial features of the four seasons extracted by the first neural network model are respectively input into the second neural network model in the order of spring, summer, autumn and winter, and the spatiotemporal fusion features are output; the second neural network model adopts the AtLSTM model.
4. The method for identifying urban functional areas by fusing multimodal time series data according to claim 1, characterized in that: The third neural network model includes a BERT model and a random forest classifier connected in sequence, and the extraction of social function semantic features from the third pre-processed POI data by the third neural network model specifically includes: The POI data corresponding to each functional area unit is encoded and segmented and then input into the pre-trained BERT model to output the corresponding embedded semantic features; The embedded semantic features are classified by the pre-trained random forest classifier, and the category probability of the urban functional area to which the corresponding functional area unit belongs is output; The embedded semantic features output by the BERT model and the category probability output by the random forest classifier are combined as social function semantic features.
5. An urban functional area identification system integrating multimodal time series data, characterized in that: The method according to any one of claims 1 to 4 is used, wherein the system comprises: Preprocessing module: used to obtain multi-temporal Sentinel-2 remote sensing images of the study area and perform the first preprocessing to obtain remote sensing images of four seasons; obtain the road network data of the study area and perform the second preprocessing, and superimpose the road network data after the second preprocessing on the remote sensing images of the four seasons respectively; Spatial feature extraction module: used for extracting spatial features from a single-season remote sensing image superimposed with road network data through a first neural network model; Temporal feature extraction module: used to extract temporal features from remote sensing images of four seasons through the second neural network model to obtain spatiotemporal fusion features; Semantic feature extraction module: used to obtain POI data of the research area and perform the third preprocessing, and extract social function semantic features from the POI data after the third preprocessing through the third neural network model; Multimodal feature fusion module: used to fuse spatiotemporal fusion features with social function semantic features to obtain the classification results of urban functional areas.
6. An electronic device, characterized in that: include: at least one processor, at least one memory, a communication interface, and a bus; Wherein, the processor, memory, and communication interface communicate with each other via the bus; The memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to implement the method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions enable a computer to implement the method according to any one of claims 1 to 4.