A multi-modal feature public building identification method and system based on remote sensing images and street view images, an electronic device, and a storage medium

Abstract

The application belongs to the field of computer vision and ground object recognition and classification, and provides a multi-modal feature public building recognition method and system based on remote sensing images and street view images, an electronic device and a storage medium, the method comprising: remote sensing image and street view image collection and public building recognition; the model training process comprising: image collection, remote sensing feature extraction, instance segmentation, HOG feature and SURF feature extraction, OCR feature extraction, feature splicing and classification training; the application deeply fuses macro-structure information of remote sensing images and micro-semantic details and text recognition information features of street view images through remote sensing features, HOG features, SURF feature extraction and OCR features, supplements facade details and functional identification, adopts geographic coordinate matching to ensure that remote sensing and street view feature spaces of the same building are consistent, and reduces classification misjudgment probability.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and land feature recognition and classification, and in particular to a method, system, electronic device and storage medium for public building recognition based on multimodal features of remote sensing images and street view images. Background Technology

[0002] Traditional building classification methods typically rely on a single data source, based solely on remote sensing or street view, resulting in semantic fragmentation and feature limitations. For example, the technology in application number CN202410114236.1 combines remote sensing image data features to analyze the image representation characteristics of typical artificial scenes, using a lower feature dimension to describe the target visual information. It improves the hidden layer probabilistic semantic analysis model by combining remote sensing image features and employs a strategy of segmenting the most characteristic visual terms. This allows for visual feature representation of the over-segmented image, utilizing the rich spatial and geometric information provided by high spatial resolution to compensate for the limited descriptive power of local feature points. By combining a vision-oriented classification method with a focus model, it achieves better classification results with low-dimensional feature description. The technology in publication number CN118332499A discloses a method, device, and storage medium for building instance function recognition. The method includes the following steps: acquiring building street view image data and preprocessing the data to obtain preprocessed street view data; relocating buildings from the preprocessed street view data to obtain relocated building data; and classifying the relocated building data based on multimodal data. The technology with application number 202010481052.0 discloses a method for identifying urban villages by fusing high-resolution remote sensing imagery and street view imagery. It includes: acquiring high-resolution remote sensing imagery of a selected area for pre-extraction and multi-scale segmentation; acquiring multi-view street view imagery of the selected area, constructing optimal street spatial quality assessment models for each viewpoint and corresponding optimal street view image feature combinations; extracting multiple types of high-resolution remote sensing imagery features for each plot to be classified; extracting multiple street quality features for each plot to be classified; heterogeneously fusing the multiple types of high-resolution remote sensing imagery features and multiple street quality features; and using machine learning algorithms to determine whether the remaining plots are urban villages. The patent extracts multiple street view imagery features, combines multiple feature vectors in a serial manner, then fuses them based on high-resolution remote sensing imagery features, and uses a random forest to construct a classifier to determine whether the remaining plots are urban villages. This method integrates street view imagery information into the urban village extraction process based on high-resolution remote sensing imagery. This allows the spectral, shape, texture, building structure, and scene features provided by the bird's-eye view high-resolution remote sensing imagery to complement the street space quality information provided by the human-centric street view imagery, thus constructing a more distinctive feature space and improving the accuracy of urban village identification.

[0003] The first two methods mentioned above both use a single data source, relying on isolated remote sensing spectral or street view deep learning models. This makes them unable to handle issues involving objects with similar spectra but different appearances, such as government buildings and hotels having similar spectra or objects with different appearances, or significant differences between the main entrance and side facade of a school. The last method identifies urban village types within urban areas, but its accuracy is limited to the plot scale and lacks the fine-grained identification of each land type, making it impossible to identify and judge every single building. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method, system, electronic device, and storage medium for identifying public buildings based on multimodal features of remote sensing images and street view images, thereby solving the problems of existing methods being unable to handle heterogeneous objects with the same spectrum and insufficient identification accuracy for each type of building when using a single data source.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A multimodal feature-based public building identification method based on remote sensing imagery and street view images includes:

[0007] Collect remote sensing images and street view images of the area to be identified to obtain the data to be identified;

[0008] The data to be identified is input into a pre-trained public building identification model for identification, thereby obtaining the public building identification result; the training process of the public building identification model includes:

[0009] Collect the remote sensing images and street view images of the study area;

[0010] The remote sensing image is preprocessed and features are extracted to obtain remote sensing features; the remote sensing features include: red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, and vegetation normalization index.

[0011] The street view image is segmented into several image blocks using a YOLOv8 network with an improved C2f_DCNv2 module.

[0012] The image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features.

[0013] Based on the OCR method, the CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image block to obtain OCR features;

[0014] The remote sensing features, HOG features, SURF features, and OCR features are stitched together to obtain stitched features;

[0015] The public building recognition model is obtained by using the spliced ​​features to classify and train the combined model of random forest and SVM according to the multi-classification strategy.

[0016] Preferably, the remote sensing image is preprocessed and feature extracted to obtain remote sensing features, including:

[0017] The remote sensing images are radiometrically calibrated and atmospherically corrected to obtain preprocessed images;

[0018] The preprocessed image is orthorectified based on the pre-collected reference image and DEM data to obtain a planar orthorectified image.

[0019] The planar orthophoto image, along with the pre-collected panchromatic and multispectral images, are fused to obtain a fused image.

[0020] The standard deviation of the red band, the standard deviation of the green band, the standard deviation of the blue band, the brightness index, the shape index, the information entropy, the contrast, and the vegetation normalization index of the fused image are calculated and integrated to obtain the remote sensing features.

[0021] Preferably, the training process of the YOLOv8 network includes:

[0022] The pre-collected street view images are categorized and labeled with public buildings to obtain labeled images;

[0023] The label image is converted into JSON format to obtain JSON data; the JSON data includes: polygon vertex coordinates and category information;

[0024] Construct the C2f_DCNv2 improved module; the expression of the C2f_DCNv2 improved module is: ;in, This is the output of the improved C2f_DCNv2 module; For convolution kernel; This is the offset; The location of the convolution kernel; It is a two-dimensional offset matrix; The normalized adjustment term; K is the number of sampling locations; The input features of the C2f_DCNv2 improved module;

[0025] Construct and integrate the box loss calculation formula and the mask loss calculation formula to obtain the head loss function;

[0026] The original C2f module of the YOLOv8 network is replaced by the C2f_DCNv2 improved module, and the YOLOv8 network is trained using the header loss function based on the JSON data to obtain the trained YOLOv8 network.

[0027] Preferably, the image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features, including:

[0028] The key points are obtained by calculating the Hessian matrix based on the image patches;

[0029] Haar wavelet response calculation was performed on a 20×20 region around the key point to obtain the SURF descriptor;

[0030] K-means clustering is performed on the SURF descriptors to obtain SURF features.

[0031] Preferably, the image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features, including:

[0032] HOG gradient calculation is performed on the image patch to obtain a gradient direction histogram;

[0033] The image block is divided into 8×8 pixel cell units, and each 2×2 cell unit is defined as a patch. The gradient orientation histogram within the patch is then L2 normalized to obtain the HOG feature.

[0034] Preferably, the image patch is subjected to text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation using a CRNN network according to the OCR method to obtain OCR features, including:

[0035] The image patch is subjected to sequence feature extraction using a ResNet network to obtain a feature map;

[0036] The feature map is modeled using an RNN sequence modeling method to obtain encoded data;

[0037] The encoded data is CTC decoded to obtain decoded data;

[0038] Based on the TF-IDF values ​​of the category keywords in the decoded data, keyword frequency data is obtained;

[0039] Based on the decoded data, the area ratio of the text region is calculated to obtain spatial distribution data;

[0040] The minimum bounding rectangle angle is calculated based on the decoded data to obtain the text direction data;

[0041] The keyword frequency data, spatial distribution data, and text direction data are integrated to obtain the OCR features.

[0042] Preferably, the public building recognition model is obtained by using the concatenated features to perform classification training on a combined model that integrates random forest and SVM according to a multi-classification strategy, including:

[0043] Random forest is used to filter the concatenated features to obtain dimensionality-reduced features;

[0044] The reduced-dimensional features are classified using an SVM kernel function based on a multi-classification strategy to obtain the public building recognition model.

[0045] Preferably, a multimodal feature-based public building recognition system based on remote sensing imagery and street view images includes:

[0046] The image acquisition module is used to collect remote sensing images and street view images of the area to be identified, and obtain the data to be identified.

[0047] The public building identification module is used to input the data to be identified into a pre-trained public building identification model for identification, and obtain the public building identification result.

[0048] Preferably, an electronic device includes: at least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the aforementioned multimodal feature-based public building recognition method based on remote sensing images and street view images.

[0049] Preferably, a non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the aforementioned multimodal feature-based public building recognition method based on remote sensing images and street view images.

[0050] The present invention discloses the following technical effects:

[0051] This invention provides a method, system, electronic device, and storage medium for public building recognition based on multimodal features of remote sensing images and street view images. By extracting remote sensing features, HOG features, SURF features, and OCR features, it solves the problems of existing methods that cannot handle heterogeneous objects with the same spectrum and have insufficient recognition accuracy for each type of building when using a single data source. It realizes the fusion of macroscopic structural information of remote sensing images with microscopic semantic details and text recognition information features of street view images. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A schematic diagram illustrating the multimodal feature-based public building recognition process based on remote sensing images and street view images provided in an embodiment of the present invention;

[0054] Figure 2 A building identification flowchart provided for embodiments of the present invention;

[0055] Figure 3 This is a schematic diagram of street view image instance segmentation provided in an embodiment of the present invention;

[0056] Figure 4(a) shows the results of remote sensing image recognition;

[0057] Figure 4(b) shows a partial street view location map;

[0058] Figure 4(c) shows the model prediction results. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] The purpose of this invention is to provide a method, system, electronic device, and storage medium for identifying public buildings based on multimodal features of remote sensing images and street view images, which solves the problems of existing methods that cannot handle objects of the same spectrum and have insufficient recognition accuracy for each type of building when using a single data source.

[0061] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0062] Figure 1 This is a schematic diagram of a multimodal feature-based public building recognition process based on remote sensing images and street view images, provided in an embodiment of the present invention. Figure 2 A building identification flowchart provided for an embodiment of the present invention, such as Figure 1 and Figure 2 As shown, this invention provides a multimodal feature-based public building identification method based on remote sensing imagery and street view images, comprising:

[0063] Step 100: Collect remote sensing images and street view images of the area to be identified to obtain the data to be identified;

[0064] Step 200: Input the data to be identified into a pre-trained public building recognition model for identification, and obtain the public building recognition result; the training process of the public building recognition model includes:

[0065] Step 201: Acquire the remote sensing images and street view images of the study area;

[0066] Step 202: Preprocess and extract features from the remote sensing image to obtain remote sensing features; the remote sensing features include: red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, and vegetation normalization index;

[0067] Step 203: Use the YOLOv8 network with the C2f_DCNv2 improved module to perform instance segmentation on the street view image to obtain several image blocks;

[0068] Step 204: Perform descriptor calculation, clustering, HOG gradient calculation, and normalization on the image patch to obtain HOG features and SURF features;

[0069] Step 205: Using the OCR method, the CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image block to obtain OCR features;

[0070] Step 206: Perform a stitching process on the remote sensing features, the HOG features, the SURF features, and the OCR features to obtain stitched features;

[0071] Step 207: Based on the multi-classification strategy, the combined model of random forest and SVM is trained for classification using the spliced ​​features to obtain the public building recognition model.

[0072] Specifically, the remote sensing image is preprocessed and its features are extracted to obtain remote sensing features, including:

[0073] The remote sensing images are radiometrically calibrated and atmospherically corrected to obtain preprocessed images;

[0074] The preprocessed image is orthorectified based on the pre-collected reference image and DEM data to obtain a planar orthorectified image.

[0075] The planar orthophoto image, along with the pre-collected panchromatic and multispectral images, are fused to obtain a fused image.

[0076] The standard deviation of the red band, the standard deviation of the green band, the standard deviation of the blue band, the brightness index, the shape index, the information entropy, the contrast, and the vegetation normalization index of the fused image are calculated and integrated to obtain the remote sensing features.

[0077] Furthermore, the training process of the YOLOv8 network includes:

[0078] The pre-collected street view images are categorized and labeled with public buildings to obtain labeled images;

[0079] The label image is converted into JSON format to obtain JSON data; the JSON data includes: polygon vertex coordinates and category information;

[0080] Construct the C2f_DCNv2 improved module; the expression of the C2f_DCNv2 improved module is: ;in, This is the output of the improved C2f_DCNv2 module; For convolution kernel; This is the offset; The location of the convolution kernel; It is a two-dimensional offset matrix; The normalized adjustment term; K is the number of sampling locations; The input features of the C2f_DCNv2 improved module;

[0081] Construct and integrate the box loss calculation formula and the mask loss calculation formula to obtain the head loss function;

[0082] The original C2f module of the YOLOv8 network is replaced by the C2f_DCNv2 improved module, and the YOLOv8 network is trained using the header loss function based on the JSON data to obtain the trained YOLOv8 network.

[0083] Specifically, the image patch undergoes descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features, including:

[0084] The key points are obtained by calculating the Hessian matrix based on the image patches;

[0085] Haar wavelet response calculation was performed on a 20×20 region around the key point to obtain the SURF descriptor;

[0086] K-means clustering is performed on the SURF descriptors to obtain SURF features.

[0087] Further, the image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features, including:

[0088] HOG gradient calculation is performed on the image patch to obtain a gradient direction histogram;

[0089] The image block is divided into 8×8 pixel cell units, and each 2×2 cell unit is defined as a patch. The gradient orientation histogram within the patch is then L2 normalized to obtain the HOG feature.

[0090] Specifically, based on the OCR method, a CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image patch to obtain OCR features, including:

[0091] The image patch is subjected to sequence feature extraction using a ResNet network to obtain a feature map;

[0092] The feature map is modeled using an RNN sequence modeling method to obtain encoded data;

[0093] The encoded data is CTC decoded to obtain decoded data;

[0094] Based on the TF-IDF values ​​of the category keywords in the decoded data, keyword frequency data is obtained;

[0095] Based on the decoded data, the area ratio of the text region is calculated to obtain spatial distribution data;

[0096] The minimum bounding rectangle angle is calculated based on the decoded data to obtain the text direction data;

[0097] The keyword frequency data, spatial distribution data, and text direction data are integrated to obtain the OCR features.

[0098] Furthermore, based on a multi-classification strategy, the combined model fusing random forest and SVM is trained for classification using the concatenated features to obtain the public building recognition model, including:

[0099] Random forest is used to filter the concatenated features to obtain dimensionality-reduced features;

[0100] The reduced-dimensional features are classified using an SVM kernel function based on a multi-classification strategy to obtain the public building recognition model.

[0101] Optionally, public buildings can be identified by utilizing remote sensing image features, street view image features, and text features extracted through model fusion. Public buildings include office buildings, commercial buildings, tourist buildings, science, education, culture and health buildings, communication buildings, and transportation buildings. This embodiment covers office buildings, commercial buildings, and science, education, culture and health buildings.

[0102] Specifically, the process of acquiring high-resolution images, downloading street view images, and selecting a test area includes the following steps: acquiring high-resolution remote sensing images with a spatial resolution better than 1m; using Python to write a web crawler to download street view data of the corresponding area in batches, with sampling points selected at the center of the road, and two images taken vertically from each sampling point at 50-meter intervals; and determining the scope of the test area by selecting densely built-up urban areas or city centers.

[0103] Furthermore, high-resolution satellite image preprocessing and feature extraction include the following steps:

[0104] 1) Radiometric calibration: Using ENVI software, input the image to be radiometrically calibrated and the image description file. In the Toolbox, select RadiometricCorrection>RadiometricCalibration. Perform radiometric calibration to obtain the radiometrically corrected surface reflectance.

[0105] 2) Atmospheric correction: In the Toolbox, select RadiometricCorrection>AtmosphericCorrectionModule>FLAASHAatmosphericCorrection to perform atmospheric correction and obtain the atmospherically corrected surface reflectance.

[0106] 3) Orthorectification: Using ENVI software, input the image to be orthorectified, the reference image, and the DEM data, output GCPs (Ground Control Points) and perform orthoprojection correction to eliminate the influence of terrain or distortion caused by camera orientation, and generate a planar orthophoto.

[0107] 4) Image fusion: Select ImageSharpening>Gram-SchmidtPanSharpeningClassic, input panchromatic image and multispectral image respectively, and obtain the fused image result.

[0108] 5) Feature extraction: Calculate six remote sensing feature parameters of the experimental area image: standard deviation, brightness, shape index, information entropy, contrast and NDVI, to obtain the remote sensing feature vector.

[0109] Standard deviation This reflects the dispersion of spectral reflectance in a building area; a larger value indicates greater differences in internal materials. The formula is:

[0110]

[0111] in, Let i represent the reflectance value of pixel i in a certain wavelength band. Calculate the reflectance in the red band separately. Green band and blue band μ represents the mean value of all pixels in this band; N represents the total number of pixels within the building area.

[0112] Brightness B represents the overall lightness or darkness of a building surface, and the formula is:

[0113]

[0114] in, Represents pixels Reflectivity values ​​in the red band, Represents pixels Reflectance value in the green band, Represents pixels In the blue band reflectivity value, N represents the total number of pixels within the building area.

[0115] The Shape Index (SI) measures the complexity of a building's shape. SI ≥ 1; the closer the value is to 1, the closer the shape is to a circle, such as a dome. A higher value indicates a more irregular shape. The formula is:

[0116]

[0117] Perimeter represents the perimeter of the building polygon; Area represents the area of ​​the building polygon.

[0118] Information entropy (IE) describes the randomness of a building's surface texture. A high entropy value indicates a complex texture. The formula for calculating information entropy is:

[0119]

[0120] in, It is the probability of gray level i appearing in the image.

[0121] The formula for calculating contrast ratio C is:

[0122]

[0123] in, It is the position in the gray-level co-occurrence matrix. The co-occurrence probability in the two directions (0°, 90°) was calculated. In this embodiment, the contrast in two directions (0°, 90°) was calculated, and then the average value was calculated.

[0124] Normalized index of vegetation indicates The formula is:

[0125]

[0126] Wherein, RED is the red band reflectance and NIR is the near-infrared band reflectance.

[0127] Obtain the feature remote sensing vector .

[0128] Specifically, street view image classification instance segmentation and feature extraction mainly include:

[0129] 1) Filter the downloaded street view images, remove invalid images, unify the size and format, and convert them into JPG image format;

[0130] 2) Public building classification and labeling were performed on the images using dedicated software. The YOLOv8 instance segmentation model was employed, and real-time high-precision segmentation of street scene buildings was achieved through dynamic label assignment and prototype mask optimization. The segmentation results are referenced. Figure 3 :

[0131] In specialized software, public buildings are categorized and labeled in the images. Each polygon is assigned a category label, and text areas are labeled. Text areas (such as building names) in the images are labeled with rectangles and the label "text" is set.

[0132] Output format: Each image generates a corresponding JSON file, containing polygon vertex coordinates and category information;

[0133] Constructing the C2f_DCNv2 module: Increase the number of deformable convolutional layers, add a modulation mechanism to the deformable convolution module, and the improved convolution feature matrix formula is as follows:

[0134]

[0135] Where W is the convolution kernel, K represents the position of each convolution kernel, and K is the number of sampling positions for the convolution kernel. It is a newly added two-dimensional offset matrix. It is the adjustment term after normalization.

[0136] Detection Head Loss Function: The total loss function L of the model is jointly established by the detection box loss and the mask loss. The WIoU loss function is used to calculate the detection box loss, and the formula is as follows:

[0137]

[0138] in, It is the detection frame loss; It is masking loss.

[0139] The loss of the detection box is calculated using the WIoU loss function. The formula for the WIoUv3 loss function is as follows:

[0140]

[0141]

[0142] In this context, x and y are the center coordinates of the prediction box; , W represents the center coordinates of the true bounding box. g and H g is the minimum size of the closed anchor frame; r is the gradient gain. The outlier is determined by the hyperparameter. , Control, when = hour, This makes r=1.

[0143] 3) Extract SURF and HOG features:

[0144] The labeled public building areas are cropped into independent image patches. SURF keypoints are detected and descriptors are calculated. Hessian matrix is ​​then used to detect keypoints. :

[0145]

[0146] in, The result is the Gaussian second derivative convolution. When det(H) > the preset threshold, it is determined as a key point. p and q are the pixel positions in the image, and σ is the kernel scale parameter, which controls the sensitivity of feature point detection.

[0147] Descriptor generation: Calculate the Haar wavelet response for a 20×20 region around the keypoint to form a 64-dimensional vector;

[0148] K-means clustering is performed on the SURF descriptors of all images to generate a visual dictionary (i.e., SURF features), and the word frequency histogram of each image is calculated.

[0149] HOG gradient calculation: Scale the image patch to 64×128 pixels and calculate the gradient direction histogram for each 8×8 pixel unit. The formula is:

[0150]

[0151]

[0152] Where I(x,y) represents the pixel intensity value (grayscale or single channel) of the image at (x,y). This represents an approximate value of the gradient in the x-direction. The gradient approximates the gradient in the y-direction. Magnitude represents the gradient magnitude, reflecting the edge strength. Orientation represents the gradient direction, ranging from 0° to 180°, and is used for histogram statistics.

[0153] Cellular Units and Block Normalization: The image is divided into 8×8 pixel cellular units, and a 9-bin gradient orientation histogram is calculated for each unit; 2×2 cellular units are grouped into a block (16×16 pixels), and L2 normalization is performed on the histogram within the block to obtain HOG features. :

[0154]

[0155] Where v represents an unnormalized histogram vector, which is usually 36-dimensional, and is formed by concatenating four 9-dimensional histograms. To minimize the value, for example, 1e-5, to prevent division by zero; after normalization, the value is pruned and then renormalized.

[0156] The final HOG feature dimension is 1764 (64×128 image).

[0157] 4) Use OCR to detect the labeled text region and employ a CRNN model for text recognition:

[0158] CNN Feature Extraction: Using ResNet to extract sequence features and output feature maps. T is the time step, and D is the feature dimension;

[0159] RNN sequence modeling: the hidden state of the RNN at each time step t The updated formula is:

[0160]

[0161]

[0162] in, It is the input vector at time step t. F is the hidden state at time step t; F is the temporal feature matrix extracted by the CNN, with shape (T, D). It is the input feature vector at the t-th time step (i.e., the t-th row of F).

[0163] CTC Decoding: Solving the problem of variable-length sequence alignment, loss function for:

[0164]

[0165] in, B represents the path, and B represents the mapping to remove duplicate characters and spaces.

[0166] Keyword frequency: Set the statistical category of keywords value:

[0167]

[0168] Where t represents a specific keyword, such as "government," d represents the current document (i.e., the text content of a single street view image), and N represents the total number of documents in the corpus, i.e., all street view images. : The number of documents containing the keyword t.

[0169] Spatial distribution: Calculate the area ratio of the text region The formula is:

[0170]

[0171] TextArea represents the total number of pixels in the text area (calculated using the OCR detection box); ImageArea represents the total number of pixels in the street view image (e.g., 256×256=65536).

[0172] Text orientation: Calculate text slant based on the angle of the minimum bounding rectangle. (0° to 90°).

[0173]

[0174] Where B represents the set of vertex coordinates of the text detection box, and minAreaRect is the function for calculating the minimum bounding rectangle. The angle of inclination of the rectangle ranges from -90° to 90°.

[0175] Furthermore, the training dataset is input into the model for optimization training, which mainly includes:

[0176] 1) Feature-level fusion: Remote sensing feature vectors, street view visual element feature vectors, and text feature vectors are concatenated and fused with standardized remote sensing, street view, and OCR features.

[0177]

[0178] It represents 8-dimensional remote sensing features (standard deviation, brightness, shape index, etc.). Represents 1864-dimensional street view features (SURF+HOG); This represents 52-dimensional OCR features (TF-IDF, text coverage, etc.).

[0179] 2) Model classification training: A combination of Random Forest and SVM is used for training and classification, mainly including the following steps:

[0180] Random Forest (RF) Feature Selection: Input concatenated features (Finput), output feature importance ranking, retain the top K most important features, and obtain the dimensionality-reduced features (Fselected).

[0181] The SVM kernel chosen is the RBF kernel. The kernel function maps the original low-dimensional feature space to a high-dimensional space. The RBF kernel function formula is as follows:

[0182]

[0183]

[0184] in, , Representing two eigenvectors, This is the output of the RBF kernel function. Controlling the kernel function width determines the decision boundary complexity; This represents the Euclidean distance between eigenvectors.

[0185] Multi-class strategy: Combine categories pairwise, train n binary SVM classifiers, and select the category with the most wins during prediction;

[0186]

[0187] Where c represents the number of categories.

[0188] Furthermore, the sample data is input into the model to obtain the building classification prediction results, as shown in Figures 4(a) to 4(c):

[0189] 1) Input the preprocessed dataset into the model for recognition.

[0190] 2) Obtain the model prediction results.

[0191] 3) The accuracy of the model is evaluated using precision (P), recall (R), and average F1-Score.

[0192]

[0193]

[0194]

[0195] Wherein, TP: positive samples predicted as positive by the model; TN: negative samples predicted as negative by the model; FP: negative samples predicted as positive by the model; FN: positive samples predicted as negative by the model.

[0196] Optionally, high-resolution image acquisition, street view image download, and selection of a test area include the following steps:

[0197] 1) Acquire Gaofen-3 remote sensing images with a spatial resolution of 0.5 meters;

[0198] 2) A web crawler was written in Python to download street view data of the corresponding area in batches. The sampling points were selected at the center of the road around the buildings. Through the Baidu Map API interface, two images were collected in the vertical direction of the road at each sampling point, resulting in no less than 20,000 images. The images were named according to the coordinate information of the sampling points to link the remote sensing data and the street view data, for example, 120.83826_24.67291_1 and 120.83826_24.67291_2.

[0199] 3) Determine the scope of the test area and select areas with abundant street view images in the central urban area.

[0200] Preferably, feature extraction involves calculating six remote sensing feature parameters of the experimental area image: standard deviation, brightness, shape index, information entropy, contrast, and NDVI, to obtain a remote sensing feature vector. :

[0201] 1) In ENVI software, load the image data, select BasicTools>Statistics>ComputeStatistics in the main menu, select Standard Deviation, and obtain the standard deviation image.

[0202] 2) In ENVI software, select BasicTools>Statistics>ComputeStatistics, and then select Histogram to calculate the brightness image.

[0203] 3) In ENVI software, calculate the NDVI index of the test area. Using Toolbox > BandAlgebra > BandMath, enter the expression (float(b4)-b3) / (b4+b3) in the Enteranexpression box. Here, b4 is the near-infrared band, b3 is the red band, and float(b4) converts the near-infrared band b4 to a floating-point number. Note that the data type used here is integer, so data type conversion is necessary; simply convert one of the variables to a floating-point number.

[0204] Specifically, in ArcGIS software, input three single-band images of the test area, select ArcToolbox > Spatial Analyst > Map Algebra > Raster Calculator, and enter the formula: "R"*0.299+"G"*0.587+"R"*0.114; where R represents the red single-band image, 0.299 is the red band coefficient; G represents the green single-band image, 0.587 is the green band coefficient; and B represents the blue single-band image, 0.114 is the blue band coefficient. The normalized grayscale image of the test area is calculated, and the information entropy and contrast of the samples are calculated.

[0205] Preferably, the street view image is subjected to classification instance segmentation and feature extraction, which mainly includes:

[0206] The downloaded street view images are filtered, invalid images are removed, and the size and format are standardized before being converted into JPG image format.

[0207] In LabelMe software, images are categorized and labeled with public buildings and text. In LabelMe software, select the filtered street view image training data, outline the public buildings, click on the image boundary to create polygon vertices, and form a closed region.

[0208] Each polygon is assigned a category label, and public buildings are divided into three categories: office buildings, science, education, culture and health buildings, and commercial buildings.

[0209] Perform instance segmentation and classification on street view images.

[0210] Further, SURF features, HOG features, and OCR text features are extracted. OCR text information feature extraction mainly includes: annotating text regions by creating rectangular annotations of text regions (such as building names) in the image and assigning the label "text"; and detecting text information in the annotated images. The training dataset is then input into the model for optimization training, mainly including:

[0211] Key parameter values: Table 1 shows the range of values ​​for key parameters, which can be adjusted according to actual circumstances.

[0212] Table 1

[0213] Set OCR keywords; Table 2 shows some keywords in different categories:

[0214] Table 2

[0215] The feature importance classification rules have been optimized. Table 3 shows some examples of the rules:

[0216] Table 3

[0217] Furthermore, data input: the preprocessed dataset is input into the model for recognition; result output: the model prediction result is obtained; prediction result accuracy statistics: Table 4 is a statistical table of the building recognition accuracy in this embodiment.

[0218] Table 4

[0219] Architecture Total quantity Correct quantity Misjudgment type Error rate accuracy office buildings 26 19 Commercial buildings 26.92% 73.08% Science, Education, Culture and Health 92 70 office buildings 23.91% 76.09% Commercial buildings 310 283 Office buildings / Science, education, culture and health 8.71% 91.29% total 428 372 13.08% 86.92%

[0220] Therefore, the overall prediction accuracy rate for building categories in this embodiment is 86.92%.

[0221] As an optional implementation, this embodiment also provides a multimodal feature-based public building recognition system based on remote sensing imagery and street view images, including:

[0222] The image acquisition module is used to collect remote sensing images and street view images of the area to be identified, and obtain the data to be identified.

[0223] The public building identification module is used to input the data to be identified into a pre-trained public building identification model for identification, and obtain the public building identification result.

[0224] As an optional implementation, this embodiment also provides an electronic device, including: at least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform the aforementioned multimodal feature public building recognition method based on remote sensing images and street view images.

[0225] As an optional implementation, this embodiment also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the aforementioned multimodal feature-based public building recognition method based on remote sensing images and street view images.

[0226] The beneficial effects of this invention are as follows:

[0227] This invention deeply integrates the macroscopic structural information of remote sensing images with the microscopic semantic details and text recognition information features of street view images through remote sensing features, HOG features, SURF features extraction, and OCR features. It supplements facade details and functional identifiers, and uses geographic coordinate matching to ensure that the remote sensing and street view features of the same building are spatially consistent, thereby reducing the probability of misclassification.

[0228] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0229] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for identifying public buildings based on multimodal features of remote sensing images and street view images, characterized in that, include: Collect remote sensing images and street view images of the area to be identified to obtain the data to be identified; The data to be identified is input into a pre-trained public building identification model for identification, and the public building identification result is obtained. The training process of the public building identification model includes: Collect the remote sensing images and street view images of the study area; The remote sensing image is preprocessed and features are extracted to obtain remote sensing features; the remote sensing features include: red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, and vegetation normalization index. The original C2f module of the YOLOv8 network is replaced with the C2f_DCNv2 improved module. The YOLOv8 network fused with the C2f_DCNv2 improved module is then used to perform instance segmentation on the street view image, resulting in several image blocks. The expression for the C2f_DCNv2 improved module is as follows: ;in, This is the output of the improved C2f_DCNv2 module; For convolution kernel; This is the offset; The location of the convolution kernel; It is a two-dimensional offset matrix; The normalized adjustment term; K is the number of sampling locations; The input features of the C2f_DCNv2 improved module; The image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features. Based on the OCR method, the CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image block to obtain OCR features; The remote sensing features, HOG features, SURF features, and OCR features are stitched together to obtain stitched features; The public building recognition model is obtained by using the spliced ​​features to classify and train the combined model of random forest and SVM according to the multi-classification strategy.

2. The method for identifying public buildings based on multimodal features of remote sensing images and street view images according to claim 1, characterized in that, The remote sensing image is preprocessed and its features are extracted to obtain remote sensing features, including: The remote sensing images are radiometrically calibrated and atmospherically corrected to obtain preprocessed images; The preprocessed image is orthorectified based on the pre-collected reference image and DEM data to obtain a planar orthorectified image. The planar orthophoto image, along with the pre-collected panchromatic and multispectral images, are fused to obtain a fused image. The standard deviation of the red band, the standard deviation of the green band, the standard deviation of the blue band, the brightness index, the shape index, the information entropy, the contrast, and the vegetation normalization index of the fused image are calculated and integrated to obtain the remote sensing features.

3. The method for identifying public buildings based on multimodal features of remote sensing images and street view images according to claim 1, characterized in that, The training process of the YOLOv8 network includes: The pre-collected street view images are categorized and labeled with public buildings to obtain labeled images; The label image is converted into JSON format to obtain JSON data; the JSON data includes: polygon vertex coordinates and category information; Construct the aforementioned C2f_DCNv2 improved module; Construct and integrate the box loss calculation formula and the mask loss calculation formula to obtain the head loss function; The original C2f module of the YOLOv8 network is replaced by the C2f_DCNv2 improved module, and the YOLOv8 network is trained using the header loss function based on the JSON data to obtain the trained YOLOv8 network.

4. The method for identifying public buildings based on multimodal features of remote sensing images and street view images according to claim 1, characterized in that, The image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features, including: The key points are obtained by calculating the Hessian matrix based on the image patches; Haar wavelet response calculation was performed on a 20×20 region around the key point to obtain the SURF descriptor; K-means clustering is performed on the SURF descriptors to obtain SURF features.

5. The method for identifying public buildings based on multimodal features of remote sensing images and street view images according to claim 1, characterized in that, The image patch is subjected to descriptor calculation, clustering, HOG gradient calculation, and normalization to obtain HOG features and SURF features, including: HOG gradient calculation is performed on the image patch to obtain a gradient direction histogram; The image block is divided into 8×8 pixel cell units, and each 2×2 cell unit is defined as a patch. The gradient orientation histogram within the patch is then L2 normalized to obtain the HOG feature.

6. The method for identifying public buildings based on multimodal features of remote sensing images and street view images according to claim 1, characterized in that, Based on the OCR method, a CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image patch to obtain OCR features, including: The image patch is subjected to sequence feature extraction using a ResNet network to obtain a feature map; The feature map is modeled using an RNN sequence modeling method to obtain encoded data; The encoded data is CTC decoded to obtain decoded data; Based on the TF-IDF values ​​of the category keywords in the decoded data, keyword frequency data is obtained; Based on the decoded data, the area ratio of the text region is calculated to obtain spatial distribution data; The minimum bounding rectangle angle is calculated based on the decoded data to obtain the text direction data; The keyword frequency data, spatial distribution data, and text direction data are integrated to obtain the OCR features.

7. The method for identifying public buildings based on multimodal features of remote sensing images and street view images according to claim 1, characterized in that, Based on a multi-classification strategy, the combined model fusing random forest and SVM is trained using the concatenated features to obtain the public building recognition model, including: Random forest is used to filter the concatenated features to obtain dimensionality-reduced features; The reduced-dimensional features are classified using an SVM kernel function based on a multi-classification strategy to obtain the public building recognition model.

8. A multimodal feature-based public building recognition system based on remote sensing imagery and street view images, characterized in that, The system applied to the multimodal feature-based public building recognition method based on remote sensing imagery and street view images as described in claim 1, the system comprising: The image acquisition module is used to collect remote sensing images and street view images of the area to be identified, and obtain the data to be identified. The public building identification module is used to input the data to be identified into a pre-trained public building identification model for identification, and obtain the public building identification result.

9. An electronic device, characterized in that, include: At least one processor, and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to enable the processor to perform a multimodal feature-based public building recognition method based on remote sensing imagery and street view imagery, as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute any one of claims 1 to 7, a method for identifying public buildings based on multimodal features of remote sensing images and street view images.

Images