Multi-modal feature public building identification method and system based on remote sensing image and streetscape image, electronic equipment and storage medium

Abstract

The invention belongs to the field of computer vision and ground object recognition and classification, and provides a multi-modal feature public building recognition method and system based on a remote sensing image and a streetscape image, electronic equipment and a storage medium, and the method comprises the steps: collecting the remote sensing image and the streetscape image, and recognizing a public building; the model training process comprises image acquisition, remote sensing feature extraction, instance segmentation, HOG feature and SURF feature extraction, OCR feature extraction, feature splicing and classification training; through remote sensing features, HOG features, SURF feature extraction and OCR features, macroscopic structure information of a remote sensing image and microscopic semantic details and character recognition information features of a streetscape image are deeply fused, facade details and function identifiers are supplemented, geographic coordinate matching is adopted to ensure that the remote sensing feature space and the streetscape feature space of the same building are consistent, and the accuracy of the remote sensing feature space and the streetscape feature space is improved. And the classification misjudgment probability is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and ground object recognition and classification, and particularly relates to a multi-modal feature public building recognition method and system based on remote sensing images and street view images, an electronic device and a storage medium. BACKGROUND

[0002] Traditional building classification methods usually rely on a single data source and are based on remote sensing or street view, which has the problems of semantic fragmentation and feature limitation. For example, the technology with the application number CN202410114236.1 combines the data characteristics of remote sensing images to analyze the image performance characteristics of typical artificial scenes, uses a lower feature dimension to describe the target visual information, improves the hidden layer probability semantic analysis model based on the characteristics of remote sensing images, uses the strategy of dividing the most characteristic words of vision, describes the visual features of the segmented images, uses the rich image spatial information and geometric structure information brought by high spatial resolution to make up for the limitation of the limited description of local feature points, and combines the visual-oriented classification method with the focus model to achieve better classification effect with low-dimensional feature description. The technology with the publication number CN118332499A discloses a building instance function recognition method, device and storage medium, which includes the steps of: obtaining building street view picture data and preprocessing the data to obtain preprocessed street view data; performing building repositioning on the preprocessed street view data to obtain repositioned building data; and performing function classification on the repositioned building data based on multi-modal data. The technology with the application number 202010481052.0 discloses a village-in-city recognition method fusing high-resolution remote sensing images and street view images. The method includes: obtaining high-resolution remote sensing images of a selected area for pre-extraction and multi-scale segmentation; obtaining multi-view street view images of the selected area, constructing an optimal street space quality evaluation model under each view and a corresponding optimal street view image feature combination mode; extracting multi-class high-resolution remote sensing image features of each to-be-classified plot; extracting multiple street quality features of each to-be-classified plot; heterogeneously fusing the multi-class high-resolution remote sensing image features and the multiple street quality features, and using a machine learning algorithm to determine whether the remaining plots are villages-in-city. The patent extracts multiple street view image features, combines multiple feature vectors in a serial manner, fuses high-resolution remote sensing image features, and uses a random forest to construct a classifier to determine whether the remaining plots are villages-in-city. This method integrates street view image information into the village-in-city extraction process based on high-resolution remote sensing images, so that the spectral, shape, texture, building structure and scene features provided by the bird's eye view high-resolution remote sensing images are complementary to the street space quality information provided by the human perspective street view images, a more distinguishable feature space is constructed, and the village-in-city recognition accuracy is improved.

[0003] The above first two methods use a single data source, and use remote sensing spectrum or street view deep learning model alone, which cannot process the same spectrum different objects problem, such as government building and hotel spectrum similar or the same object different scene problem, such as the difference between the school front door and the side facade. The last method identifies the village type in the urban area, and the precision is at the plot scale, which cannot identify and judge each building. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a multi-modal feature public building recognition method, system, electronic equipment and storage medium based on remote sensing image and street view image, which solves the problem that the existing method cannot process the same spectrum different objects and the recognition accuracy of each building is insufficient.

[0005] To achieve the above purpose, the present application provides the following scheme:

[0006] A multi-modal feature public building recognition method based on remote sensing image and street view image, comprising:

[0007] Collecting remote sensing images and street view images of the to-be-identified area to obtain to-be-identified data;

[0008] Inputting the to-be-identified data into a pre-trained public building recognition model for identification to obtain a public building recognition result; the training process of the public building recognition model comprises:

[0009] Collecting the remote sensing images and the street view images of the research area;

[0010] Pretreating and extracting features from the remote sensing images to obtain remote sensing features; the remote sensing features include red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, and vegetation normalized index;

[0011] Using a YOLOv8 network with a fusion C2f_DCNv2 improved module to perform instance segmentation on the street view images to obtain a plurality of image blocks;

[0012] Performing descriptor calculation, clustering processing, HOG gradient calculation and normalization processing on the image blocks to obtain HOG features and SURF features;

[0013] According to an OCR method, using a CRNN network to perform text recognition, keyword frequency statistics, spatial distribution calculation and text direction calculation on the image blocks to obtain OCR features;

[0014] Splicing the remote sensing features, the HOG features, the SURF features and the OCR features to obtain spliced features;

[0015] According to the multi-classification strategy, the splicing features are used for classification training of a combined model of a fusion random forest and an SVM, so as to obtain the public building recognition model.

[0016] Preferably, the remote sensing image is preprocessed and feature extraction is performed to obtain remote sensing features, including:

[0017] The remote sensing image is radiometrically calibrated and atmospherically corrected to obtain a preprocessed image;

[0018] According to the pre-collected reference image and DEM data, the preprocessed image is orthorectified to obtain a planar orthographic image;

[0019] The planar orthographic image and pre-collected panchromatic image and multispectral image are fused to obtain a fused image;

[0020] The red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, and vegetation normalized index of the fused image are calculated and integrated to obtain the remote sensing features.

[0021] Preferably, the training process of the YOLOv8 network includes:

[0022] The pre-collected street view images are classified and labeled for public buildings to obtain label images;

[0023] The label images are converted into JSON format to obtain JSON data; the JSON data includes polygon vertex coordinates and class information;

[0024] The C2f_DCNv2 improvement module is constructed; the expression of the C2f_DCNv2 improvement module is: wherein y(p) is the output of the C2f_DCNv2 improvement module; w k is a convolution kernel; p k is the position of the convolution kernel; Δp k is a two-dimensional offset matrix; Δm k is a normalized adjustment term; K is the number of sampling positions; x(·) is the input feature of the C2f_DCNv2 improvement module;

[0025] The head loss function is obtained by constructing and integrating the bounding box loss calculation formula and the mask loss calculation formula.

[0026] Replace the original C2f_DCNv2 module of the YOLOv8 network with the C2f_DCNv2 improved module, and train the YOLOv8 network according to the JSON data using the head loss function to obtain the trained YOLOv8 network.

[0027] Preferably, the image block is subjected to descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing to obtain HOG features and SURF features, including:

[0028] According to the image block, a Hessian matrix is calculated to obtain a key point.

[0029] A 20x20 region around the key point is subjected to Haar wavelet response calculation to obtain a SURF descriptor.

[0030] The SURF descriptor is subjected to K-means clustering to obtain a SURF feature.

[0031] Preferably, the image block is subjected to descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing to obtain HOG features and SURF features, including:

[0032] The image block is subjected to HOG gradient calculation to obtain a gradient direction histogram.

[0033] The image block is divided into 8x8 pixel cell units, 2x2 range cell units are determined as a block, and L2 normalization is performed on the gradient direction histogram in the block to obtain the HOG feature.

[0034] Preferably, according to an OCR method, a CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and character direction calculation on the image block to obtain an OCR feature, including:

[0035] A ResNet network is used to perform sequence feature extraction on the image block to obtain a feature map.

[0036] A feature map network is used to perform RNN sequence modeling on the feature map to obtain encoded data.

[0037] The encoded data is subjected to CTC decoding to obtain decoded data.

[0038] According to the decoded data, a TF-IDF value of a category keyword is calculated to obtain keyword frequency data.

[0039] According to the decoded data, a character area proportion is calculated to obtain spatial distribution data.

[0040] According to the decoding data, minimum outer rectangle angle calculation is performed to obtain character direction data;

[0041] The keyword frequency data, the spatial distribution data and the character direction data are integrated to obtain the OCR feature.

[0042] Preferably, according to a multi-classification strategy, the splicing feature is used for classification training of a combined model of a fusion random forest and an SVM to obtain the public building recognition model, including:

[0043] According to the multi-classification strategy, the SVM kernel function is used for classification processing of the reduced dimension feature to obtain the public building recognition model.

[0044] According to the multi-classification strategy, the SVM kernel function is used for classification processing of the reduced dimension feature to obtain the public building recognition model.

[0045] Preferably, a multi-modal feature public building recognition method based on remote sensing images and street view images includes:

[0046] An image collection module is configured to collect remote sensing images and street view images of a to-be-recognized region to obtain to-be-recognized data.

[0047] A public building recognition module is configured to input the to-be-recognized data into a pre-trained public building recognition model for recognition to obtain a public building recognition result.

[0048] Preferably, an electronic device includes at least one processor and a memory connected with the processor in communication; the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to implement the foregoing multi-modal feature public building recognition method based on remote sensing images and street view images.

[0049] Preferably, a non-transient computer readable storage medium stores computer instructions for enabling a computer to implement the foregoing multi-modal feature public building recognition method based on remote sensing images and street view images.

[0050] The present application discloses the following technical effects:

[0051] The present application provides a multi-modal feature public building recognition method, system, electronic device and storage medium based on remote sensing images and street view images, which extracts remote sensing features, HOG features, SURF features and OCR features, solves the problem that a single data source cannot handle the same spectrum and different materials and the insufficient recognition accuracy of each type of building in the prior art, and realizes the fusion of macro-structure information of remote sensing images and micro-semantic details and character recognition information features of street view images. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments. Obviously, the drawings described below only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0053] Figure 1 A multi-modal feature public building identification process based on remote sensing images and street view images is provided for the embodiments of the present application.

[0054] Figure 2 A building identification process diagram is provided for the embodiments of the present application.

[0055] Figure 3 A street view image instance segmentation diagram is provided for the embodiments of the present application.

[0056] Figure 4 An identification result diagram is provided for the embodiments of the present application. Figure 4 (a) is a remote sensing image identification result diagram, Figure 4 (b) is a partial street view point diagram, Figure 4 (c) is a model prediction result diagram. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0058] The purpose of the present application is to provide a multi-modal feature public building identification method, system, electronic device and storage medium based on remote sensing images and street view images, which solves the problem that the existing method cannot handle the same spectrum and different materials and the identification accuracy of each type of building is insufficient.

[0059] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0060] Figure 1 A multi-modal feature public building identification process based on remote sensing images and street view images is provided for the embodiments of the present application, Figure 2 A building identification process diagram is provided for the embodiments of the present application, such as Figure 1 and Figure 2As shown, the present application provides a multi-modal feature public building recognition method based on remote sensing images and street view images, comprising:

[0061] Step 100: Collect remote sensing images and street view images of the region to be identified to obtain to-be-identified data;

[0062] Step 200: inputting the to-be-identified data into a pre-trained public building recognition model for recognition to obtain a public building recognition result; the training process of the public building recognition model comprises:

[0063] Step 201: collecting the remote sensing images and the street view images of the research area;

[0064] Step 202: pre-processing and feature extraction of the remote sensing images to obtain remote sensing features; the remote sensing features include: red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, vegetation normalized index;

[0065] Step 203: using the YOLOv8 network of the fusion C2f_DCNv2 improvement module to perform instance segmentation on the street view images to obtain a plurality of image blocks;

[0066] Step 204: description sub-computation, clustering processing, HOG gradient computation and normalization processing of the image blocks to obtain HOG features and SURF features;

[0067] Step 205: according to the OCR method, using the CRNN network to perform text recognition, keyword frequency statistics, spatial distribution calculation and character direction calculation on the image blocks to obtain OCR features;

[0068] Step 206: splicing processing of the remote sensing features, the HOG features, the SURF features and the OCR features to obtain spliced features;

[0069] Step 207: according to the multi-classification strategy, using the spliced features to perform classification training on the combined model of fusion random forest and SVM to obtain the public building recognition model.

[0070] Specifically, the remote sensing images are pre-processed and feature-extracted to obtain remote sensing features, comprising:

[0071] Radiometric calibration and atmospheric correction are performed on the remote sensing images to obtain pre-processed images;

[0072] Orthorectification is performed on the pre-processed images according to the pre-collected reference images and DEM data to obtain planar orthographic images;

[0073] Fusing the planar orthographic image and the pre-collected panchromatic image and multispectral image to obtain a fused image;

[0074] Calculating and integrating the red band standard deviation, the green band standard deviation, the blue band standard deviation, the brightness index, the shape index, the information entropy, the contrast, and the vegetation normalized index of the fused image to obtain the remote sensing features.

[0075] Further, the training process of the YOLOv8 network comprises:

[0076] Classifying and labeling the pre-collected street view images as public buildings to obtain label images;

[0077] Converting the label images into JSON format to obtain JSON data; the JSON data comprises polygon vertex coordinates and category information;

[0078] Constructing the C2f_DCNv2 improvement module; the expression of the C2f_DCNv2 improvement module is: wherein y(p) is the output of the C2f_DCNv2 improvement module; w k is a convolution kernel; p is an offset; p k is the position of the convolution kernel; Δp k is a two-dimensional offset matrix; Δm k is a normalized adjustment term; K is the number of sampling positions; x(·) is the input feature of the C2f_DCNv2 improvement module;

[0079] Constructing and integrating a bounding box loss calculation formula and a mask loss calculation formula to obtain a head loss function;

[0080] Replacing the original C2f_DCNv2 module of the YOLOv8 network with the C2f_DCNv2 improvement module, and training the YOLOv8 network according to the JSON data using the head loss function to obtain the trained YOLOv8 network.

[0081] Specifically, the image block is subjected to descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing to obtain HOG features and SURF features, comprising:

[0082] Calculating the Hessian matrix according to the image block to obtain key points;

[0083] Calculating the Haar wavelet response of a 20x20 region around the key points to obtain SURF descriptors;

[0084] K-means clustering is performed on the SURF descriptor to obtain a SURF feature.

[0085] Further, a descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing are performed on the image block to obtain a HOG feature and a SURF feature, including:

[0086] A HOG gradient calculation is performed on the image block to obtain a gradient direction histogram.

[0087] The image block is divided into 8x8 pixel cell units, 2x2 range of the cell units are determined as a block, and L2 normalization is performed on the gradient direction histogram in the block to obtain the HOG feature.

[0088] Specifically, according to an OCR method, a CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and character direction calculation on the image block to obtain an OCR feature, including:

[0089] A ResNet network is used to extract a sequence feature from the image block to obtain a feature map.

[0090] A feature map network is used to model an RNN sequence of the feature map to obtain encoded data.

[0091] The encoded data is subjected to CTC decoding to obtain decoded data.

[0092] According to the decoded data, a TF-IDF value of a category keyword is calculated to obtain keyword frequency data.

[0093] According to the decoded data, a character area proportion calculation is performed to obtain spatial distribution data.

[0094] According to the decoded data, a minimum circumscribed rectangle angle calculation is performed to obtain character direction data.

[0095] The keyword frequency data, the spatial distribution data, and the character direction data are integrated to obtain the OCR feature.

[0096] Further, according to a multi-classification strategy, the spliced feature is used to perform classification training on a combined model of a random forest and an SVM to obtain the public building recognition model, including:

[0097] A random forest is used to perform feature screening on the spliced feature to obtain a reduced dimension feature.

[0098] According to a multi-classification strategy, an SVM kernel function is used to perform classification processing on the reduced dimension feature to obtain the public building recognition model.

[0099] Optionally, the remote sensing image features, street view image features and text features extracted by model fusion are used to identify public buildings. The public buildings include office buildings, commercial buildings, tourist buildings, science and education and health buildings, communication buildings and transportation buildings. The embodiments of the present application relate to office buildings, commercial buildings and science and education and health buildings.

[0100] Specifically, high-resolution image acquisition, street view image download, and selected test area include the following steps: acquiring high-resolution remote sensing images with a spatial resolution better than 1 m; using Python language to write a web crawler to download street view data in the corresponding area in batches, and the sampling points are selected at the center of the road, every 50 meters, and 2 images in the vertical direction of the road are collected at each sampling point; determining the test area range, selecting a densely built urban area or a city center area.

[0101] Further, high-resolution satellite image preprocessing and feature extraction include the following steps:

[0102] 1) Radiometric calibration: using ENVI software, input the image to be radiometrically calibrated and the image description file, in the Toolbox, select Radiometric Correction > Radiometric Calibration. Perform radiometric calibration to obtain the surface reflectance after radiometric correction.

[0103] 2) Atmospheric correction: in the Toolbox, select Radiometric Correction > Atmospheric Correction Module > FLAASH Atmospheric Correction, perform atmospheric correction to obtain the ground reflectance after atmospheric correction.

[0104] 3) Ortho correction: using ENVI software, input the image to be ortho corrected, the reference image and the DEM data, output GCPs (Ground Control Points) and perform ortho projection correction to eliminate the influence of terrain or camera orientation caused by deformation, etc., and generate a plane orthographic image.

[0105] 4) Image fusion: select ImageSharpening > Gram-SchmidtPanSharpeningClassic, input the panchromatic image and multispectral image to obtain the fused image result.

[0106] 5) Feature extraction: calculate the standard deviation, brightness, shape index, information entropy, contrast and NDVI of the test area image 6 remote sensing feature parameters to obtain the remote sensing feature vector.

[0107] The standard deviation σ reflects the dispersion degree of the spectral reflectance of the building area, and the greater the value, the greater the difference in internal materials. The formula is:

[0108]

[0109] where x i represents the reflectance value of pixel i in a certain waveband, and σ Red , σ Green and σ Blue are respectively calculated for the red waveband, the green waveband and the blue waveband; μ represents the mean value of all pixels in the waveband; and N represents the total number of pixels in the building area.

[0110] The brightness B represents the overall light and dark degree of the building surface, and the formula is:

[0111]

[0112] where Red i represents the reflectance value of pixel i in the red waveband, Green i represents the reflectance value of pixel i in the green waveband, Blue i represents the reflectance value of pixel i in the blue waveband, and N represents the total number of pixels in the building area.

[0113] The shape index SI is used to measure the complexity of the shape of the building, and SI≥1. The closer the value is to 1, the closer the shape is to a circle, such as a dome building. The greater the value, the more irregular the shape of the building. The formula is:

[0114]

[0115] where Perimeter represents the perimeter of the building polygon, and Area represents the area of the building polygon.

[0116] The information entropy IE represents the randomness of the description of the texture of the building surface. The higher the entropy value, the more complex the texture. The formula is:

[0117]

[0118] where p i is the probability of the occurrence of gray level i in the image.

[0119] The contrast C is calculated by the formula:

[0120] Contrast =∑ i＝1 ∑ j＝1 (i-j) 2 P(i-j)

[0121] Wherein, P(i-j) is the co-occurrence probability at position (i-j) in the gray level co-occurrence matrix. In this embodiment, the contrast of two directions (0°, 90°) is calculated, and then the average value is calculated.

[0122] The vegetation normalized index represents NDVI, and the formula is:

[0123]

[0124] Wherein, RED is the red band reflectivity, and NIR is the near-infrared band reflectivity.

[0125] The characteristic remote sensing vector d1 is obtained as follows: d1 = [σ Red ,σ Green ,σ Blue , B, SI, IE, C, NDBI].

[0126] Specifically, the street view image classification instance segmentation and feature extraction mainly include:

[0127] 1) Screening the downloaded street view images, screening the street view images, eliminating invalid images, unifying the size and format, and converting to JPG picture format;

[0128] 2) In the special software, the image is classified and labeled for public buildings, the YOLOv8 instance segmentation model is adopted, the dynamic label allocation mechanism and the prototype mask optimization are adopted, the real-time high-precision segmentation of the street view buildings is realized, and the segmentation result is as follows: Figure 3 :

[0129] In the special software, the image is classified and labeled for public buildings, each polygon is given a class label, the text area is labeled, and the text area (such as building name) in the image is rectangularly labeled, and the label text is set;

[0130] Output format: each image generates a corresponding JSON file, including polygon vertex coordinates and class information;

[0131] Constructing the C2f_DCNv2 module: increasing the number of deformable convolution layers, increasing the modulation mechanism in the deformable convolution module, and improving the feature matrix formula of the post-convolution as follows:

[0132]

[0133] Wherein, W is the convolution kernel, p k is the position of each convolution kernel, K is the sampling position number of the convolution kernel, Δp k is a newly added two-dimensional offset matrix, Δm k is the normalized adjustment term.

[0134] The total loss function L of the model is established by the bounding box loss and the mask loss, the bounding box loss is calculated by using the WIoU loss function, and the formula is

[0135] L = L WIoUv3 + L mask

[0136] Wherein, L WIoUv3 is the bounding box loss; L mask is the mask loss.

[0137] The bounding box loss is calculated by using the WIoU loss function, and the formula of the WIoUv3 loss function is:

[0138]

[0139] Wherein, x, y are the center coordinates of the predicted box; x gt , y gt are the center coordinates of the real box; W g and H g are the minimum size of the closed anchor box; r is the gradient gain, and β is the outlying degree, which is controlled by the hyperparameters α and δ, when β = δ, δ makes r = 1.

[0140] 3) Perform SURF feature and HOG feature extraction:

[0141] Cut the labeled public building area into an independent image block, detect the SURF key point and calculate the descriptor, and detect the key point H(p, q, σ) by using the Hessian matrix:

[0142]

[0143] Wherein, L pp is the convolution result of the second derivative of Gauss, when det(H) > preset threshold, it is determined as a key point, p, q are pixel positions in the image, and σ is a kernel size parameter, which controls the sensitivity of feature point detection.

[0144] Descriptor generation: calculate the Haar wavelet response of the 20x20 area around the key point to form a 64-dimensional vector;

[0145] Perform K-means clustering on the SURF descriptors of all images to generate a visual dictionary (i.e. SURF feature), and count the word frequency histogram of each image;

[0146] HOG gradient calculation: scale the image block to 64x128 pixels, calculate the gradient direction histogram of each 8x8 pixel unit, and the formula is:

[0147] G x = I(x+1, y)-I(x-1, y), Gy = I(x, y+1) - (x, y-1)

[0148]

[0149] where I(x, y) denotes the pixel intensity value (grayscale or single channel) of the image at (x, y), G x denotes the gradient approximation in x direction, G y denotes the gradient approximation in y direction, Magnitude denotes the gradient magnitude, reflecting the edge intensity, and Orientation denotes the gradient direction, ranging from 0° to 180°, used for histogram statistics.

[0150] Cell unit and block normalization: divide the image into 8x8 pixel cell units, calculate the 9-bin gradient direction histogram for each unit; 2x2 cell units form a block (16x16 pixels), L2 normalize the block histogram to obtain the HOG feature v norm :

[0151]

[0152] where v denotes the unnormalized histogram vector, usually 36-dimensional, composed of 4 9-dimensional histograms; ε is a small value, for example 1e-5, to prevent division by zero; after normalization, the values are clipped and renormalized.

[0153] The final HOG feature dimension is 1764 (64x128 image).

[0154] 4) Use OCR method to detect the labeled text region, use CRNN model for text recognition:

[0155] CNN feature extraction: use ResNet to extract sequence features, output feature map F, T is the time step, and D is the feature dimension;

[0156] RNN sequence modeling: the hidden state h t of each time step t of RNN is updated according to the following formula:

[0157] h t = LSTM(x t , h t -1)

[0158] x t = F t

[0159] where x t is the input vector of time step t, h t is the hidden state of time step t; F is the time sequence feature matrix extracted by CNN, with shape (T, D), xt is the input feature vector at the t-th time step (i.e., the t-th row of F).

[0160] CTC decoding: solve the alignment problem of variable-length sequence, loss function L CTC is:

[0161]

[0162] where π represents the path, and B represents the mapping of removing repeated characters and spaces.

[0163] Keyword frequency: set the TF-IDF (t, d) value of the statistical category keyword:

[0164]

[0165] where t is a specific keyword, such as "government", d is the current document, i.e., the text content of a single street view image, N represents the total number of documents in the corpus, i.e., all street view images, DocsContaining(t): the number of documents containing the keyword t.

[0166] Spatial distribution: calculate the text coverage TextCoverage, the formula is:

[0167]

[0168] TextArea represents the total number of text area pixels (calculated by the OCR detection box); ImageArea represents the total number of pixels of the street view image (such as 256x256 = 65536).

[0169] Text direction: calculate the text inclination θ (0° to 90°) based on the angle of the minimum bounding rectangle.

[0170] θ = angle(minAreaRect(B))

[0171] where B represents the vertex coordinate set of the text detection box, minAreaRect is the minimum bounding rectangle calculation function, and θ is the rectangle inclination angle, ranging from -90° to 90°.

[0172] Further, the training data set is input into the model for optimization training, mainly including:

[0173] 1) Feature level fusion: concatenate the remote sensing feature vector, street view visual element feature vector, and text feature vector, and fuse the standardized remote sensing, street view visual, and OCR features:

[0174]

[0175] Represents 8-dimensional remote sensing features (standard deviation, brightness, shape index, etc.), Represents 1864-dimensional street scene features (SURF+HOG); Represents 52-dimensional OCR features (TF-IDF, text coverage, etc.).

[0176] 2) Model classification training: A random forest + SVM combination model is used for training and classification, which mainly includes the following steps:

[0177] Random Forest (RF) feature screening: input the concatenated feature Finput, output the feature importance ranking, retain the important Top_K features, and obtain the reduced dimension feature Fselected.

[0178] The SVM kernel function selects the RBF kernel. The kernel function is used to map the original low-dimensional feature space to a high-dimensional space. The RBF kernel function formula is:

[0179] K(F i , F j )=exp(-γ||F i -F j || 2 )

[0180]

[0181] Among them, F i , F j represents two eigenvectors, K(F i , F j ) is the output of the RBF kernel function, γ controls the kernel function width and determines the complexity of the decision boundary; ||F i -F j || represents the Euclidean distance between feature vectors.

[0182] Multi-classification strategy: Combine categories in pairs, train n binary SVM classifiers, and select the category with the most wins when making predictions;

[0183]

[0184] Where c represents the number of categories.

[0185] Furthermore, the sample data is input into the model to obtain the building classification prediction results. Figure 4 (a) to Figure 4 (c)

[0186] 1) Input the preprocessed data set into the model for recognition.

[0187] 2) Obtain the model prediction results.

[0188] 3) Evaluate the model accuracy using precision P, recall R and average F1-Score.

[0189]

[0190]

[0191] Wherein, TP: positive samples predicted as positive class by the model; TN: negative samples predicted as negative class by the model; FP: negative samples predicted as positive class by the model; FN: positive samples predicted as negative class by the model.

[0192] Optionally, high-resolution image acquisition, street view image download, selected test area, including the following steps:

[0193] 1) Obtain high Jing 3 remote sensing image, image spatial resolution is 0.5 meters;

[0194] 2) Write network crawler in Python language to download street view data in the corresponding area in batches, sample points are selected in the center of the road around the building, collect 2 images in the vertical direction of each sample point through Baidu map API interface, get not less than 20,000 images, name the image according to the coordinate information of the sample point, for example, 120.83826_24.67291_1, 120.83826_24.67291_2. Hang up the remote sensing data and street view data.

[0195] 3) Determine the test area range, select the area with more street view images in the central city.

[0196] Preferably, feature extraction: calculate the standard deviation, brightness, shape index, information entropy, contrast and NDVI of the test area image 6 remote sensing feature parameters, get the remote sensing feature vector d1:

[0197] 1) In ENVI software, load image data, select BasicTools> Statistics> ComputeStatistics in the main menu, select StandardDeviation (StandardDeviation), get the standard deviation image.

[0198] 2) In ENVI software, select BasicTools> Statistics> ComputeStatistics, then select Histogram (Histogram) to calculate the brightness image.

[0199] 3) In ENVI software, calculate the NDVI index of the test area, through the tool Toolbox > BandAlgebra > BandMath, enter the expression in the Enter an expression box: (float(b4)-b3) / (b4+b3); Wherein, b4 is the near-infrared band, b3 is the red band, and float(b4) means converting the b4 near-infrared band to floating point type. Note that the data type here, the data type used this time is integer, so data type conversion is required, convert one of the variables to floating point type.

[0200] Specifically, in Arcgis software, input three single-band images of the test area, select ArcToolbox > SpatialAnalyst tool > map algebra > raster calculator, and input the formula: "R" * 0.299 + "G" * 0.587 + "R" * 0.114; Wherein, R represents the red single-band image, 0.299 is the red band coefficient; G represents the green single-band image, 0.587 is the green band coefficient; B represents the blue single-band image, and 0.114 is the blue band coefficient. The normalized gray image of the test area is calculated, and the information entropy and contrast of the sample are calculated.

[0201] Preferably, the street view image is classified instance segmentation and feature extraction, mainly including:

[0202] Screening the downloaded street view image, screening the street view image, eliminating invalid images, unifying size and format, and converting to JPG picture format;

[0203] In LabelMe software, the image is classified and labeled with public building, and in LabelMe software, the selected street view image training data is selected, the public building is outlined, the polygon vertex is clicked to create a closed area;

[0204] Each polygon is assigned a class label, and the public building is divided into three categories, namely office building, science and education, and commercial building;

[0205] The street view image is classified and segmented.

[0206] Further, SURF feature, HOG feature and OCR text feature extraction; OCR text information feature extraction mainly includes: labeling text area, rectangular labeling of text area (such as building name) in image, and setting label text; The labeled image is subjected to text information detection. The training data set is input into the model for optimization training, mainly including:

[0207] Key parameter values, Table 1 is the key parameter value range, which can be adjusted according to actual conditions:

[0208] Table 1

[0209]

[0210]

[0211] Set the OCR keywords, and Table 2 is part of the different categories of keywords:

[0212] Table 2

[0213]

[0214]

[0215] Optimize the feature importance classification rules, and Table 3 is part of the rule examples:

[0216] Table 3

[0217]

[0218]

[0219] Further, data input: input the preprocessed data set into the model for recognition; result output: get the model prediction result; prediction result accuracy statistics: Table 4 is the statistical table of building recognition accuracy in this embodiment.

[0220] Table 4

[0221]

[0222] Therefore, it is known that the comprehensive prediction accuracy of building categories in this embodiment is 86.92%.

[0223] As an optional implementation, the embodiment also provides a multi-modal feature public building recognition system based on remote sensing images and street view images, comprising:

[0224] An image collection module is configured to collect remote sensing images and street view images of a to-be-recognized area to obtain to-be-recognized data.

[0225] A public building recognition module is configured to input the to-be-recognized data into a pre-trained public building recognition model for recognition to obtain a public building recognition result.

[0226] As an optional implementation, the embodiment further provides an electronic device, comprising at least one processor and a memory connected with the processor in communication; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor to enable the processor to implement the foregoing method for identifying public buildings based on multi-modal features of remote sensing images and street view images.

[0227] As an optional implementation, the embodiment further provides a non-transient computer readable storage medium storing computer instructions, and the computer instructions are used to enable a computer to implement the foregoing method for identifying public buildings based on multi-modal features of remote sensing images and street view images.

[0228] The beneficial effects of the present application are as follows:

[0229] The present application extracts remote sensing features, HOG features, SURF features and OCR features, deeply fuses macro-structure information of remote sensing images and micro-semantic details and text recognition information features of street view images, supplements facade details and functional identification, adopts geographic coordinate matching to ensure that remote sensing and street view features of the same building are consistent in space, and reduces the probability of misjudgment in classification.

[0230] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other.

[0231] The principles and implementation manners of the present application are described by using specific examples in the present application, and the above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, the specific implementation manners and application ranges can be changed according to the idea of the present application. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A multimodal feature public building recognition method based on remote sensing images and street view images, characterized in that: include: Collect remote sensing images and street view images of the area to be identified to obtain the data to be identified; Inputting the data to be identified into a pre-trained public building identification model for identification to obtain a public building identification result; The training process of the public building recognition model includes: Collecting the remote sensing images and the street view images of the study area; Preprocessing and feature extraction are performed on the remote sensing image to obtain remote sensing features; the remote sensing features include: red band standard deviation, green band standard deviation, blue band standard deviation, brightness index, shape index, information entropy, contrast, and vegetation normalization index; Perform instance segmentation on the street view image using a YOLOv8 network integrated with the C2f_DCNv2 improved module to obtain a number of image blocks; Performing descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing on the image block to obtain HOG features and SURF features; According to the OCR method, the CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image block to obtain OCR features; Performing splicing processing on the remote sensing feature, the HOG feature, the SURF feature, and the OCR feature to obtain a spliced ​​feature; The combined model of random forest and SVM is trained for classification using the splicing features according to a multi-classification strategy to obtain the public building recognition model.

2. The multimodal feature public building recognition method based on remote sensing images and street view images according to claim 1 is characterized in that: Preprocessing and feature extraction are performed on the remote sensing image to obtain remote sensing features, including: Performing radiometric calibration and atmospheric correction on the remote sensing image to obtain a pre-processed image; orthorectifying the pre-processed image according to pre-collected reference images and DEM data to obtain a planar orthorectified image; fusing the planar orthophoto and the pre-collected panchromatic and multispectral images to obtain a fused image; The red band standard deviation, the green band standard deviation, the blue band standard deviation, the brightness index, the shape index, the information entropy, the contrast, and the vegetation normalization index of the fused image are calculated and integrated to obtain the remote sensing feature.

3. The multimodal feature public building recognition method based on remote sensing images and street view images according to claim 1 is characterized in that: The training process of the YOLOv8 network includes: Classifying and labeling public buildings on the pre-collected street view images to obtain labeled images; Convert the label image into JSON format to obtain JSON data; the JSON data includes: polygon vertex coordinates and category information; Construct the C2f_DCNv2 improved module; the expression of the C2f_DCNv2 improved module is: Where y(p) is the output of the C2f_DCNv2 improvement module; w k is the convolution kernel; p is the offset; p k is the position of the convolution kernel; Δp k is a two-dimensional offset matrix; Δm k is the normalized adjustment term; K is the number of sampling positions; x(·) is the input feature of the C2f_DCNv2 improvement module; Construct and integrate the detection box loss calculation formula and the mask loss calculation formula to obtain the head loss function; The original C2f_DCNv2 module of the YOLOv8 network is replaced by the C2f_DCNv2 improved module, and the YOLOv8 network is trained using the head loss function according to the JSON data to obtain the trained YOLOv8 network.

4. The multimodal feature public building recognition method based on remote sensing images and street view images according to claim 1 is characterized in that: Perform descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing on the image block to obtain HOG features and SURF features, including: Calculating the Hessian matrix according to the image block to obtain key points; Perform Haar wavelet response calculation on the 20×20 area around the key point to obtain the SURF descriptor; K-means clustering is performed on the SURF descriptor to obtain SURF features.

5. The multimodal feature public building recognition method based on remote sensing images and street view images according to claim 1 is characterized in that: Perform descriptor calculation, clustering processing, HOG gradient calculation, and normalization processing on the image block to obtain HOG features and SURF features, including: Performing HOG gradient calculation on the image block to obtain a gradient direction histogram; The image block is divided into 8×8 pixel cell units, the cell units in a 2×2 range are determined as a block, and the gradient direction histogram in the block is L2 normalized to obtain the HOG feature.

6. The multimodal feature public building recognition method based on remote sensing images and street view images according to claim 1, characterized in that: According to the OCR method, the CRNN network is used to perform text recognition, keyword frequency statistics, spatial distribution calculation, and text direction calculation on the image block to obtain OCR features, including: Using a ResNet network to extract sequence features from the image blocks to obtain a feature map; Performing RNN sequence modeling on the feature graph using a feature graph network to obtain encoded data; Performing CTC decoding on the encoded data to obtain decoded data; Counting the TF-IDF values ​​of the category keywords according to the decoded data to obtain keyword frequency data; Calculate the area ratio of the text area according to the decoded data to obtain spatial distribution data; Calculate the minimum circumscribed rectangle angle based on the decoded data to obtain text direction data; The keyword frequency data, the spatial distribution data, and the text direction data are integrated to obtain the OCR feature.

7. The multimodal feature public building recognition method based on remote sensing images and street view images according to claim 1 is characterized in that: The combined model of random forest and SVM is trained for classification using the splicing features according to a multi-classification strategy to obtain the public building recognition model, including: Using random forest to perform feature screening on the splicing features to obtain dimensionality reduction features; The dimensionality reduction features are classified using an SVM kernel function according to a multi-classification strategy to obtain the public building recognition model.

8. A multimodal feature public building recognition method based on remote sensing images and street view images, characterized in that: The multimodal feature public building recognition method based on remote sensing images and street view images as described in claim 1, wherein the system comprises: The image collection module is used to collect remote sensing images and street view images of the area to be identified to obtain the data to be identified; The public building recognition module is used to input the data to be recognized into the pre-trained public building recognition model for recognition, and obtain a public building recognition result.

9. An electronic device, characterized in that: include: At least one processor and a memory communicatively connected to the processor; wherein the memory stores instructions executable by the processor, and the instructions are executed by the processor so that the processor can execute a multimodal feature public building recognition method based on remote sensing images and street view images as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute a multimodal feature public building recognition method based on remote sensing images and street view images according to any one of claims 1 to 7.

Images