Methods, devices, equipment, and media for building function identification based on visual language knowledge reasoning

Abstract

This application discloses a method, apparatus, system, and medium for building function recognition based on visual language knowledge reasoning. The method includes: extracting features from remote sensing images of buildings to obtain a global context feature map; inputting the global context feature map into a multilayer perceptron to obtain an initial category probability map; identifying classification uncertainty regions in the initial category probability map to obtain the topological features of each node in the building's edge contour; correcting the global context feature map based on the topological features to obtain a global visual feature vector; encoding textual description data corresponding to the building's remote sensing image to obtain a global semantic feature vector; aligning the global visual feature vector and the global semantic feature vector to obtain a global feature vector; and inputting the global feature vector into a building function classification model to obtain the building function recognition result. The method provided in this application improves the accuracy of building function recognition from remote sensing images.

Description

Technical Field

[0001] This application relates to the field of remote sensing image processing technology, and in particular to a method, apparatus, equipment and medium for building function recognition based on visual language knowledge reasoning. Background Technology

[0002] Buildings are crucial carriers of human functions, directly revealing socioeconomic patterns of human activity and playing a vital role in understanding urban socioeconomic dynamics and promoting sustainable urban development. Simultaneously, rapid urbanization has brought about dramatic changes in the number and function of buildings globally. Therefore, timely and accurate surveys and statistics on building numbers, and the subsequent differentiation of building land use attributes and functions, can provide fundamental geographic data for land use optimization policies such as housing supply regulation, commercial development planning, and the allocation of educational and medical resources. This is essential for improving the rationality of spatial layout, human livability, reducing greenhouse gas emissions, and promoting sustainable urban development.

[0003] Satellite remote sensing imagery offers advantages such as a large observation range and high spatial resolution. Combined with artificial intelligence semantic segmentation algorithms, it can achieve automated identification of buildings at the footprint scale. However, semantic segmentation algorithms based on image visual information perception lack brain-like reasoning capabilities and cannot determine the attributes and functions of buildings in remote sensing images by considering the surrounding environment and common sense. Summary of the Invention

[0004] This application aims to provide a method, apparatus, equipment, and medium for building function recognition based on visual language knowledge reasoning, in order to solve the technical problem of insufficient building remote sensing image function recognition capability.

[0005] To address the aforementioned technical problems, embodiments of this application provide a building function recognition method based on visual language knowledge reasoning, including:

[0006] Feature extraction is performed on the acquired remote sensing images of buildings to obtain a global context feature map of the building remote sensing images. The global context feature map is then input into a pre-constructed multilayer perceptron for classification to obtain an initial category probability map of the building remote sensing images.

[0007] Identify the classification uncertainty regions in the initial category probability map, and filter the classification uncertainty regions based on machine learning algorithms to obtain the classification map of the building remote sensing image;

[0008] The classification map is processed using the circular convolution operation method to obtain the topological features of each node in the building edge contour in the building remote sensing image. The global context feature map is then corrected based on the topological features to obtain the global visual feature vector.

[0009] The obtained text description data corresponding to the remote sensing image of the building is split into word vectors, and each word vector is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0010] The global visual feature vector and the global semantic feature vector are mapped to a multimodal space for alignment processing to obtain a multimodal global feature vector. The multimodal global feature vector is then input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

[0011] As one preferred embodiment, the step of extracting features from the acquired remote sensing image of the building to obtain a global contextual feature map of the building remote sensing image includes:

[0012] The remote sensing images of the building are subjected to denoising, contrast enhancement, and correction processes to obtain preprocessed remote sensing images.

[0013] The preprocessed remote sensing image is segmented to obtain multiple feature blocks. Nested multi-level key-value query operations are performed between each feature block to obtain the operation results, which reflect the spatial relationship between different feature blocks.

[0014] A neural network is constructed based on the calculation results. The preprocessed remote sensing image is then input into the neural network for feature extraction to obtain a global context feature map of the building remote sensing image.

[0015] As one preferred embodiment, identifying the classification uncertainty region in the initial category probability map and obtaining the classification map of the building remote sensing image by relearning the classification uncertainty region includes:

[0016] Calculate the error statistical distribution of the initial category probability map, and identify the classification uncertainty region in the initial category probability map based on the error statistical distribution;

[0017] Feature extraction is performed on the region of classification uncertainty to obtain edge feature vectors;

[0018] The edge feature vector is input into a pre-constructed multilayer perceptron for reclassification. The initial category probability map is updated based on the reclassification result to obtain a classification map of the building remote sensing image.

[0019] As one preferred embodiment, the processing of the classification map based on the circular convolution operation method to obtain the topological features of each node in the building edge contour in the building remote sensing image includes:

[0020] The initial outline map of the building in the remote sensing image is extracted based on the classification map, and multiple outline nodes are extracted based on the initial outline map;

[0021] Based on circular convolution operation, feature extraction is performed on each of the contour nodes to obtain the first topological structure feature;

[0022] The first topological feature is input into a pre-built coordinate decoder for optimization to obtain the topological features of each node in the building edge contour in the building remote sensing image. The optimization is designed to make the coordinates of the contour nodes close to the true coordinates of the building in the building remote sensing image through gradient backpropagation.

[0023] As one preferred embodiment, the acquired text description data corresponding to the remote sensing image of the building is split into word vectors, and each word vector representation is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector, including:

[0024] The text description data is split into multiple word units using a word segmenter.

[0025] Each of the aforementioned words is input into a pre-built BERT model to obtain word vectors;

[0026] The word vectors are encoded using a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0027] As a preferred embodiment, the step of mapping the global visual feature vector and the global semantic feature vector to a multimodal space for alignment processing to obtain a multimodal global feature vector includes:

[0028] A linear transformation is used to map the global visual feature vector and the global semantic feature vector to the same multimodal space;

[0029] Calculate the cosine similarity between the global visual feature vector and the global semantic feature vector after linear transformation, and align the global visual feature vector and the global semantic feature vector according to the cosine similarity;

[0030] The Transformer structure based on attention mechanism fusion fuses the aligned global visual feature vector and the global semantic feature vector to obtain the multimodal global feature vector.

[0031] As one preferred embodiment, the step of inputting the multimodal global feature vector into a pre-constructed building function classification model, and determining the result output by the building function classification model as the function recognition result of the building remote sensing image, includes:

[0032] The multimodal global feature vector is input into a pre-constructed building function classification model, and the output reflects the probability distribution of building types in the remote sensing image of the building.

[0033] Assign a corresponding functional category label to each pixel of the building remote sensing image according to the probability distribution;

[0034] The segmentation result of the remote sensing image of the building is obtained based on the functional category label.

[0035] Another embodiment of this application provides a building function recognition device based on visual language knowledge reasoning, including:

[0036] The visual feature extraction module is used to extract features from the acquired building remote sensing images to obtain a global context feature map of the building remote sensing images. The global context feature map is then input into a pre-constructed multilayer perceptron for classification to obtain an initial category probability map of the building remote sensing images.

[0037] The filtering module is used to identify the classification uncertainty region in the initial category probability map, and filter the classification uncertainty region based on the machine learning algorithm to obtain the classification map of the building remote sensing image;

[0038] The correction module is used to process the classification map based on the circular convolution operation method to obtain the topological structure features of each node in the building edge contour in the building remote sensing image, and to correct the global context feature map according to the topological structure features to obtain a global visual feature vector.

[0039] The semantic feature extraction module is used to split the acquired text description data corresponding to the remote sensing image of the building into word vectors, and encode each word vector representation through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0040] The recognition module is used to map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector. The multimodal global feature vector is then input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

[0041] Another embodiment of this application provides a building function recognition device based on visual language knowledge reasoning, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the building function recognition method based on visual language knowledge reasoning as described above.

[0042] In another embodiment of this application, a computer-readable storage medium is provided, which stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the building function recognition method based on visual language knowledge reasoning as described above.

[0043] Compared to the prior art, the beneficial effects of the embodiments of this application are at least one of the following:

[0044] (1) By extracting global contextual feature maps, this application can capture the overall information of the building and its surrounding environment, thereby more accurately identifying different functional areas of the building. In addition, by relearning the classification uncertainty areas, this application can further optimize the segmentation results and improve the segmentation accuracy.

[0045] (2) This application uses circular convolution operation to process the classification map, which can extract the topological features of each node in the building edge contour. These features are of great significance for handling fuzzy boundaries and complex scenes. At the same time, the multimodal feature fusion technology combines global visual feature vectors and global semantic feature vectors, further enhancing the robustness of the model.

[0046] (3) This application combines deep learning technologies, such as multilayer perceptrons and multilayer bidirectional self-attention mechanisms, which can efficiently process large amounts of remote sensing image data and improve the efficiency of building function segmentation. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a building function recognition method based on visual language knowledge reasoning in one embodiment of this application.

[0048] Figure 2 This is a schematic diagram of the structure of a building function recognition device based on visual language knowledge reasoning in one embodiment of this application;

[0049] Figure 3 This is a diagram of a building function recognition device based on visual language knowledge reasoning in one embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0051] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0052] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0053] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in the specification of this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0054] It is important to note beforehand that traditional building semantic segmentation techniques typically employ a bottom-up, layer-by-layer abstraction approach for high-level semantic recognition. This method struggles to intuitively represent the architectural geometry or geospatial context features upon which the interpretation of building attributes and functions relies. For example, office buildings are often taller than school buildings, and playgrounds and school buildings, as well as green belts and residential buildings, frequently coexist. This results in poor interpretability and insufficient self-cognitive reasoning ability in traditional models during semantic segmentation, contradicting the top-down cognitive approach of the human brain. On the other hand, while traditional land use classification methods can utilize spectral information, spatial texture, or social media information for functional attribute identification, they often discretize and abstract geographical objects, making it difficult to effectively express the spatial and temporal relationships between buildings. Furthermore, the fine-grained nature of land use functions at the plot scale is insufficient to meet the needs of dividing human activity spaces based on building units.

[0055] How to overcome the multimodal heterogeneity between the structured representation of the functional spatial association knowledge of buildings and the Euclidean space of visual images, and realize the footprint-scale inference of building functional attributes guided by spatial association knowledge, is a key problem that urgently needs to be solved.

[0056] In summary, one embodiment of this application provides a method for building function recognition based on visual language knowledge reasoning. For details, please refer to [link to relevant documentation]. Figure 1 , Figure 1 The diagram shown is a flowchart of a building function recognition method based on visual language knowledge reasoning in one embodiment of this application, including steps S1-S5:

[0057] S1: Extract features from the acquired building remote sensing image to obtain the global context feature map of the building remote sensing image. Input the global context feature map into a pre-constructed multilayer perceptron for classification to obtain the initial category probability map of the building remote sensing image.

[0058] Understandably, in the task of classifying remote sensing images of buildings, the purpose of feature extraction is to extract useful information from the original image, which can help the subsequent classifier better understand and distinguish different image contents.

[0059] Preferably, in one embodiment of this application, the step of extracting features from the acquired remote sensing image of the building to obtain a global contextual feature map of the remote sensing image of the building includes:

[0060] The remote sensing images of the building are subjected to denoising, contrast enhancement, and correction processes to obtain preprocessed remote sensing images.

[0061] The preprocessed remote sensing image is segmented to obtain multiple feature blocks. Nested multi-level key-value query operations are performed between each feature block to obtain the operation results, which reflect the spatial relationship between different feature blocks.

[0062] A neural network is constructed based on the calculation results. The preprocessed remote sensing image is then input into the neural network for feature extraction to obtain a global context feature map of the building remote sensing image.

[0063] Specifically, remote sensing images may be affected by various noises during acquisition and transmission, such as Gaussian noise and salt-and-pepper noise. These noises severely impact image quality and subsequent feature extraction. Therefore, image denoising is necessary before feature extraction. Common denoising methods include mean filtering, median filtering, and Gaussian filtering. Contrast is the difference in brightness between different regions of an image. Enhancing contrast can improve the visibility of important features in an image, making targets more prominent. Common contrast enhancement methods include histogram equalization and adaptive contrast enhancement. Correction processing mainly includes geometric correction and radiometric correction. Geometric correction is used to correct geometric distortions in the image, such as tilt and warping. Radiometric correction is used to adjust the radiometric characteristics of the image to better reflect reality.

[0064] In this step, the preprocessed remote sensing image is segmented to obtain multiple feature blocks. These feature blocks are local regions in the image, containing different features. There are many segmentation methods, such as threshold-based segmentation, edge-based segmentation, and region-based segmentation. Nested multi-level key-value lookup operations are performed between each feature block to obtain results reflecting the spatial relationships between different feature blocks. This method utilizes the attention mechanism in deep learning, calculating the similarity and weights between different feature blocks to capture their spatial relationships.

[0065] Based on the results of nested multi-layer key-value query operations, a neural network is constructed for feature extraction. This neural network can be a convolutional neural network (CNN) or other types of deep learning models. The preprocessed remote sensing image is input into the neural network, and feature extraction and dimensionality reduction are performed through convolutional layers, pooling layers, and other structures to finally obtain the global context feature map of the building remote sensing image.

[0066] S2: Identify the classification uncertainty region in the initial category probability map, and filter the classification uncertainty region based on the machine learning algorithm to obtain the classification map of the building remote sensing image;

[0067] Understandably, the global context feature map contains global features and contextual information from the image, which is crucial for subsequent classification tasks. By inputting the global context feature map into a pre-built multilayer perceptron for classification, an initial class probability map of the building remote sensing image can be obtained.

[0068] Preferably, in one embodiment of this application, identifying the classification uncertainty region in the initial category probability map and obtaining the classification map of the building remote sensing image by relearning the classification uncertainty region includes:

[0069] Calculate the error statistical distribution of the initial category probability map, and identify the classification uncertainty region in the initial category probability map based on the error statistical distribution;

[0070] Feature extraction is performed on the region of classification uncertainty to obtain edge feature vectors;

[0071] The edge feature vector is input into a pre-constructed multilayer perceptron for reclassification. The initial category probability map is updated based on the reclassification result to obtain a classification map of the building remote sensing image.

[0072] In this step, to identify regions of classification uncertainty, it is necessary to calculate the error statistical distribution of the initial class probability map. This application achieves this by comparing the initial class probability map with the true labels (if any), or by evaluating the dispersion between different class probabilities.

[0073] After calculating the error statistical distribution, a threshold is set to identify regions of classification uncertainty. These regions are typically those with larger errors and more dispersed class probability distributions. By setting a reasonable threshold, these uncertain regions can be extracted from the initial class probability map.

[0074] For identified regions of classification uncertainty, more detailed feature extraction is performed. This can be achieved using deep convolutional neural networks (CNNs) or other feature extraction algorithms. The extracted features yield edge feature vectors, which contain edge information, texture information, and other data about the pixels in the region of classification uncertainty. This information is used to improve classification accuracy.

[0075] These edge feature vectors are then input into a pre-built multilayer perceptron (or other classifier) ​​for reclassification. This multilayer perceptron can be specifically designed to handle these types of features or it can be transferred from other similar tasks. Through reclassification, new class probabilities for pixels in these uncertain regions can be obtained. The initial class probability map is updated based on the reclassification results to obtain a more accurate classification map of the building remote sensing image.

[0076] S3: The classification map is processed based on the circular convolution operation method to obtain the topological structure features of each node in the building edge contour in the building remote sensing image. The global context feature map is corrected according to the topological structure features to obtain the global visual feature vector.

[0077] Preferably, in one embodiment of this application, the processing of the classification map based on the circular convolution operation method to obtain the topological features of each node in the building edge contour in the building remote sensing image includes:

[0078] The initial outline map of the building in the remote sensing image is extracted based on the classification map, and multiple outline nodes are extracted based on the initial outline map;

[0079] Based on circular convolution operation, feature extraction is performed on each of the contour nodes to obtain the first topological structure feature;

[0080] The first topological feature is input into a pre-built coordinate decoder for optimization to obtain the topological features of each node in the building edge contour in the building remote sensing image. The optimization is designed to make the coordinates of the contour nodes close to the true coordinates of the building in the building remote sensing image through gradient backpropagation.

[0081] Specifically, the classification image is binarized to distinguish between building and non-building areas, thereby outlining the initial contours of the buildings. Image processing techniques (such as edge detection and contour tracking) are then used to extract multiple contour nodes from the initial contour image. These contour nodes represent key locations on the building edges.

[0082] For each extracted contour node, circular convolution is applied for feature extraction. Circular convolution is a special type of convolution operation that uses a circular filter to slide across the image, extracting features by calculating the dot product of the filter and a local region of the image. This method can capture local texture, shape, and other information around the contour node, forming the first topological structure feature. These features reflect the local geometric characteristics of the building's edge contour.

[0083] The first topological feature is input into a pre-built coordinate decoder for optimization. The coordinate decoder is a deep learning model that learns the mapping relationship between the topological features of the building's edge contour and the coordinates of its nodes, enabling it to predict and optimize the coordinates of the contour nodes. During optimization, the error between the predicted and true coordinates is calculated using a gradient backpropagation algorithm, and the parameters of the coordinate decoder are continuously adjusted so that the predicted coordinates gradually approach the true coordinates of the building in the remote sensing image.

[0084] After optimization by the coordinate decoder, more accurate topological features of each node in the building edge contour are obtained. These features not only contain local geometric information of the building edge but also reflect the spatial relationships between nodes. Next, these topological features are used to correct the global context feature map. The correction process may involve weighting, filtering, or replacing specific regions in the global context feature map to better reflect the topological information of the building edge. The corrected global context feature map is further processed to generate a global visual feature vector.

[0085] S4: The obtained text description data corresponding to the remote sensing image of the building is split into word vectors, and each word vector is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector;

[0086] By downloading publicly available "remote sensing image-text description" sample pairs as input (such as the RSICD dataset, RSITMD dataset, UCMerced dataset, etc., whose data organization is remote sensing image data such as airport scenes and their corresponding descriptive text such as "Some planes are parked in an airport near a piece of green trees"), the aim is to train a language model that is aligned with the visual features of the images and can understand visual cues to perform spatial reasoning through the building spatial association knowledge language instruction encoding module.

[0087] This language model learns the semantic associations of building functional attribute words from image-description text. It embeds spatially related object concepts into their vicinity in the latent space, thereby learning the topological relationships between objects and aiding in the reasoning of building functions. Simultaneously, it aligns the visual encodings of object entities with the corresponding textual semantic concepts obtained from the language model in the latent space, enabling the language model to understand visual cues in the images.

[0088] Preferably, in one embodiment of this application, the step of splitting the acquired text description data corresponding to the remote sensing image of the building into word vectors, and encoding each word vector representation through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector includes:

[0089] The text description data is split into multiple word units using a word segmenter.

[0090] Each of the aforementioned words is input into a pre-built BERT model to obtain word vectors;

[0091] The word vectors are encoded using a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0092] In this step, a tokenizer is used to process the text description data, splitting it into multiple independent tokens. The tokenizer identifies and segments words or phrases in the text based on language rules (such as Chinese word segmentation rules) or a pre-trained model. Each token is then fed into a pre-built BERT (Bidirectional Encoder Representations from Transformers) model. BERT is a pre-trained language representation model based on the Transformer architecture, capable of capturing the contextual relationships between tokens and generating token vectors containing rich semantic information.

[0093] Within the BERT model, word vectors undergo processing through multiple layers of bidirectional self-attention mechanisms. This mechanism allows each word to simultaneously consider the textual information before and after it during encoding, thereby more accurately capturing global semantic features. Through multi-layer stacking, the model can progressively abstract higher-level semantic information, ultimately generating a global semantic feature vector.

[0094] S5: Map the global visual feature vector and the global semantic feature vector to the multimodal space for alignment processing to obtain a multimodal global feature vector. Input the multimodal global feature vector into a pre-constructed building function classification model. The result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

[0095] Preferably, in one embodiment of this application, the step of mapping the global visual feature vector and the global semantic feature vector to a multimodal space for alignment processing to obtain a multimodal global feature vector includes:

[0096] A linear transformation is used to map the global visual feature vector and the global semantic feature vector to the same multimodal space;

[0097] Calculate the cosine similarity between the global visual feature vector and the global semantic feature vector after linear transformation, and align the global visual feature vector and the global semantic feature vector according to the cosine similarity;

[0098] The Transformer structure based on attention mechanism fusion fuses the aligned global visual feature vector and the global semantic feature vector to obtain the multimodal global feature vector.

[0099] Specifically, to align the global visual feature vector (the result of processing building remote sensing images) with the global semantic feature vector (the result of processing text description data), this application employs a linear transformation to map them into the same multimodal space. This process can be achieved by learning a linear mapping matrix that can convert the two feature vectors into new vectors with the same dimension and similar distribution.

[0100] In multimodal space, the cosine similarity between the global visual feature vector and the global semantic feature vector is calculated. Cosine similarity is an index that measures the similarity of the directions of two vectors; the closer the value is to 1, the more consistent the directions of the two vectors are. Based on the calculation results of cosine similarity, the global visual feature vector and the global semantic feature vector can be aligned, that is, their directions and / or lengths can be adjusted to make them closer in multimodal space.

[0101] This paper employs a Transformer architecture based on an attention mechanism to fuse aligned global visual feature vectors and global semantic feature vectors. The Transformer architecture is a powerful sequence modeling tool capable of capturing long-term dependencies between elements in a sequence. This application utilizes the Transformer's attention mechanism to dynamically adjust the weights between the two feature vectors, thereby achieving effective fusion. The fused result is a multimodal global feature vector that integrates global information from building remote sensing images and textual description data, providing strong support for subsequent tasks such as building recognition, classification, and retrieval.

[0102] Preferably, in one embodiment of this application, the step of inputting the multimodal global feature vector into a pre-constructed building function classification model, and determining the result output by the building function classification model as the function recognition result of the building remote sensing image, includes:

[0103] The multimodal global feature vector is input into a pre-constructed building function classification model, and the output reflects the probability distribution of building types in the remote sensing image of the building.

[0104] Assign a corresponding functional category label to each pixel of the building remote sensing image according to the probability distribution;

[0105] The segmentation result of the remote sensing image of the building is obtained based on the functional category label.

[0106] Specifically, this application inputs a multimodal global feature vector into a pre-constructed building function classification model. This model has been optimized using a large amount of training data and is capable of accurately understanding and identifying multiple functional features in remote sensing images of buildings. After receiving the multimodal global feature vector, the building function classification model uses its complex internal neural network structure to perform calculations, ultimately outputting a probability distribution reflecting the building types in the remote sensing image. Each element in this distribution represents a possible building type, and its value indicates the probability of that type appearing in the image.

[0107] Based on the probability distribution output by the building function classification model, each pixel in the remote sensing image of the building is assigned a corresponding function category label. This mapping process may involve some additional processing steps, such as threshold setting and maximum probability selection, to ensure that each pixel can be accurately classified into its most probable function category.

[0108] After the above steps, a segmentation result reflecting the functional category of each pixel in the remote sensing image of a building is obtained. This result is presented in the form of an image, where different colors or markers represent different building functional categories. The segmentation result can be used in various practical applications, such as urban planning, environmental monitoring, and disaster assessment.

[0109] Compared to the prior art, the beneficial effects of the embodiments of this application are at least one of the following:

[0110] (1) By extracting global contextual feature maps, this application can capture the overall information of the building and its surrounding environment, thereby more accurately identifying different functional areas of the building. In addition, by relearning the classification uncertainty areas, this application can further optimize the segmentation results and improve the segmentation accuracy.

[0111] (2) This application uses circular convolution operation to process the classification map, which can extract the topological features of each node in the building edge contour. These features are of great significance for handling fuzzy boundaries and complex scenes. At the same time, the multimodal feature fusion technology combines global visual feature vectors and global semantic feature vectors, further enhancing the robustness of the model.

[0112] (3) This application combines deep learning technologies, such as multilayer perceptrons and multilayer bidirectional self-attention mechanisms, which can efficiently process large amounts of remote sensing image data and improve the efficiency of building function segmentation.

[0113] Another embodiment of the present invention provides a building function recognition device based on visual language knowledge reasoning. For details, please refer to [link to relevant documentation]. Figure 2 , Figure 2The diagram shows a structural schematic of a building function recognition device based on visual language knowledge reasoning provided in one embodiment of the present invention, including: a visual feature extraction module 11, a filtering module 12, a correction module 13, a semantic feature extraction module 14, and a recognition module 15, wherein...

[0114] The visual feature extraction module 11 is used to extract features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, and input the global context feature map into a pre-constructed multilayer perceptron for classification to obtain an initial category probability map of the building remote sensing image.

[0115] The filtering module 12 is used to identify the classification uncertainty region in the initial category probability map, and filter the classification uncertainty region based on the machine learning algorithm to obtain the classification map of the building remote sensing image;

[0116] The correction module 13 is used to process the classification map based on the circular convolution operation method to obtain the topological structure features of each node in the building edge contour in the building remote sensing image, and to correct the global context feature map according to the topological structure features to obtain a global visual feature vector.

[0117] The semantic feature extraction module 14 is used to split the acquired text description data corresponding to the remote sensing image of the building into word vectors, and encode each word vector representation through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0118] The recognition module 15 is used to map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector. The multimodal global feature vector is then input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

[0119] Another embodiment of the present invention provides a building function recognition device based on visual language knowledge reasoning; for details, see [link to specific documentation]. Figure 3 This is a structural block diagram of a building function recognition device based on visual language knowledge reasoning provided in this application embodiment. The building function recognition device based on visual language knowledge reasoning provided in this application embodiment includes a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, it implements the steps as described in the above-described embodiment of the building function recognition method based on visual language knowledge reasoning, for example... Figure 1The steps S1 to S5 described above; or, when the processor 21 executes the computer program, it implements the functions of each module in the above-described device embodiments, such as the visual feature extraction module 11.

[0120] For example, the computer program can be divided into one or more modules, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the building function recognition device based on visual language knowledge reasoning. For example, the computer program can be divided into a visual feature extraction module 11, a filtering module 12, a correction module 13, a semantic feature extraction module 14, and a recognition module 15.

[0121] The building function recognition device based on visual language knowledge reasoning may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of a building function recognition device based on visual language knowledge reasoning and does not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components. For example, the building function recognition device based on visual language knowledge reasoning may also include input / output devices, network access devices, buses, etc.

[0122] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the building function recognition device based on visual language knowledge reasoning, connecting all parts of the device via various interfaces and lines.

[0123] The memory 22 can be used to store the computer program and / or modules. The processor 21 implements various functions of the building function recognition device based on visual language knowledge reasoning by running or executing the computer program and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0124] The integrated module of the building function recognition device based on visual language knowledge reasoning, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0125] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0126] Accordingly, embodiments of this application provide a computer-readable storage medium, which includes a stored computer program. When the computer program is executed, it controls the device containing the computer-readable storage medium to perform steps in the building function recognition method based on visual language knowledge reasoning as described in the above embodiments, for example... Figure 1 Steps S1 to S5 as described above.

[0127] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for recognizing building functions based on visual language knowledge reasoning, characterized in that, include: Feature extraction is performed on the acquired remote sensing images of buildings to obtain a global context feature map of the building remote sensing images. The global context feature map is then input into a pre-constructed multilayer perceptron for classification to obtain an initial category probability map of the building remote sensing images. Identify the classification uncertainty regions in the initial category probability map, and filter the classification uncertainty regions based on machine learning algorithms to obtain the classification map of the building remote sensing image; The classification map is processed using the circular convolution operation method to obtain the topological features of each node in the building edge contour in the building remote sensing image. The global context feature map is then corrected based on the topological features to obtain the global visual feature vector. The obtained text description data corresponding to the remote sensing image of the building is split into word vectors, and each word vector is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector. The global visual feature vector and the global semantic feature vector are mapped to a multimodal space for alignment processing to obtain a multimodal global feature vector. The multimodal global feature vector is then input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

2. The building function recognition method based on visual language knowledge reasoning as described in claim 1, characterized in that, The step of extracting features from the acquired remote sensing images of buildings to obtain a global contextual feature map of the building remote sensing images includes: The remote sensing images of the building are subjected to denoising, contrast enhancement, and correction processes to obtain preprocessed remote sensing images. The preprocessed remote sensing image is segmented to obtain multiple feature blocks. Nested multi-level key-value query operations are performed between each feature block to obtain the operation results, which reflect the spatial relationship between different feature blocks. A neural network is constructed based on the calculation results. The preprocessed remote sensing image is then input into the neural network for feature extraction to obtain a global context feature map of the building remote sensing image.

3. The building function recognition method based on visual language knowledge reasoning as described in claim 1, characterized in that, The process of identifying classification uncertainty regions in the initial category probability map and obtaining a classification map of the building remote sensing image by relearning these regions includes: Calculate the error statistical distribution of the initial category probability map, and identify the classification uncertainty region in the initial category probability map based on the error statistical distribution; Feature extraction is performed on the region of classification uncertainty to obtain edge feature vectors; The edge feature vector is input into a pre-constructed multilayer perceptron for reclassification. The initial category probability map is updated based on the reclassification result to obtain a classification map of the building remote sensing image.

4. The building function recognition method based on visual language knowledge reasoning as described in claim 1, characterized in that, The classification map is processed using the circular convolution operation method to obtain the topological features of each node in the building edge contour in the building remote sensing image, including: The initial outline map of the building in the remote sensing image is extracted based on the classification map, and multiple outline nodes are extracted based on the initial outline map; Based on circular convolution operation, feature extraction is performed on each of the contour nodes to obtain the first topological structure feature; The first topological feature is input into a pre-built coordinate decoder for optimization to obtain the topological features of each node in the building edge contour in the building remote sensing image. The optimization is designed to make the coordinates of the contour nodes close to the true coordinates of the building in the building remote sensing image through gradient backpropagation.

5. The building function recognition method based on visual language knowledge reasoning as described in claim 1, characterized in that, The process involves splitting the acquired text description data corresponding to the remote sensing image of the building into word vectors, encoding each word vector representation through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector, including: The text description data is split into multiple word units using a word segmenter. Each of the aforementioned words is input into a pre-built BERT model to obtain word vectors; The word vectors are encoded using a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

6. The building function recognition method based on visual language knowledge reasoning as described in claim 1, characterized in that, The step of mapping the global visual feature vector and the global semantic feature vector to a multimodal space for alignment processing to obtain a multimodal global feature vector includes: A linear transformation is used to map the global visual feature vector and the global semantic feature vector to the same multimodal space; Calculate the cosine similarity between the global visual feature vector and the global semantic feature vector after linear transformation, and align the global visual feature vector and the global semantic feature vector according to the cosine similarity; The Transformer structure based on attention mechanism fusion fuses the aligned global visual feature vector and the global semantic feature vector to obtain the multimodal global feature vector.

7. The building function recognition method based on visual language knowledge reasoning as described in claim 1, characterized in that, The step of inputting the multimodal global feature vector into a pre-constructed building function classification model, and determining the result output by the building function classification model as the function recognition result of the building remote sensing image, includes: The multimodal global feature vector is input into a pre-constructed building function classification model, and the output reflects the probability distribution of building types in the remote sensing image of the building. Assign a corresponding functional category label to each pixel of the building remote sensing image according to the probability distribution; The segmentation result of the remote sensing image of the building is obtained based on the functional category label.

8. A building function recognition device based on visual language knowledge reasoning, characterized in that, include: The visual feature extraction module is used to extract features from the acquired building remote sensing images to obtain a global context feature map of the building remote sensing images. The global context feature map is then input into a pre-constructed multilayer perceptron for classification to obtain an initial category probability map of the building remote sensing images. The filtering module is used to identify the classification uncertainty region in the initial category probability map, and filter the classification uncertainty region based on the machine learning algorithm to obtain the classification map of the building remote sensing image; The correction module is used to process the classification map based on the circular convolution operation method to obtain the topological structure features of each node in the building edge contour in the building remote sensing image, and to correct the global context feature map according to the topological structure features to obtain a global visual feature vector. The semantic feature extraction module is used to split the acquired text description data corresponding to the remote sensing image of the building into word vectors, and encode each word vector representation through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector. The recognition module is used to map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector. The multimodal global feature vector is then input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

9. A building function recognition device based on visual language knowledge reasoning, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the building function recognition method based on visual language knowledge reasoning as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the building function recognition method based on visual language knowledge reasoning as described in any one of claims 1 to 7.

Images