Building function identification method and device based on visual language knowledge reasoning, equipment and medium

Abstract

The invention discloses a building function recognition method, device and system based on visual language knowledge reasoning and a medium. The method comprises the steps that feature extraction is conducted on a building remote sensing image, a global context feature map is obtained, the global context feature map is input into a multi-layer perceptron, and an initial category probability map is obtained; identifying a classification uncertainty region in the initial category probability graph to obtain topological structure features of each node in the building edge contour, and correcting the global context feature graph according to the topological structure features to obtain a global visual feature vector; encoding the text description data corresponding to the building remote sensing image to obtain a global semantic feature vector; and aligning the global visual feature vector and the global semantic feature vector to obtain a global feature vector, and inputting the global feature vector into a building function classification model to obtain a building function recognition result. According to the method provided by the invention, the accuracy of remote sensing image building function identification is improved.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a method, device, equipment and medium for identifying building functions based on visual language knowledge reasoning. Background Art

[0002] Buildings are key carriers of human use functions and can directly reveal the socio-economic patterns of human activities. They are essential for understanding urban socio-economic dynamics and promoting sustainable urban development. At the same time, the rapid urbanization process has brought about dramatic changes in the number and functions of buildings around the world. Therefore, timely and accurate surveys and statistics on the number of buildings and the distinction of the land use attributes and functions of buildings on this basis can provide basic geographic data for land use optimization policies such as housing supply regulation, commercial development planning, and allocation of education and medical resources, which is essential for improving the rationality of spatial layout, human livability, reducing greenhouse gas emissions, and sustainable urban development.

[0003] Satellite remote sensing images have the advantages of large observation range and high spatial resolution. With the help of artificial intelligence semantic segmentation algorithms, they can realize the automatic identification of footprint-scale buildings. However, semantic segmentation algorithms based on image visual information perception lack brain-like reasoning capabilities and cannot determine the attribute functions of buildings in remote sensing images by considering the surrounding environment and common sense. Summary of the invention

[0004] The present application aims to provide a method, device, equipment and medium for building function recognition based on visual language knowledge reasoning to solve the technical problem of insufficient building function recognition capability in remote sensing images.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for identifying building functions based on visual language knowledge reasoning, including:

[0006] Performing feature extraction on the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, inputting the global context feature map into a pre-built multi-layer perceptron for classification, and obtaining an initial category probability map of the building remote sensing image;

[0007] Identifying a classification uncertainty region in the initial class probability map, screening the classification uncertainty region based on a machine learning algorithm, and obtaining a classification map of the building remote sensing image;

[0008] The classification map is processed based on a circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, and the global context feature map is corrected according to the topological structure features to obtain a global visual feature vector;

[0009] Splitting the acquired text description data corresponding to the remote sensing image of the building into word element vectors, encoding each word element vector representation through a multi-layer bidirectional self-attention mechanism, and obtaining a global semantic feature vector;

[0010] The global visual feature vector and the global semantic feature vector are mapped into a multimodal space for alignment processing to obtain a multimodal global feature vector, the multimodal global feature vector is input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

[0011] As one of the preferred solutions, the step of extracting features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image includes:

[0012] The remote sensing image of the building is subjected to denoising, contrast enhancement and correction processing respectively to obtain a preprocessed remote sensing image;

[0013] Segmenting the preprocessed remote sensing image to obtain a plurality of feature blocks, performing a nested multi-layer key-value query operation between each of the feature blocks to obtain an operation result, wherein the operation result reflects the spatial association between different feature blocks;

[0014] A neural network is constructed according to the calculation results, and the preprocessed remote sensing image is input into the neural network for feature extraction to obtain a global context feature map of the building remote sensing image.

[0015] As one of the preferred solutions, the step of identifying the classification uncertainty region in the initial category probability map and obtaining the classification map of the building remote sensing image by relearning the classification uncertainty region includes:

[0016] Calculating an error statistical distribution of the initial class probability map, and identifying a classification uncertainty region in the initial class probability map based on the error statistical distribution;

[0017] Extracting features from the classification uncertainty region to obtain an edge feature vector;

[0018] The edge feature vector is input into a pre-built multi-layer perceptron for reclassification, and the initial category probability map is updated according to the reclassification result to obtain a classification map of the building remote sensing image.

[0019] As one of the preferred solutions, the classification map is processed based on the circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, including:

[0020] Extracting an initial contour map of the building in the building remote sensing image according to the classification map, and extracting a plurality of contour nodes according to the initial contour map;

[0021] Based on the circular convolution operation, feature extraction is performed on each of the contour nodes to obtain a first topological structure feature;

[0022] The first topological structure feature is input into a pre-built coordinate decoder for optimization to obtain the topological structure feature of each node in the edge contour of the building in the remote sensing image of the building, wherein the optimization is designed to make the coordinates of the contour nodes close to the real coordinates of the building in the remote sensing image of the building through gradient backpropagation.

[0023] As one of the preferred solutions, the text description data corresponding to the remote sensing image of the building is split into word element vectors, and each word element vector representation is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector, including:

[0024] Splitting the text description data into word units by a word segmenter to obtain a plurality of word units;

[0025] Input each of the word units into the pre-built BERT model to obtain a word unit vector;

[0026] The word element vector is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0027] As one of the preferred solutions, mapping the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector includes:

[0028] Mapping the global visual feature vector and the global semantic feature vector to the same multimodal space using a linear transformation;

[0029] Calculating the cosine similarity between the global visual feature vector and the global semantic feature vector after linear transformation, and aligning the global visual feature vector and the global semantic feature vector according to the cosine similarity;

[0030] The Transformer structure based on the attention mechanism fusion fuses the aligned global visual feature vector and the global semantic feature vector to obtain the multimodal global feature vector.

[0031] As one preferred solution, the step of inputting the multimodal global feature vector into a pre-constructed building function classification model and determining the result output by the building function classification model as the function recognition result of the building remote sensing image comprises:

[0032] Inputting the multimodal global feature vector into a pre-built building function classification model, and outputting a probability distribution reflecting the building type in the remote sensing image of the building;

[0033] Assigning a corresponding functional category label to each pixel point of the building remote sensing image according to the probability distribution;

[0034] A segmentation result of the building remote sensing image is obtained according to the functional category label.

[0035] Another embodiment of the present application provides a building function recognition device based on visual language knowledge reasoning, comprising:

[0036] A visual feature extraction module is used to extract features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, input the global context feature map into a pre-built multi-layer perceptron for classification, and obtain an initial category probability map of the building remote sensing image;

[0037] A screening module, used for identifying a classification uncertainty region in the initial classification probability map, screening the classification uncertainty region based on a machine learning algorithm, and obtaining a classification map of the building remote sensing image;

[0038] A correction module is used to process the classification map based on a circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, and to correct the global context feature map according to the topological structure features to obtain a global visual feature vector;

[0039] A semantic feature extraction module, used for splitting the acquired text description data corresponding to the remote sensing image of the building into word element vectors, encoding each word element vector representation through a multi-layer bidirectional self-attention mechanism, and obtaining a global semantic feature vector;

[0040] The recognition module is used to map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector, input the multimodal global feature vector into a pre-constructed building function classification model, and determine the result output by the building function classification model as the function recognition result of the building remote sensing image.

[0041] Another embodiment of the present application provides a building function identification device based on visual language knowledge reasoning, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements the building function identification method based on visual language knowledge reasoning as described above.

[0042] Yet another embodiment of the present application provides a computer-readable storage medium storing a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the method for identifying building functions based on visual language knowledge reasoning as described above is implemented.

[0043] Compared with the prior art, the beneficial effects of the embodiments of the present application are at least one of the following:

[0044] (1) By extracting the global context feature map, the present application can capture the overall information of the building and its surrounding environment, thereby more accurately identifying the different functional areas of the building. In addition, by relearning the classification uncertainty area, the present application can further optimize the segmentation results and improve the segmentation accuracy.

[0045] (2) This application uses circular convolution operations to process classification graphs, which can extract the topological structure features of each node in the edge contour of the building. These features are of great significance for processing fuzzy boundaries and complex scenes. At the same time, multimodal feature fusion technology combines global visual feature vectors and global semantic feature vectors to further enhance the robustness of the model.

[0046] (3) This application combines deep learning techniques, such as multi-layer perceptrons and multi-layer bidirectional self-attention mechanisms, to efficiently process large amounts of remote sensing image data and improve the efficiency of building function segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of a method for identifying building functions based on visual language knowledge reasoning in one embodiment of the present application;

[0048] Figure 2 It is a structural schematic diagram of a building function identification device based on visual language knowledge reasoning in one embodiment of the present application;

[0049] Figure 3 It is a diagram of building function identification equipment based on visual language knowledge reasoning in one of the embodiments of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. The purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] In the description of this application, the terms "first", "second", "third", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0052] In the description of the present application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be a connection between the two elements. The terms "vertical", "horizontal", "left", "right", "upper", "lower" and similar expressions used in this article are only for illustrative purposes, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on this application. The term "and / or" used in this article includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0053] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as those commonly understood by those skilled in the art. The terms used in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. For those of ordinary skill in the art, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0054] It should be noted in advance that traditional building semantic segmentation technology usually uses a bottom-up layer-by-layer abstraction method for high-level semantic recognition. This method is difficult to intuitively reflect the architectural geometry or geographic spatial context features that the interpretation of building attribute functions depends on. For example, office buildings have higher floors and teaching buildings have lower floors. Playgrounds and teaching buildings, green belts and residential buildings usually appear together, resulting in poor interpretability of traditional models in semantic segmentation, insufficient self-cognitive reasoning ability, and contradictory to the top-down cognitive method of the human brain. On the other hand, although traditional land use classification methods can use spectral information, spatial texture or social media information for functional attribute recognition, they often discretize and abstract geographic objects, making it difficult to effectively express the spatial and temporal associations between buildings, and the fine-grained land use functions at the plot scale are difficult to meet the division of human activity space based on building units.

[0055] How to break through the multi-modal heterogeneity problem between the structured expression of the spatial association knowledge of building functions and the Euclidean space of visual images, and realize the reasoning of building functional attributes at footprint scale guided by spatial association knowledge, is a key issue that needs to be solved urgently.

[0056] In summary, an embodiment of the present application provides a method for identifying building functions based on visual language knowledge reasoning. For details, see Figure 1 , Figure 1 The flowchart of the method for identifying building functions based on visual language knowledge reasoning in one embodiment of the present application is shown, including steps S1-S5:

[0057] S1: extracting features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, inputting the global context feature map into a pre-built multi-layer perceptron for classification, and obtaining an initial category probability map of the building remote sensing image;

[0058] It is understandable that in the task of classifying remote sensing images of buildings, the purpose of feature extraction is to extract useful information from the original image, which can help subsequent classifiers better understand and distinguish different image contents.

[0059] Preferably, in one embodiment of the present application, extracting features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image includes:

[0060] The remote sensing image of the building is subjected to denoising, contrast enhancement and correction processing respectively to obtain a preprocessed remote sensing image;

[0061] Segmenting the preprocessed remote sensing image to obtain a plurality of feature blocks, performing a nested multi-layer key-value query operation between each of the feature blocks to obtain an operation result, wherein the operation result reflects the spatial association between different feature blocks;

[0062] A neural network is constructed according to the calculation results, and the preprocessed remote sensing image is input into the neural network for feature extraction to obtain a global context feature map of the building remote sensing image.

[0063] Specifically, remote sensing images may be interfered by various noises during the acquisition and transmission process, such as Gaussian noise, salt and pepper noise, etc. These noises will seriously affect the quality of the image and the subsequent feature extraction effect. Therefore, before feature extraction, the image needs to be denoised. Common denoising methods include mean filtering, median filtering, Gaussian filtering, etc. Contrast is the brightness difference between different areas in an image. Enhancing contrast can improve the visibility of important features in the image and make the target in the image more prominent. Common contrast enhancement methods include histogram equalization, adaptive contrast enhancement, etc. Correction processing mainly includes geometric correction and radiation correction. Geometric correction is used to correct geometric distortion in the image, such as tilt, distortion, etc. Radiation correction is used to adjust the radiation characteristics of the image to make it more consistent with the actual situation.

[0064] In this step, the preprocessed remote sensing image is segmented to obtain multiple feature blocks. These feature blocks are local areas in the image, which contain different features in the image. There are many methods for segmentation, such as threshold-based segmentation, edge-based segmentation, region-based segmentation, etc. By performing nested multi-layer key-value query operations between each feature block, the operation results reflecting the spatial association between different feature blocks can be obtained. This method utilizes the attention mechanism in deep learning and captures the spatial relationship between different feature blocks by calculating the similarity and weight between them.

[0065] Based on the results of the nested multi-layer key-value query operation, a neural network is constructed for feature extraction. This neural network can be a convolutional neural network (CNN) or other types of deep learning models. The preprocessed remote sensing image is input into the neural network, and feature extraction and dimensionality reduction are performed through structures such as convolutional layers and pooling layers, and finally a global context feature map of the building remote sensing image is obtained.

[0066] S2: identifying a classification uncertainty region in the initial category probability map, screening the classification uncertainty region based on a machine learning algorithm, and obtaining a classification map of the building remote sensing image;

[0067] It can be understood that the global context feature map contains global features and context information in the image, which is very important for the subsequent classification task. By inputting the global context feature map into the pre-built multi-layer perceptron for classification, the initial category probability map of the building remote sensing image can be obtained.

[0068] Preferably, in one embodiment of the present application, the identifying the classification uncertainty region in the initial class probability map and obtaining the classification map of the building remote sensing image by relearning the classification uncertainty region includes:

[0069] Calculating an error statistical distribution of the initial class probability map, and identifying a classification uncertainty region in the initial class probability map based on the error statistical distribution;

[0070] Extracting features from the classification uncertainty region to obtain an edge feature vector;

[0071] The edge feature vector is input into a pre-built multi-layer perceptron for reclassification, and the initial category probability map is updated according to the reclassification result to obtain a classification map of the building remote sensing image.

[0072] In this step, in order to identify the classification uncertainty area, it is necessary to calculate the error statistical distribution of the initial class probability map. This is achieved by comparing the difference between the initial class probability map and the true label (if any), or by evaluating the discreteness between different class probabilities.

[0073] After calculating the error statistics, a threshold is set to identify the classification uncertainty regions. Classification uncertainty regions are usually those regions with large errors and more dispersed class probability distributions. By setting a reasonable threshold, these uncertainty regions can be extracted from the initial class probability map.

[0074] For the identified classification uncertainty area, more detailed feature extraction is performed. This can be achieved by using deep convolutional neural networks (CNN) or other feature extraction algorithms. The edge feature vector is obtained through feature extraction. The vector contains the edge information and texture information of the pixel points in the classification uncertainty area. This information is used to improve the classification accuracy.

[0075] These edge feature vectors are input into a pre-built multi-layer perceptron (or other classifier) ​​for reclassification. This multi-layer perceptron can be specially designed to process such features, or it can be migrated from other similar tasks. Through reclassification, the new category probabilities of the pixels in these uncertainty areas can be obtained. According to the results of the reclassification, the initial category probability map is updated to obtain a more accurate classification map of the building remote sensing image.

[0076] S3: Processing the classification map based on a circular convolution operation method to obtain topological structure features of each node in the edge contour of the building in the building remote sensing image, and correcting the global context feature map according to the topological structure features to obtain a global visual feature vector;

[0077] Preferably, in one embodiment of the present application, the classification graph is processed based on the circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, including:

[0078] Extracting an initial contour map of the building in the building remote sensing image according to the classification map, and extracting a plurality of contour nodes according to the initial contour map;

[0079] Based on the circular convolution operation, feature extraction is performed on each of the contour nodes to obtain a first topological structure feature;

[0080] The first topological structure feature is input into a pre-built coordinate decoder for optimization to obtain the topological structure feature of each node in the edge contour of the building in the remote sensing image of the building, wherein the optimization is designed to make the coordinates of the contour nodes close to the real coordinates of the building in the remote sensing image of the building through gradient backpropagation.

[0081] Specifically, the classification map is binarized to distinguish between building areas and non-building areas, and then the preliminary outline of the building is outlined; image processing techniques (such as edge detection, contour tracking, etc.) are used to extract multiple contour nodes from the initial contour map. These contour nodes represent the key positions of the building edge.

[0082] For each extracted contour node, circular convolution operation is applied for feature extraction. Circular convolution is a special convolution operation that uses a circular filter to slide on the image and extracts features by calculating the dot product between the filter and the local area of ​​the image. This method can capture the local texture, shape and other information around the contour node to form the first topological structure features. These features reflect the local geometric characteristics of the building edge contour.

[0083] The first topological structure feature is input into the pre-built coordinate decoder for optimization. The coordinate decoder is a deep learning model that can predict and optimize the coordinates of contour nodes by learning the mapping relationship between the topological structure features of the building edge contour and the node coordinates. During the optimization process, the error between the predicted coordinates and the true coordinates is calculated by using the gradient backpropagation algorithm, and the parameters of the coordinate decoder are continuously adjusted so that the predicted coordinates gradually approach the true coordinates of the building in the building remote sensing image.

[0084] After the coordinate decoder is optimized, more accurate topological features of each node in the building edge outline are obtained. These features not only contain the local geometric information of the building edge, but also reflect the spatial relationship between nodes. Next, these topological features are used to correct the global context feature map. The correction process may involve weighting, filtering, or replacing specific areas in the global context feature map to better reflect the topological information of the building edge. The corrected global context feature map is further processed to generate a global visual feature vector.

[0085] S4: splitting the acquired text description data corresponding to the remote sensing image of the building into word element vectors, encoding each word element vector representation through a multi-layer bidirectional self-attention mechanism, and obtaining a global semantic feature vector;

[0086] By downloading the public "remote sensing image-text description" sample pairs as input (such as the RSICD dataset, RSITMD dataset, UCMerced dataset, etc., whose data organization form is remote sensing image data such as airport scenes, and their corresponding description texts such as "Some planes are parked in an airport near a piece of green trees"), through the building spatial association knowledge language instruction encoding module, it aims to train a language model that is aligned with the visual features of the image and can understand visual clues for spatial reasoning.

[0087] The language model learns the semantic association of building function attribute words from the image description language text, embeds the spatially associated ground object entity concepts near each other in the latent space, thereby learning the topological association relationship between the ground objects and assisting the reasoning of the building function. At the same time, the ground object entity visual encoding and the corresponding text semantic concept encoding obtained by the language model are aligned in the latent space, so that the language model can understand the image visual clues.

[0088] Preferably, in one embodiment of the present application, the acquired text description data corresponding to the remote sensing image of the building is split into word element vectors, and each word element vector representation is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector, including:

[0089] Splitting the text description data into word units by a word segmenter to obtain a plurality of word units;

[0090] Input each of the word units into the pre-built BERT model to obtain a word unit vector;

[0091] The word element vector is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

[0092] In this step, a tokenizer is used to process the text description data and split it into multiple independent tokens. The tokenizer identifies and segments words or phrases in the text based on language rules (such as Chinese word segmentation rules) or pre-trained models. Each token is input into the pre-built BERT (Bidirectional Encoder Representations from Transformers) model. BERT is a pre-trained language representation model based on the Transformer architecture, which can capture the contextual relationship between tokens and generate token vectors containing rich semantic information.

[0093] In the BERT model, word vectors are processed by a multi-layer bidirectional self-attention mechanism. This mechanism allows each word to pay attention to the text information before and after it when encoding, thereby capturing global semantic features more accurately. Through multi-layer stacking, the model can gradually abstract higher-level semantic information and ultimately generate a global semantic feature vector.

[0094] S5: Map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector, input the multimodal global feature vector into a pre-constructed building function classification model, and determine the result output by the building function classification model as the function recognition result of the building remote sensing image.

[0095] Preferably, in one embodiment of the present application, mapping the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector comprises:

[0096] Mapping the global visual feature vector and the global semantic feature vector to the same multimodal space using a linear transformation;

[0097] Calculating the cosine similarity between the global visual feature vector and the global semantic feature vector after linear transformation, and aligning the global visual feature vector and the global semantic feature vector according to the cosine similarity;

[0098] The Transformer structure based on the attention mechanism fusion fuses the aligned global visual feature vector and the global semantic feature vector to obtain the multimodal global feature vector.

[0099] Specifically, in order to achieve the alignment of the global visual feature vector (processing result from the building remote sensing image) and the global semantic feature vector (processing result from the text description data), this application uses linear transformation to map the two into the same multimodal space. This process can be achieved by learning a linear mapping matrix that can transform the two feature vectors into new vectors with the same dimension and similar distribution.

[0100] The cosine similarity between the global visual feature vector and the global semantic feature vector is calculated in the multimodal space. Cosine similarity is an indicator that measures the directional similarity of two vectors. The closer its value is to 1, the more consistent the directions of the two vectors are. Based on the calculation result of the cosine similarity, the global visual feature vector and the global semantic feature vector can be aligned, that is, their directions and / or lengths can be adjusted to make them closer in the multimodal space.

[0101] The Transformer structure based on attention mechanism fusion is used to fuse the aligned global visual feature vector and the global semantic feature vector. The Transformer structure is a powerful sequence modeling tool that can capture the long-term dependencies between elements in a sequence. This application uses the attention mechanism of the Transformer to dynamically adjust the weights between the two feature vectors, thereby achieving effective fusion of the two feature vectors. The fused result is a multimodal global feature vector, which integrates the global information in the building remote sensing image and text description data, and provides strong support for subsequent tasks (such as building recognition, classification, retrieval, etc.).

[0102] Preferably, in one embodiment of the present application, the step of inputting the multimodal global feature vector into a pre-constructed building function classification model and determining the result output by the building function classification model as the function recognition result of the building remote sensing image comprises:

[0103] Inputting the multimodal global feature vector into a pre-built building function classification model, and outputting a probability distribution reflecting the building type in the remote sensing image of the building;

[0104] Assigning a corresponding functional category label to each pixel point of the building remote sensing image according to the probability distribution;

[0105] A segmentation result of the building remote sensing image is obtained according to the functional category label.

[0106] Specifically, the present application inputs the multimodal global feature vector into a pre-built building function classification model. The model has been optimized through a large amount of training data and can accurately understand and identify multiple functional features in building remote sensing images. After receiving the multimodal global feature vector, the building function classification model will use its internal complex neural network structure to perform calculations and finally output a probability distribution reflecting the building type in the building remote sensing image. Each element in this distribution represents a possible building type, and its value represents the probability of that type appearing in the image.

[0107] According to the probability distribution output by the building function classification model, a corresponding functional category label is assigned to each pixel of the building remote sensing image. This mapping process may involve some additional processing steps, such as threshold setting, maximum probability selection, etc., to ensure that each pixel can be accurately classified into its most likely functional category.

[0108] After the above steps, a segmentation result reflecting the functional category of each pixel in the building remote sensing image will be obtained. This result is presented in the form of an image, where different colors or marks represent different functional categories of buildings. The segmentation result can be used in a variety of practical application scenarios, such as urban planning, environmental monitoring, disaster assessment, etc.

[0109] Compared with the prior art, the beneficial effects of the embodiments of the present application are at least one of the following:

[0110] (1) By extracting the global context feature map, the present application can capture the overall information of the building and its surrounding environment, thereby more accurately identifying the different functional areas of the building. In addition, by relearning the classification uncertainty area, the present application can further optimize the segmentation results and improve the segmentation accuracy.

[0111] (2) This application uses circular convolution operations to process classification graphs, which can extract the topological structure features of each node in the edge contour of the building. These features are of great significance for processing fuzzy boundaries and complex scenes. At the same time, multimodal feature fusion technology combines global visual feature vectors and global semantic feature vectors to further enhance the robustness of the model.

[0112] (3) This application combines deep learning techniques, such as multi-layer perceptrons and multi-layer bidirectional self-attention mechanisms, to efficiently process large amounts of remote sensing image data and improve the efficiency of building function segmentation.

[0113] Another embodiment of the present invention provides a building function recognition device based on visual language knowledge reasoning. For details, see Figure 2 , Figure 2The structure diagram of the building function recognition device based on visual language knowledge reasoning provided in one embodiment of the present invention is shown, including: a visual feature extraction module 11, a screening module 12, a correction module 13, a semantic feature extraction module 14, and a recognition module 15, wherein:

[0114] The visual feature extraction module 11 is used to extract features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, input the global context feature map into a pre-built multi-layer perceptron for classification, and obtain an initial category probability map of the building remote sensing image;

[0115] A screening module 12 is used to identify the classification uncertainty region in the initial category probability map, and screen the classification uncertainty region based on a machine learning algorithm to obtain a classification map of the building remote sensing image;

[0116] A correction module 13 is used to process the classification map based on a circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, and to correct the global context feature map according to the topological structure features to obtain a global visual feature vector;

[0117] A semantic feature extraction module 14 is used to split the acquired text description data corresponding to the building remote sensing image into word element vectors, encode each word element vector representation through a multi-layer bidirectional self-attention mechanism, and obtain a global semantic feature vector;

[0118] The recognition module 15 is used to map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector, input the multimodal global feature vector into a pre-constructed building function classification model, and determine the result output by the building function classification model as the function recognition result of the building remote sensing image.

[0119] Another embodiment of the present invention provides a building function recognition device based on visual language knowledge reasoning. Figure 3 , which is a structural block diagram of a building function recognition device based on visual language knowledge reasoning provided in an embodiment of the present application. The building function recognition device based on visual language knowledge reasoning provided in an embodiment of the present application includes a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21. When the processor 21 executes the computer program, the steps in the above-mentioned building function recognition method embodiment based on visual language knowledge reasoning are implemented, for example Figure 1or, when the processor 21 executes the computer program, the functions of the modules in the above-mentioned device embodiments are implemented, such as the visual feature extraction module 11.

[0120] Exemplarily, the computer program can be divided into one or more modules, and the one or more modules are stored in the memory 22 and executed by the processor 21 to complete the present application. The one or more modules can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the building function recognition device based on visual language knowledge reasoning. For example, the computer program can be divided into a visual feature extraction module 11, a screening module 12, a correction module 13, a semantic feature extraction module 14, and a recognition module 15.

[0121] The building function recognition device based on visual language knowledge reasoning may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art may understand that the schematic diagram is only an example of the building function recognition device based on visual language knowledge reasoning, and does not constitute a limitation on the building function recognition device based on visual language knowledge reasoning. It may include more or less components than shown in the figure, or combine certain components, or different components. For example, the building function recognition based on visual language knowledge reasoning may also include input and output devices, network access devices, buses, etc.

[0122] The processor 21 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 21 is the control center of the building function recognition device based on visual language knowledge reasoning, and uses various interfaces and lines to connect various parts of the entire building function recognition device based on visual language knowledge reasoning.

[0123] The memory 22 can be used to store the computer program and / or module. The processor 21 realizes various functions of the building function recognition device based on visual language knowledge reasoning by running or executing the computer program and / or module stored in the memory 22 and calling the data stored in the memory 22. The memory 22 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0124] Wherein, if the module integrated with the building function recognition device based on visual language knowledge reasoning is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0125] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0126] Accordingly, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to perform the steps in the building function recognition method based on visual language knowledge reasoning as described in the above embodiment, for example Figure 1 Steps S1 to S5 described in .

[0127] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A building function recognition method based on visual language knowledge reasoning, characterized in that: include: Performing feature extraction on the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, inputting the global context feature map into a pre-built multi-layer perceptron for classification, and obtaining an initial category probability map of the building remote sensing image; Identifying a classification uncertainty region in the initial class probability map, screening the classification uncertainty region based on a machine learning algorithm, and obtaining a classification map of the building remote sensing image; The classification map is processed based on a circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, and the global context feature map is corrected according to the topological structure features to obtain a global visual feature vector; Splitting the acquired text description data corresponding to the remote sensing image of the building into word element vectors, encoding each word element vector representation through a multi-layer bidirectional self-attention mechanism, and obtaining a global semantic feature vector; The global visual feature vector and the global semantic feature vector are mapped into a multimodal space for alignment processing to obtain a multimodal global feature vector, the multimodal global feature vector is input into a pre-constructed building function classification model, and the result output by the building function classification model is determined as the function recognition result of the building remote sensing image.

2. The method for identifying building functions based on visual language knowledge reasoning as claimed in claim 1, characterized in that: The step of extracting features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image includes: The remote sensing image of the building is subjected to denoising, contrast enhancement and correction processing respectively to obtain a preprocessed remote sensing image; Segmenting the preprocessed remote sensing image to obtain a plurality of feature blocks, performing a nested multi-layer key-value query operation between each of the feature blocks to obtain an operation result, wherein the operation result reflects the spatial association between different feature blocks; A neural network is constructed according to the calculation results, and the preprocessed remote sensing image is input into the neural network for feature extraction to obtain a global context feature map of the building remote sensing image.

3. The method for identifying building functions based on visual language knowledge reasoning as claimed in claim 1, characterized in that: The step of identifying a classification uncertainty region in the initial classification probability map and obtaining a classification map of the building remote sensing image by relearning the classification uncertainty region includes: Calculating an error statistical distribution of the initial class probability map, and identifying a classification uncertainty region in the initial class probability map based on the error statistical distribution; Extracting features from the classification uncertainty region to obtain an edge feature vector; The edge feature vector is input into a pre-built multi-layer perceptron for reclassification, and the initial category probability map is updated according to the reclassification result to obtain a classification map of the building remote sensing image.

4. The method for identifying building functions based on visual language knowledge reasoning as claimed in claim 1, characterized in that: The circular convolution operation method is used to process the classification graph to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, including: Extracting an initial contour map of the building in the building remote sensing image according to the classification map, and extracting a plurality of contour nodes according to the initial contour map; Based on the circular convolution operation, feature extraction is performed on each of the contour nodes to obtain a first topological structure feature; The first topological structure feature is input into a pre-built coordinate decoder for optimization to obtain the topological structure feature of each node in the edge contour of the building in the remote sensing image of the building, wherein the optimization is designed to make the coordinates of the contour nodes close to the real coordinates of the building in the remote sensing image of the building through gradient backpropagation.

5. The method for identifying building functions based on visual language knowledge reasoning as claimed in claim 1, characterized in that: The obtained text description data corresponding to the building remote sensing image is split into word element vectors, and each word element vector representation is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector, including: Splitting the text description data into word units by a word segmenter to obtain a plurality of word units; Input each of the word units into the pre-built BERT model to obtain a word unit vector; The word element vector is encoded through a multi-layer bidirectional self-attention mechanism to obtain a global semantic feature vector.

6. The method for identifying building functions based on visual language knowledge reasoning as claimed in claim 1, characterized in that: Mapping the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector includes: Mapping the global visual feature vector and the global semantic feature vector to the same multimodal space using a linear transformation; Calculating the cosine similarity between the global visual feature vector and the global semantic feature vector after linear transformation, and aligning the global visual feature vector and the global semantic feature vector according to the cosine similarity; The Transformer structure based on the attention mechanism fusion fuses the aligned global visual feature vector and the global semantic feature vector to obtain the multimodal global feature vector.

7. The method for identifying building functions based on visual language knowledge reasoning as claimed in claim 1, characterized in that: The step of inputting the multimodal global feature vector into a pre-constructed building function classification model and determining the result output by the building function classification model as the function recognition result of the building remote sensing image comprises: Inputting the multimodal global feature vector into a pre-built building function classification model, and outputting a probability distribution reflecting the building type in the remote sensing image of the building; Assigning a corresponding functional category label to each pixel point of the building remote sensing image according to the probability distribution; A segmentation result of the building remote sensing image is obtained according to the functional category label.

8. A building function recognition device based on visual language knowledge reasoning, characterized in that: include: A visual feature extraction module is used to extract features from the acquired building remote sensing image to obtain a global context feature map of the building remote sensing image, input the global context feature map into a pre-built multi-layer perceptron for classification, and obtain an initial category probability map of the building remote sensing image; A screening module, used for identifying a classification uncertainty region in the initial classification probability map, screening the classification uncertainty region based on a machine learning algorithm, and obtaining a classification map of the building remote sensing image; A correction module is used to process the classification map based on a circular convolution operation method to obtain the topological structure features of each node in the edge contour of the building in the building remote sensing image, and to correct the global context feature map according to the topological structure features to obtain a global visual feature vector; A semantic feature extraction module, used for splitting the acquired text description data corresponding to the remote sensing image of the building into word element vectors, encoding each word element vector representation through a multi-layer bidirectional self-attention mechanism, and obtaining a global semantic feature vector; The recognition module is used to map the global visual feature vector and the global semantic feature vector into a multimodal space for alignment processing to obtain a multimodal global feature vector, input the multimodal global feature vector into a pre-constructed building function classification model, and determine the result output by the building function classification model as the function recognition result of the building remote sensing image.

9. A building function recognition device based on visual language knowledge reasoning, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, the method for identifying building functions based on visual language knowledge reasoning as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the device where the computer-readable storage medium is located executes the computer program, the method for identifying building functions based on visual language knowledge reasoning as described in any one of claims 1 to 7 is implemented.

Images