A training method and device for building use recognition model based on multi-task learning

Abstract

The present application discloses a training method and device for a building use recognition model based on multi-task learning, wherein the building use recognition model determines a building use recognition result and a building roof recognition result based on a sample remote sensing image of a sample area; determines a building use loss based on the building use recognition result and a building use label, wherein the building use label indicates the building use of the sample area, and the building use loss is used to characterize the deviation between the building use recognition result and the building use label; determines a building roof loss based on the building roof recognition result and the building roof label, wherein the building roof label characterizes the building roof of the sample area, and the building roof loss characterizes the deviation between the building roof recognition result and the building roof label; determines a comprehensive loss based on the building use loss and the building roof loss; and adjusts the parameters of the building use recognition model based on the comprehensive loss. The embodiments of the present application improve the accuracy of the model in identifying the building use.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a training method and device for a building use recognition model based on multi-task learning. Background Art

[0002] Building use estimation has strong social and economic value. In the insurance industry, different uses of buildings often reflect different vulnerabilities of buildings. Accurately knowing the building use of the insured building can help better assess the potential loss risk of the insured building, obtain more reasonable insurance pricing, and better share risks. In the field of urban planning, accurate information on building use can be used for better urban analysis, simulation, and policy formulation.

[0003] At present, some studies have trained artificial intelligence models based on remote sensing images. The input of the model includes remote sensing images, and the model recognizes the use of buildings based on the visual features of the buildings contained in the remote sensing images.

[0004] The building roof information in remote sensing images is of great help in estimating the building use. However, current models often pay insufficient attention to the building roof information when performing building use identification tasks, resulting in poor accuracy of the model in performing building use identification tasks. Summary of the invention

[0005] The embodiments of the present application provide a training method and device for a building use recognition model based on multi-task learning. By designing a building roof recognition task as an auxiliary task of the building use recognition task, the accuracy of the model in performing the building use recognition task is improved.

[0006] In a first aspect, a training method for a building use recognition model based on multi-task learning is provided, the method comprising:

[0007] Determine building use recognition results and building roof recognition results based on sample remote sensing images of sample areas through a building use recognition model;

[0008] Determine a building use loss based on the building use identification result and the building use label, wherein the building use label is used to indicate the building use of the sample area, and the building use loss is used to characterize the deviation between the building use identification result and the building use label;

[0009] Determine a building roof loss based on the building roof recognition result and the building roof label, wherein the building roof label is used to characterize the building roof of the sample area, and the building roof loss is used to characterize the deviation between the building roof recognition result and the building roof label;

[0010] Determine a comprehensive loss based on the building use loss and the building roof loss, wherein the comprehensive loss is used to characterize the overall deviation of the building use identification result and the building roof identification result;

[0011] The parameters of the building use identification model are adjusted based on the comprehensive loss so that the deviation of the building use identification result and the deviation of the building roof identification result output by the building use identification model both meet conditions.

[0012] In some embodiments, determining the comprehensive loss based on the loss of use of the building and the loss of the roof of the building includes:

[0013] summing the loss of the building's use and the loss of the building's roof to obtain a combined loss; or,

[0014] Based on the weight corresponding to the loss of the building's use and the weight corresponding to the loss of the building's roof, a weighted sum is performed on the loss of the building's use and the loss of the building's roof to obtain a comprehensive loss.

[0015] In some embodiments, the method further comprises:

[0016] If the deviation of the building use identification result output by the building use identification model is less than the deviation threshold and the deviation of the building roof identification result is greater than the deviation threshold, reduce the weight corresponding to the building use loss or increase the weight corresponding to the building roof loss;

[0017] If the deviation of the building use identification result output by the building use identification model is greater than the deviation threshold and the deviation of the building roof identification result is less than the deviation threshold, reduce the weight corresponding to the building roof loss or increase the weight corresponding to the building use loss.

[0018] In some embodiments, determining the building use recognition result and the building roof recognition result based on the sample remote sensing image of the sample area by using the building use recognition model includes:

[0019] Based on the sample remote sensing image of the sample area, the building use recognition result of each pixel point in the sample remote sensing image and the building roof recognition result of each pixel point are determined by the building use recognition model; accordingly,

[0020] The pixel value of each pixel point in the building use label is a first pixel value, a second pixel value, a third pixel value or a fourth pixel value, the first pixel value represents the background, the second pixel value represents that the building use is residential, the third pixel value represents that the building use is commercial, and the fourth pixel value represents that the building use is industrial, and the pixel value of each pixel point in the building roof label is a fifth pixel value or a sixth pixel value, the fifth pixel value represents the background, and the sixth pixel value represents the roof; accordingly,

[0021] The determining of the building use loss based on the building use identification result and the building use label includes:

[0022] Determine the building use loss based on the probability that the building use of each pixel point in the building use identification result is identified as residential, the probability that the building use of each pixel point is identified as commercial, the probability that the building use of each pixel point is identified as industrial, and the pixel value of each pixel point in the building use label;

[0023] The determining of the building roof loss based on the building roof recognition result and the building roof label includes:

[0024] The building roof loss is determined based on the probability that the building use of each pixel point in the building use identification result is identified as a roof and the pixel value of each pixel point in the building roof label.

[0025] In some embodiments, determining the building use recognition result and the building roof recognition result based on the sample remote sensing image of the sample area by using the building use recognition model includes:

[0026] fusing the sample remote sensing image of the sample area and the social attribute data of the sample area to obtain sample heterogeneous data;

[0027] A building use recognition result and a building roof recognition result are determined based on the sample heterogeneous data through a building use recognition model.

[0028] In some implementations, fusing the sample remote sensing image of the sample area and the social attribute data of the sample area to obtain sample heterogeneous data includes:

[0029] The sample remote sensing image and the sample social attribute data are spliced ​​in the channel dimension to obtain sample heterogeneous data, wherein the sample remote sensing image includes m channels, the sample social attribute data includes n channels, and the sample heterogeneous data includes the m channels of the sample remote sensing image and the n channels of the sample social attribute data;

[0030] The building use recognition model includes an encoder and a decoder, and determining a building use recognition result and a building roof recognition result based on the sample heterogeneous data by using the building use recognition model includes:

[0031] Encoding the sample heterogeneous data by the encoder to obtain fusion features of the sample remote sensing image and the sample social attribute data;

[0032] The decoder is used to decode the fusion features of the sample remote sensing image and the sample social attribute data to obtain the building use recognition result and the building roof recognition result of the sample area.

[0033] In some embodiments, the area of ​​the sample region identified at a single time is between 500 meters*500 meters and 1 kilometer*1 kilometer.

[0034] In a second aspect, a training device for a building use recognition model based on multi-task learning is provided, the device comprising:

[0035] An identification unit, used to determine a building use identification result and a building roof identification result based on a sample remote sensing image of a sample area through a building use identification model;

[0036] A determination unit is configured to determine a building use loss based on the building use identification result and the building use label, wherein the building use label is used to indicate the building use of the sample area, and the building use loss is used to characterize the deviation between the building use identification result and the building use label; determine a building roof loss based on the building roof identification result and the building roof label, wherein the building roof label is used to characterize the building roof of the sample area, and the building roof loss is used to characterize the deviation between the building roof identification result and the building roof label; determine a comprehensive loss based on the building use loss and the building roof loss, wherein the comprehensive loss is used to characterize the deviation of the building use identification result and the building roof identification result as a whole;

[0037] An adjustment unit is used to adjust the parameters of the building use identification model based on the comprehensive loss so that the deviation of the building use identification result and the deviation of the building roof identification result output by the building use identification model both meet conditions.

[0038] In some embodiments, the determination unit is used to sum the loss of the building use and the loss of the building roof to obtain a comprehensive loss; or, based on the weight corresponding to the loss of the building use and the weight corresponding to the loss of the building roof, perform weighted summation of the loss of the building use and the loss of the building roof to obtain a comprehensive loss.

[0039] In some embodiments, the adjustment unit is further used to reduce the weight corresponding to the building use loss or increase the weight corresponding to the building roof loss if the deviation of the building use identification result output by the building use identification model is less than the deviation threshold and the deviation of the building roof identification result is greater than the deviation threshold; if the deviation of the building use identification result output by the building use identification model is greater than the deviation threshold and the deviation of the building roof identification result is less than the deviation threshold, reduce the weight corresponding to the building roof loss or increase the weight corresponding to the building use loss.

[0040] In some embodiments, the determination unit is used to determine the building use recognition result of each pixel in the sample remote sensing image and the building roof recognition result of each pixel based on the sample remote sensing image of the sample area through the building use recognition model; accordingly, the pixel value of each pixel in the building use label is a first pixel value, a second pixel value, a third pixel value or a fourth pixel value, the first pixel value represents the background, the second pixel value represents that the building use is residential, the third pixel value represents that the building use is commercial, the fourth pixel value represents that the building use is industrial, and the pixel value of each pixel in the building roof label is a fifth pixel value or a sixth pixel value, the fifth pixel value represents the background, and the sixth pixel value represents the roof; accordingly, based on the probability that the building use of each pixel in the building use recognition result is recognized as residential, the probability that the building use of each pixel is recognized as commercial, the probability that the building use of each pixel is recognized as industrial, and the pixel value of each pixel in the building use label, the building use loss is determined; based on the probability that the building use of each pixel in the building use recognition result is recognized as roof and the pixel value of each pixel in the building roof label, the building roof loss is determined.

[0041] In some embodiments, the device further comprises: a fusion unit, configured to fuse the sample remote sensing image of the sample area and the social attribute data of the sample area to obtain sample heterogeneous data;

[0042] The recognition unit is used to determine a building use recognition result and a building roof recognition result based on the sample heterogeneous data through a building use recognition model.

[0043] In some embodiments, the fusion unit is used to splice the sample remote sensing image and the sample social attribute data in the channel dimension to obtain sample heterogeneous data, the sample remote sensing image includes m channels, the sample social attribute data includes n channels, and the sample heterogeneous data includes the m channels of the sample remote sensing image and the n channels of the sample social attribute data;

[0044] The building use identification model includes an encoder and a decoder. The identification unit is used to encode the sample heterogeneous data through the encoder to obtain the fusion features of the sample remote sensing image and the sample social attribute data; and decode the fusion features of the sample remote sensing image and the sample social attribute data through the decoder to obtain the building use identification result and the building roof identification result of the sample area.

[0045] In some embodiments, the area of ​​the sample region identified at a single time is between 500 meters*500 meters and 1 kilometer*1 kilometer.

[0046] According to a third aspect, a computer device is provided, comprising: a processor, the processor being coupled to a memory, the memory storing at least one computer program instruction, the at least one computer program instruction being loaded and executed by the processor, so that the computer device implements the method described in the first aspect or any optional method of the first aspect.

[0047] In a fourth aspect, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and when the instruction is executed on a computer, the computer executes the method described in the first aspect or any optional method of the first aspect.

[0048] In a fifth aspect, a computer program product is provided, which includes one or more computer program instructions. When the computer program instructions are loaded and executed by a computer, the computer executes the method described in the first aspect or any optional method of the first aspect.

[0049] In a sixth aspect, a chip is provided, which includes a programmable logic circuit and / or program instructions, and when the chip is running, it is used to implement the method described in the first aspect or any optional method of the first aspect.

[0050] It can be seen that the embodiments of the present application have the following beneficial effects:

[0051] An embodiment of the present application provides a model with a multi-task architecture for the task of identifying the use of a building. Taking into account that the building roof information is of great help in identifying the use of a building, for example, the size, shape, and material of the roof all have a certain correlation with the use of the building, by adding the building roof recognition task as an auxiliary task on the output side of the model, that is, letting the model be responsible for determining both the building use recognition result and the building roof recognition result, the model can better pay attention to the building roof information in the remote sensing image during the forward propagation process, share the roof information with the building use recognition task, and improve the accuracy of the model in identifying the building use. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a schematic diagram of a building use identification architecture provided by an embodiment of the present application;

[0053] Figure 2 It is a schematic diagram of a U-Net structure for implementing a building use recognition model provided in an embodiment of the present application;

[0054] Figure 3 It is a structural diagram of a ResNet residual block provided in an embodiment of the present application;

[0055] Figure 4 This is a comparison diagram of heterogeneous data before and after use provided by an embodiment of the present application;

[0056] Figure 5 It is a structural diagram of a building use recognition model based on multi-task learning provided in an embodiment of the present application;

[0057] Figure 6 It is a flowchart of a training method of a building use recognition model based on multi-task learning provided in an embodiment of the present application;

[0058] Figure 7 This is a comparison chart of 10-meter and 1-meter resolution remote sensing images provided in an embodiment of the present application;

[0059] Figure 8 This is a POI point map conversion effect diagram provided by an embodiment of the present application;

[0060] Fig. 9 This is a diagram showing the effect of implementing a 1-meter resolution POI distance color temperature map provided in an embodiment of the present application;

[0061] Fig.10 is a schematic diagram of roof offset of a remote sensing image provided by an embodiment of the present application;

[0062] Fig.11 It is a schematic diagram of roof comparison between a shp layer and a remote sensing image layer before and after manual annotation provided in an embodiment of the present application;

[0063] Fig.12 This is a comparison chart of the multi-task recognition effect and the single-task recognition effect provided by the embodiment of the present application;

[0064] Fig.13 It is a rendering of a building use interpretation provided by an embodiment of the present application;

[0065] Fig.14 It is a structural schematic diagram of a building use identification device provided in an embodiment of the present application;

[0066] Fig.15 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0068] The following is an explanation of some terminology concepts involved in the embodiments of the present application.

[0070] Catastrophes have a huge impact on life and property, social structure, economic development potential, ecological and environmental balance, etc., which makes it impossible for government emergency response and various industries to ignore the risks brought by catastrophes. Among all industries, the financial insurance industry has the most direct perception of catastrophe risks. Catastrophes directly affect the financial stability and profitability of financial insurance companies, and even affect their normal operations. Therefore, the financial insurance industry is also the most eager to manage catastrophe risks. However, compared with traditional risks, the extraordinary risks brought by catastrophes often present the characteristics of low frequency and high loss, wide and long-term impact, high complexity and uncertainty. Therefore, financial insurance catastrophe risk management often requires a comprehensive system framework, covering multiple key links such as risk identification, risk quantification, risk control and loss compensation. For the financial insurance industry, the core of the comprehensive system framework of catastrophe risk management is the catastrophe model, which helps insurance companies quantify potential losses by simulating the occurrence and impact of catastrophe events, thereby conducting risk management and pricing. Catastrophe models usually include: (1) Disaster module: using historical disaster data combined with physical mechanisms to simulate random event sets. (2) Risk exposure module: Starting from the perspective of spatial distribution, it collects detailed spatial information and attribute information of the disaster-prone objects. (3) Engineering module: This module focuses on vulnerability, and the vulnerability model considers the sensitivity of different disaster-prone objects to disasters. (4) Financial module: It is used to assess the financial impact of disasters on insurance companies and other financial institutions.

[0071] Risk exposure is the data basis for converting hazard calculations and vulnerability calculations into economic value loss calculations. It is one of the most basic data requirements for catastrophe models. It accurately reflects the economic value and geographic spatial distribution of the main disaster-bearing bodies, which is crucial to the scientificity and accuracy of model simulation results. Risk exposure data is an important input requirement for the insurance industry to further calculate catastrophe economic losses, and is also the data basis for reinsurance companies to reasonably split and calculate unified policies based on economic value distribution.

[0072] Building use refers to the intended function and purpose of a building or group of buildings. Different building uses will have different design requirements, structural characteristics and usage specifications. In the insurance industry, the different uses of buildings often reflect the different vulnerabilities of buildings, and are often divided into three categories: residential, commercial and industrial. Building use is a very important attribute, but there is currently no nationwide work on building use identification.

[0073] The national scope refers to the land area across China, covering an area of ​​approximately 9.6 million square kilometers, including urban and rural areas.

[0074] Building use identification refers to identifying the functional use of a single building. In this document, it specifically refers to identifying three types of building uses: residential, commercial, and industrial. These building uses meet the insurance use requirements. The identified building uses can be linked to specific building roofs through geographic location.

[0075] POI data refers to a point of interest (Place Of Interest). In a geographic information system, a POI can be a house, a shop, a mailbox, a bus stop, etc. Generally, POI data includes latitude and longitude information, name information, address information, purpose information, etc.

[0076] Remote sensing interpretation refers to the process of identifying and extracting information about land objects by analyzing and processing remote sensing images. It includes methods such as visual interpretation and computer digital image processing, using image features such as shape, size, color, texture, combined with professional knowledge and experience to determine the attributes and status of land objects. Remote sensing interpretation is widely used in geological surveys, environmental monitoring, resource management and other fields.

[0077] A neural network is a computational model that mimics the way neurons in the human brain connect and process information. It consists of a large number of nodes (called neurons), which are usually arranged in multiple layers. Each neuron receives an input signal, applies a nonlinear activation function after weighted summation, and generates an output signal to the next layer of neurons. Neural networks learn complex relationships and patterns between input data by adjusting the weights of connections between neurons, and are widely used in image recognition, speech processing, natural language processing and other fields. With the development of deep learning, the layers of neural networks have become more complex and in-depth, enabling the capture of more abstract data features.

[0078] Multi-task learning (MTL) is a machine learning method that improves the performance of each task by learning multiple related tasks simultaneously. The core idea of ​​multi-task learning is to use the shared information between tasks to improve the generalization ability of the model. Specifically, MTL can be achieved by sharing model parameters or feature representations, so that different tasks can learn from each other and pass useful information to improve the recognition accuracy of learning tasks.

[0079] Hidden layers are one or more layers between the input layer and the output layer of a neural network. Their purpose is to capture and learn nonlinear features and complex patterns in the input data. In these layers, each neuron receives weighted input from the previous layer, processes it through an activation function, and passes the result to the next layer. Since this information processing process of the hidden layer is invisible to the outside world, they are called "hidden" layers. The number of hidden layers and the number of neurons in each layer are important hyperparameters that need to be determined when building a neural network. They directly affect the learning ability and complexity of the model.

[0080] Forward propagation is the process of passing input data layer by layer and calculating output results in a neural network. Specifically, the input data is processed by weighted summation and activation function of each layer, and then gradually passed to the next layer until the output layer generates the final recognition result. Forward propagation is the basic step for neural network recognition and classification, ensuring that the input data can be calculated through each layer of the network to obtain the output result.

[0081] Model inference refers to the process of using a trained model to identify or classify new input data in machine learning or deep learning. Simply put, it is to apply the trained model to actual data to generate output results. For example, in image recognition, model inference is to input an unseen picture into the model and obtain the recognition of the category to which the picture belongs.

[0082] High-definition remote sensing data refers to the image data of the earth's surface obtained through high-resolution remote sensing technology, which has extremely high spatial resolution (usually 1 meter or less) and can clearly capture the details of the ground objects. These data are widely used in land use, urban planning, environmental monitoring and other fields, providing rich ground object texture information and multi-spectral band data, supporting fine ground object classification and identification.

[0083] Building vector data refers to the geometric information and attribute data of buildings expressed in vector form, which usually includes the location, shape, size and other related attribute information of the building, such as height, material, purpose, etc. These data can be edited and analyzed by geographic information system (GIS) software, and are often used in urban planning, architectural design, asset management and geospatial analysis. Building vector data can be obtained by professional surveying and mapping personnel through field measurement, or it can be automatically or semi-automatically extracted through technical means such as remote sensing images and aerial photography.

[0084] Ground truth data refers to accurate and error-free data sets used to train and validate models in the fields of machine learning and pattern recognition. These data contain input features and their corresponding correct output labels, which are the benchmarks for building and evaluating model performance. Ground truth data are usually obtained through manual annotation, historical records, or verified measurements, and are critical for supervised learning algorithms because they ensure that the model can learn the correct mapping relationship during training and accurately measure the model's recognition ability in subsequent evaluations.

[0085] Image Channel refers to an independent information source that represents color information in a digital image. In a color image, there are usually three color channels: red (R), green (G), and blue (B). Each channel contains the intensity information of the color in the image. In a grayscale image, there is only one channel, which represents brightness information. With the diversification of applications, the channels of an image can be very diverse. For example, in the field of remote sensing, other optical bands besides visible light can also be put into remote sensing images, so the file format of some remote sensing images can support ten or even dozens of channels. Image channel is a basic concept in image processing and analysis, and is often used in operations such as image enhancement, filter effects, and image segmentation.

[0086] CNN, or Convolutional Neural Network, is a deep learning model that excels in image and video recognition, natural language processing, and other fields. CNN simulates the perception mechanism of biological visual cortex on images and uses convolutional layers to automatically and layer by layer extract features of input data, which gradually evolve from simple edges and textures to complex patterns. The key features of CNN include local receptive field, weight sharing, and pooling operations, which make it very effective in processing data with grid structures, and can reduce the number of model parameters, improve training efficiency, and improve generalization capabilities.

[0087] Urban Functional Zone refers to a specific area divided according to the different functions and uses of various activities within the city. They are the basic units of urban structure and layout. These areas usually include residential areas, commercial areas, industrial areas, office areas, educational areas, entertainment areas, etc. Each functional area carries specific social and economic activities and has corresponding architectural characteristics, land use patterns and population distribution. The division of urban functional zones is of great significance for urban planning, traffic management, environmental monitoring, public service facilities configuration and urban sustainable development. Urban functional zones can be identified and divided through remote sensing images, geographic information systems (GIS) and social perception data.

[0088] Semantic Segmentation is a technique in computer vision that aims to assign each pixel in an image to a specific category label. In this way, each pixel in the image is accurately classified into a semantic category, such as road, building, pedestrian, etc. Semantic segmentation usually uses deep learning models, especially convolutional neural networks (CNN), to achieve this goal. It has a wide range of applications in autonomous driving, medical image analysis, robotic vision and other fields, helping machines better understand and parse visual scenes by providing pixel-level classification information.

[0089] The technical problem to be solved by some embodiments of the present application is to design a modeling solution for identifying the use of buildings throughout China. To solve this major technical problem, there are three core sub-technical problems that need to be solved:

[0090] (1) Design an artificial intelligence model framework suitable for nationwide modeling of building uses. After researching a large number of literature, it is found that the largest application area of ​​the current building use identification solution is one or more cities, and the modeling framework used is often difficult to promote nationwide. The reasons for the difficulty in promoting include difficulty in data acquisition and high computational complexity. Therefore, it is necessary to design a model framework that has the possibility of nationwide building use identification, the model input data used can cover the whole of China, the framework structure is simple and easy to use, the computational workload is moderate, and it can be calculated nationwide.

[0091] (2) Design an effective multi-source heterogeneous data fusion method. The use of buildings has social attributes, and it is difficult to accurately judge the use of buildings based solely on their visual appearance. Therefore, it is necessary to fuse multiple data so that the fused data includes both visual features and social features. The fused data is used as model input to enable the model to better identify the use of buildings. Existing heterogeneous data fusion methods often require a lot of data preprocessing and rule-based geographic association, which leads to a lot of information loss in the processing of heterogeneous data features. The subsequent fusion substructure can only fuse limited information, which limits the effect of the model.

[0092] (3) It is necessary to design effective auxiliary learning tasks to improve the accuracy of the building use identification task. Building roof information is of great help in identifying the purpose of the building. For example, the size, shape, and material of the roof are all related to the building use. In practical applications, it is often difficult to find existing roof information across the country for use in identifying the building use. In some embodiments of the present application, a building roof identification task is added to the output side of the model so that the model notices the roof information during the forward propagation process and shares the roof information with the building use identification task to improve the latter's recognition accuracy. This method of adding auxiliary learning tasks belongs to the category of multi-task learning technology. At present, no work has been found to use this technology in the task of building use identification, so it is necessary to reasonably design the model structure to apply this technology.

[0093] The following is an example of an application scenario of the embodiment of the present application.

[0094] The embodiments of the present application are applied to the scenario of identifying the purpose of a building.

[0095] At present, building use identification is relatively active in the field of academic research, but its application and promotion in social institutions and industries is relatively lacking. The test area for building use identification in the field of academic research generally covers an area of ​​several square kilometers to tens of square kilometers. A variety of finely processed data can be used in a relatively small area without worrying about data acquisition bottlenecks and computing power bottlenecks. From the existing literature on building use identification, the integration and use of multiple heterogeneous data can effectively improve the accuracy of building use identification. At the same time, in order to realize industrial applications, it is necessary to upgrade and expand the existing building use identification algorithm so that it can be applied to a larger area and solve practical social problems. At the same time, there is a nationwide building interpretation work in the field of remote sensing interpretation, which can be used as a reference for upgrading the building use algorithm to a larger area.

[0096] To identify building attributes nationwide, we need feature data that covers the entire country and an application framework that is suitable for model reasoning across the country. The aerospace information industry has developed rapidly in recent years. There are thousands of remote sensing satellites in orbit around the world, and a single satellite can transmit up to 20TB of data per day. Therefore, remote sensing data can cover the entire China. At the same time, as the resolution increases, remote sensing data (meter-level resolution) can provide rich ground object information to assist in interpreting building attributes.

[0097] After the model training is completed and the evaluation is available, the model can be inferred in the national area. The basic principle of the nationwide model inference is, for example, to divide the large remote sensing image into small remote sensing images that can be processed by the artificial intelligence model. After the artificial intelligence model completes the recognition, the recognition results of the small blocks are pieced back into the recognition results of the complete large geographical area. Among them, the remote sensing image itself carries accurate longitude and latitude information, and the recognition results also carry the corresponding longitude and latitude information, which can be associated with the longitude and latitude of the actual administrative area.

[0098] In addition to remote sensing images from the sky, street view image information, POI (point of interest) data, and building vector information are all helpful for identifying building uses. However, it is difficult to obtain complete street view image information and building vector information across the country, so the solution of integrating remote sensing images with POI data is considered.

[0099] POI usage can be divided into three categories that the insurance industry is concerned about: "residential", "commercial" and "industrial". Where POIs of this category are concentrated, nearby buildings are often of the same category.

[0100] Converting POI point data into POI map data and fusing it with remote sensing images using a machine vision model structure will help achieve automated identification of building uses in large areas.

[0101] Converting POI point data into a distance color temperature map can effectively depict the characteristic that "the usage classification of adjacent POIs and buildings is correlated".

[0102] In the specific implementation, for each POI category, a blank remote sensing channel of the same size as the remote sensing image of the corresponding area is created. The value of each pixel on each channel is the closest Euclidean distance from the pixel to the POI point of the corresponding category. All channels are superimposed together to form the converted POI color temperature map (also called POI distance heat map).

[0103] Multi-task learning (MTL) is a machine learning method that improves the performance of each task by learning multiple related tasks simultaneously. The core idea of ​​multi-task learning is to use the shared information between tasks to improve the generalization ability of the model. Specifically, MTL can be achieved by sharing model parameters or feature representations, so that different tasks can learn from each other and pass useful information to improve the recognition accuracy of learning tasks.

[0104] The roof information in remote sensing images is very helpful for identifying building uses. Multi-task learning technology can be used to add auxiliary tasks of building roof interpretation to help the building use identification task better notice the building roof information in remote sensing images, thereby achieving more accurate building use identification. Currently, no work has been found using multi-task learning technology for building use identification tasks.

[0105] The shortcomings of current building attribute reasoning frameworks include:

[0106] (1) The existing building use identification framework is a small-scale regional framework and does not consider the scalability of large-scale reasoning applications.

[0107] Specifically, existing building use identification frameworks usually rely on street view image information, building vector information and building roof data as model inputs, and before model processing, perform data preprocessing steps to find the building's street view data and building roof based on the building's location, and associate the building roof with other attributes such as the building's street view. However, street view image information and building roof data are not necessarily available nationwide, and the data preprocessing steps are more dependent on manpower, resulting in technical difficulties when the solution is applied to nationwide building use prediction.

[0108] In some embodiments of the present application, heterogeneous data obtained by splicing remote sensing images and social attribute images are used as input to the model. Remote sensing images and social attribute images are data that are easy to obtain and process automatically nationwide, and the model does not rely on building roof data as input to the model, and there is no need to perform preprocessing steps to associate the building roof with other attributes of the building. This expands the scope of application of the solution and improves the scalability of the solution.

[0109] Furthermore, some embodiments of the present application perform training and identification of building uses at a pixel point granularity, and the building use corresponding to each pixel point in the sample remote sensing image is marked in the label used in training, so that the model can identify the building use corresponding to each pixel point in the remote sensing image during the inference stage. Compared with the method of training and identifying building uses at a functional area granularity, the training and identification method is more refined, reducing the impact of the variable number of buildings in a single image block in the input remote sensing image on the recognition results.

[0110] (2) The existing national building attribute recognition framework does not take into account heterogeneous data input and is difficult to use for accurate building use identification.

[0111] Some embodiments of the present application use remote sensing data of some regions and social attribute images of some regions to train models based on heterogeneous data input, and use national remote sensing data and national social attribute images to identify national building uses based on heterogeneous data input.

[0112] The shortcomings of the heterogeneous data fusion model structure of building use in related technologies are as follows: the model extracts the visual information and social information from the remote sensing data and POI data in the same functional area separately, and then fuses the two types of information. This method can be regarded as information fusion based on functional areas, which may be conducive to the division of functional areas, but is not conducive to the identification of the uses of independent buildings.

[0113] In some embodiments of the present application, since the remote sensing image and the social attribute image are spliced ​​and the heterogeneous data obtained by the splicing is used as the input of the model, when the shallow layer of the model (such as the convolution layer close to the input layer in the encoder) encodes the heterogeneous data (feature extraction), the visual features of the building in the remote sensing image and the social features of the building in the social attribute image can be fully integrated during the encoding process, and the amount of information of the pixels in the remote sensing image and the social attribute image in the shallow layer of the model is better retained, which reduces the risk of losing texture information compared to feature fusion in the deep layer of the model. This fusion method can be regarded as information fusion based on pixels, so the fusion features obtained by fusion can fully express the visual features and social features of the building. After the decoder decodes the fusion features, the obtained building use recognition result is more accurate and more conducive to the use recognition of independent buildings.

[0114] The disadvantage of the current auxiliary task of building use identification is that a multi-task framework for the task of building use identification has not been found. Some implementation methods of the present application are based on a multi-task framework, and the model outputs are building use and building roof. The building roof identification task is used as the main task, and the building use identification task is used as the auxiliary task. The two tasks share a model backbone (an encoder for extracting remote sensing image features). After the encoder inputs the features extracted from the remote sensing image into the decoder, the decoder will use the image features to perform both the building roof identification task and the building use identification task, thereby achieving deep information sharing between the two tasks of building roof identification and building use identification, which helps the encoder extract image features to retain more details of the building roof, thereby allowing the decoder to pay more attention to the roof of the building when decoding the building use, which is beneficial to help the model improve the accuracy of building use identification.

[0115] The following is an example of the framework design for building use identification and roof identification in the embodiment of the present application.

[0116] Attached Figure 1 This is a schematic diagram of a building use identification architecture provided by an embodiment of the present application, as shown in the attached Figure 1 As shown, remote sensing images are mainly used in data usage. Optionally, social attribute images are added as auxiliary. Remote sensing images and social attribute images can achieve nationwide coverage. Since remote sensing images contain longitude and latitude information, the building use identified based on remote sensing images has its own longitude and latitude information, which can be easily associated with the target building in subsequent use. For example, the user inputs the longitude and latitude of the target building, matches the input longitude and latitude of the target building and the longitude and latitude information in the identified building use, and determines the building use of the target building corresponding to the input longitude and latitude.

[0117] In addition, a heterogeneous input fusion module is introduced for joint optimization. For example, the feature extraction of remote sensing images and the feature extraction of social attribute images can be performed simultaneously to achieve efficient feature utilization. For example, since heterogeneous data are encoded at the shallow level of the model, the visual features of the buildings in the remote sensing images and the social features of the buildings in the social attribute images can be fused at the shallow level of the model, reducing the risk of losing texture information due to feature fusion in the deep level of the model and improving the efficiency of feature utilization. This feature is not available in the heterogeneous national remote sensing interpretation framework.

[0118] In addition, batch recognition and end-to-end recognition are used in the recognition method. For example, batch recognition is to divide the remote sensing image and the social attribute image into blocks and splice them respectively, and then input multiple heterogeneous data blocks into the model in parallel. The encoder encodes the multiple heterogeneous data blocks in parallel, and the decoder decodes the multiple fusion features encoded by the encoder in parallel. Then, the multiple recognition result data blocks obtained by parallel decoding are spliced ​​to obtain the overall building use recognition result and building roof recognition result of the identified area. End-to-end recognition, for example, is to realize building use recognition and building roof recognition through the same model, and there is no need to perform building use recognition and building roof recognition separately through two models. In addition, no assumption is made about where the building is, and the architecture is simple and scalable. This method is not available in the existing heterogeneous data fusion building function recognition framework.

[0119] The following is an example of the model structure design of the embodiment of the present application.

[0120] In real life, buildings are large and small, so there are small building roofs in remote sensing images taken from satellites. Since what needs to be identified is the purpose of each building, the model needs to be able to have good recognition effect in small areas of the image. U-Net also has good effect on small object semantic tasks, so this model structure is used as the basic model.

[0121] U-Net was proposed by Olaf Ronneberger, Phillip Fischer, and Thomas Brox in 2015. The structure of U-Net is U-shaped. It consists of an encoder (Encoder, downsampling path) and a decoder (Decoder, upsampling path). The feature map in the encoder is combined with the feature map in the decoder through skip connections to retain more spatial information and improve positioning accuracy. U-Net combines the global and local features of the image through skip connections, which makes it perform well in detail recovery. At the same time, it also retains multi-level feature maps, which is very friendly to the semantic segmentation task of remote sensing data with a lot of image features. For example, please refer to the attached Figure 2 , attached Figure 2is a schematic diagram of the U-Net structure for implementing a building use recognition model provided in an embodiment of the present application, such as Figure 2 As shown, the building use recognition model includes an encoder and a decoder. The encoder receives input heterogeneous data. The output layer of the encoder is connected to the input layer of the decoder, and there is a jump connection between the encoder and the decoder. The decoder performs classification based on the features of the heterogeneous data output by the encoder output layer and the shallow features of the heterogeneous data transmitted from the jump connection, thereby outputting the building use recognition result and the building roof recognition result.

[0122] The encoder part of the building use recognition model uses ResNet (residual network). Specifically, considering that the embodiment of the present application needs to realize building use mining in addition to building roof mining, a deeper network structure is required to extract image features of multiple channels, and the features need to cover the bottom edge information; the middle texture information and the high-level use information.

[0123] Please refer to the attached Figure 3 , attached Figure 3 : This is a schematic diagram of the structure of a ResNet residual block provided in an embodiment of the present application. ResNet is a relatively good choice as an encoder. It was proposed by Kaiming He et al. in 2015 and achieved excellent results in the ImageNet Large Scale Visual Recognition Challenge (ILSVRC). ResNet solves the gradient vanishing / exploding problem and degradation problem in deep networks by introducing "residual blocks", making it possible to train very deep networks. ResNet34 is a deep residual network containing 34 layers. Its basic building block is a residual block, each of which usually contains two 3x3 convolutional layers, each of which is followed by a batch normalization layer and a ReLU activation function. The key to the residual block is the introduction of a skip connection (or shortcut connection), which bypasses one or more layers and directly adds the input to the output after several layers of transformation. This structure allows information and gradients to propagate forward and backward more easily. ResNet34 uses a unique residual block design to solve the network degradation problem, gradient vanishing and explosion problem, and parameter efficiency problem caused by the deepening of the network depth.

[0124] In the embodiment of the present application, the model identifies the building purpose of each pixel and whether each pixel is a building roof. Therefore, the fusion between the visual features of the building in the remote sensing image and the social features of the building in the social attribute data (such as POI images) can be started in the shallow network to realize the building purpose recognition and building roof recognition of each pixel.

[0125] Considering that the processed POI image and remote sensing image have been aligned in terms of geographical location, this embodiment directly splices the channels of the POI image and the channels of the remote sensing image together and inputs them into the building use recognition model (such as the UNet network), so that the building use recognition model can fully extract and fuse the features of the POI image and the remote sensing image at every extraction depth from shallow to deep layers.

[0126] Please refer to the attached Figure 4 , attached Figure 4 is a comparison chart of heterogeneous data before and after use provided by the embodiment of the present application, Figure 4 (a) is a schematic diagram of single-source input of remote sensing images. Figure 4 (b) is a schematic diagram of the heterogeneous data fusion structure. Figure 4 Here, encoder represents the encoder in the model, decoder represents the decoder in the model, and remote sensing images are images with a length and width of 224 pixels and three channels (red channel, green channel, and blue channel). POI data are images with a length and width of 224 pixels and three layers (residential layer, commercial layer, and industrial layer).

[0127] As attached Figure 4 As shown in (a), before using heterogeneous data, the remote sensing image is input to the encoder, and the encoder encodes the remote sensing image to obtain the visual features of the building in the remote sensing image. The visual features of the building obtained by the encoder are output to the decoder, and the decoder decodes the visual features of the building to obtain the building use recognition result and the building roof recognition result. Figure 4 As shown in (b), after using heterogeneous data, the remote sensing image and POI data are spliced ​​in the channel dimension to obtain heterogeneous fusion data, which contains 6 channels, of which 3 channels are the red channel, green channel and blue channel of the remote sensing image, and the other 3 channels are the residential layer, commercial layer and industrial layer of the POI data. The heterogeneous data is input to the encoder, and the encoder encodes the heterogeneous data to obtain the visual features and social features of the building in the heterogeneous data. The visual features and social features of the building obtained by the encoder are output to the decoder, and the decoder decodes the visual features and social features of the building to obtain the building use recognition results and the building roof recognition results.

[0128] The following is an example of the multi-task learning model structure provided in the embodiments of the present application.

[0129] The embodiment of the present application adopts a multi-task learning structure. During model training, the building roof recognition results and the building use recognition results are output simultaneously. The loss can be calculated based on the building roof recognition results, the building use recognition results, the building roof labels and the building use labels. The recognition results of the two tasks of building roof recognition and building use recognition are optimized through iteration based on the loss. The advantage of this is that while the model optimizes each task, it also shares information between tasks. The building use recognition task has a strong dependence on the building roof recognition task itself. In the process of joint optimization, the multi-task framework can achieve higher accuracy than the single-task framework. Appendix Figure 5 This is a structural diagram of the building use recognition model based on multi-task learning provided in an embodiment of the present application. It can be seen that the building roof recognition and building use recognition tasks share a backbone feature extraction network, and there is sufficient information sharing between the two tasks.

[0130] Attached Figure 6 : is a flowchart of a training method for a building use recognition model based on multi-task learning provided in an embodiment of the present application, the method is applied to a computer device, and the method includes:

[0131] Step S610, obtaining a sample remote sensing image of a sample area.

[0132] The remote sensing image contains visual features of buildings in the identified area. In some embodiments, sample remote sensing images provided by remote sensing data sources that meet the conditions are screened by randomly sampling remote sensing data samples. For example, remote sensing images with a target resolution, no obstructions (such as clouds), and light illumination brightness that meets the requirements are screened as sample remote sensing images to make the remote sensing images clear and usable. The target resolution can be determined based on engineering performance and image clarity. For example, the target resolution is 1 meter. The 1-meter resolution remote sensing image source generally provides processed RGB images, which are not suitable for data clarity processing. Please refer to the attached Figure 7 , attached Figure 7 This is a comparison diagram of 10-meter and 1-meter resolution remote sensing images provided in the embodiment of the present application. Figure 7 It can be seen that in the same area, Figure 7 The 1-meter resolution remote sensing image shown in (b) is compared Figure 7 The 10-meter resolution remote sensing image shown in (a) contains more information from the surrounding Figure 7 In (b), each building and its surroundings can be seen more clearly.

[0133] Regarding the selection of sample remote sensing images, in some embodiments, the sample remote sensing images of the sample area include remote sensing images of provincial-level administrative regions across the country, remote sensing images with building distribution density higher than a density threshold, remote sensing images with building distribution density lower than a density threshold, and remote sensing images with building use identification result deviations higher than a deviation threshold output by the building use identification model in the test set.

[0134] For example, when selecting sample remote sensing images, considering that there are great differences in architecture in regions with different terrains and different cultural preferences in China, in order to make the sample remote sensing images cover architecture in regions with various terrains and various cultural preferences, remote sensing images of various provincial administrative regions across the country are sampled as sample remote sensing images. In a possible implementation, the remote sensing images of each provincial administrative region are randomly cut into blocks, for example, into 2 km*2 km remote sensing image blocks, and 3 blocks are randomly selected from the remote sensing image blocks of each provincial administrative region as sample remote sensing images.

[0135] For example, in order to improve the recognition ability of the model, difficult samples are added to the sample remote sensing images. Remote sensing images with too sparse building distribution and remote sensing images with too dense building distribution are both difficult samples for the model, so remote sensing images with building distribution density higher than the density threshold and remote sensing images with building distribution density lower than the density threshold are selected as sample remote sensing images.

[0136] For example, during the i-th round of training of the model, the performance of the model on the test set can be tested. If the model performs poorly on some sample remote sensing images in the test set (such as the deviation of the building use recognition result is higher than the deviation threshold), these sample remote sensing images can be used as difficult samples. During the (i+1)-th round of training of the model, these sample remote sensing images can be used to train the model.

[0137] In some embodiments, a sample social attribute image of the sample area is also obtained. The sample social attribute image includes social features of buildings in the identified area. The size of the sample social attribute image is the same as the size of the sample remote sensing image. For example, the number of pixels in the sample social attribute image is the same as the number of pixels in the sample remote sensing image. The geographic coordinates of the sample social attribute image correspond to the geographic coordinates of the sample remote sensing image. For example, the sample social attribute image and the sample remote sensing image belong to the same area.

[0138] The sample social attribute image includes the social characteristics of the building. There are many types of sample social attribute images. In some embodiments, the sample social attribute image includes at least one of the data of points of interest (POI), the population distribution data of the identified area, the age distribution data of the identified area, the GDP data of the identified area, or the county annals of the identified area. For example, at night, the probability that the buildings in densely populated areas are used for residential purposes is relatively high, and during the day, the probability that the buildings in densely populated areas are used for commercial or industrial purposes is relatively high. In areas with a large GDP, the buildings with commercial purposes tend to be more numerous. The following article focuses on using the sample social attribute image as an example of POI data.

[0139] POI (Point Of Interest) data, usually translated into Chinese as "point of interest" or "information point", can be provided by map-related cooperative suppliers. POI data includes the location of the POI in the identified area (such as longitude and latitude information), the name information of the POI, the address information of the POI, the category of the POI, and the purpose information of the POI. From the category point of view, the categories of POI include "residential", "commercial" and "industrial" which are of concern to the insurance industry. In addition, where POIs of a certain category are concentrated, the nearby buildings are often of the same category.

[0140] In order to improve the accuracy of building use identification, POI data is introduced in the process of model learning to assist the model in determining the building use. The original POI data storage format is a table. Considering that the final modeling result is output in the form of an image, the POI data is converted into an image format to unify the input data and output data formats and retain the spatial correlation between POI and buildings. POI data in image form can be, for example, a POI point map or a POI color temperature map.

[0141] For the generation process of the POI point map, for example, based on the position of the POI in the identified area in the POI data and the category of the POI, multiple POI point maps are created, the size of each POI point map is the same as the size of the remote sensing image, and each POI point map corresponds to a category of POI. Each POI point map is, for example, a binary image. The pixel value of the pixel point of the POI of the corresponding category in each POI point map is a first pixel value, and the pixel value of the pixel point of the POI of the corresponding category that does not exist in each POI point map is a second pixel value. The first pixel value and the second pixel value are different. For example, the first pixel value is 1, and the second pixel value is 0. After obtaining multiple POI point maps, the remote sensing image and the multiple POI point maps can be spliced ​​in the channel dimension to obtain heterogeneous data, and the heterogeneous data includes m channels of the remote sensing image and each channel of each POI point map in the multiple POI point maps.

[0142] As a specific example, the multiple POI point maps created include a residential POI point map, a commercial POI point map, and an industrial POI point map. The residential POI point map corresponds to POIs of the residential category, and the pixel values ​​of the pixels of the POIs of the residential category that exist in the residential POI point map are 1, and the pixel values ​​of the pixels of the POIs of the residential category that do not exist in the residential POI point map are 0. The commercial POI point map corresponds to POIs of the commercial category, and the pixel values ​​of the pixels of the POIs of the commercial category that exist in the commercial POI point map are 1, and the pixel values ​​of the pixels of the POIs of the commercial category that do not exist in the commercial POI point map are 0. The industrial POI point map corresponds to POIs of the industrial category, and the pixel values ​​of the pixels of the POIs of the industrial category that exist in the industrial POI point map are 1, and the pixel values ​​of the pixels of the POIs of the industrial category that do not exist in the industrial POI point map are 0. The residential POI point map, the commercial POI point map, and the industrial POI point map are three layers with the same size as the remote sensing image.

[0143] The POI point map conversion effect is as follows Figure 8 As shown, attached Figure 8 (a) shows the remote sensing image, Figure 8 (b), Attachment Figure 8 (c) and Figure 8 (d) are all POI point maps with a resolution of 10 meters in the corresponding areas of remote sensing images. Figure 8 (b) shows a residential POI point map, and it can be seen that residential POI points are basically distributed in residential areas. Figure 8 (c) is a commercial POI point map. It can be seen that commercial POI points are more densely distributed along the streets and in densely populated areas. Figure 8 (d) is the industrial POI point map. It can be seen that there are relatively few industrial POIs, which are concentrated in the areas corresponding to the red or blue remote sensing roofs.

[0144] In some implementations, the POI point map is a direct reflection of the POI location, but the distribution of values ​​in the POI point map is not continuous. When POI points are sparse, most of the values ​​in the entire POI point map are 0, which increases the difficulty of information extraction for the AI ​​model: it is difficult for the model to infer the distance from the rest of the image to the neighboring POI points based on the sparse POI points, and it is difficult to use POI data for building type recognition. In view of this, the distance color temperature map method is used to convert the POI point map into a POI color temperature map.

[0145] The POI color temperature map is also called the POI distance heat map. In the process of converting the POI point map into the POI color temperature map, specifically, based on the distance between each pixel point and the nearest adjacent POI pixel point in each of the POI point maps, multiple POI color temperature maps are created; each POI color temperature map includes the POI distance information corresponding to the POI point map, that is, the distance between each pixel point representing the POI in the POI point map and the nearest adjacent pixel point representing the POI (such as Euclidean distance).

[0146] For example, based on the distance between each pixel in the residential POI point map and the nearest adjacent POI pixel, a residential POI color temperature map is created; the residential POI color temperature map includes the distance between each pixel in the residential POI point map and the nearest adjacent residential POI pixel. Based on the distance between each pixel in the commercial POI point map and the nearest adjacent POI pixel, a commercial POI color temperature map is created; the commercial POI color temperature map includes the distance between each pixel in the commercial POI point map and the nearest adjacent commercial POI pixel. Based on the distance between each pixel in the industrial POI point map and the nearest adjacent POI pixel, an industrial POI color temperature map is created; the industrial POI color temperature map includes the distance between each pixel in the industrial POI point map and the nearest adjacent industrial POI pixel.

[0147] For example, the specific calculation formula for the pixel value of each pixel in the POI color temperature map is as follows:

[0148]

[0149] DH stands for distance heat map, which indicates the coordinates of a pixel representing a POI within a specified range. The Euclidean distance formula is used to calculate the distance from the pixel to the POI. After the distances from the pixel to all surrounding POIs are calculated, the closest distance from the pixel to all POIs is selected.

[0150] In some embodiments, different maximum ranges of influence are configured for different categories of POIs, and a POI color temperature map is generated based on the distance between POI pixels in the POI point map and the maximum range of influence. The maximum range of influence is used to determine the range of POI pixels used when calculating the pixel value of a pixel in the color temperature map. For example, considering that there are more data in the accommodation area and the commercial area within the unit range, it can be converted into a relatively limited influence of a certain accommodation area or commercial area on the surrounding area, and the industrial area has less data and is located in a more remote location, which can be converted into a greater influence of the industrial area on the surrounding area. Based on this, the maximum range of influence of the POI point map of the industrial category is configured to be greater than the maximum range of influence of the POI point map of the residential category, and greater than the maximum range of influence of the POI point map of the commercial category. In the process of generating the POI color temperature map, for each pixel in each of the POI point maps, the distance between the pixel and each POI pixel in the POI point map to which the pixel belongs that belongs to the maximum range of influence is determined; the closest distance value between the pixel and all POI pixels is selected as the pixel value of the pixel in the POI color temperature map. Exemplarily, the maximum range of influence of the accommodation area or commercial area is set to 500 meters, and the maximum range of influence of the industrial layer is set to 1000 meters.

[0151] For example, please refer to the attached Fig. 9 , attached Fig. 9 This is a diagram showing the effect of implementing a 1-meter resolution POI distance color temperature map provided in an embodiment of the present application, Fig. 9 (a) shows the remote sensing image, Fig. 9 (b) shows the residential POI distance color temperature map of the corresponding area of ​​the remote sensing image. It can be seen that the residential POI points are basically distributed in the residential area; Fig. 9 (c) shows the commercial POI distance color temperature map. It can be seen that the commercial POI points are more densely distributed on the street side and in densely populated areas. Fig. 9 (d) shows the industrial POI distance color temperature map. It can be seen that there are relatively few industrial POIs, and most of them appear in areas where blue greenhouses are concentrated.

[0152] In actual modeling, using distance color temperature maps has two advantages. First, if the POI point information is not accurate, the surrounding buildings can still be affected by the POI point according to the closest distance to the POI, thereby increasing the accuracy of building use classification; second, it also solves the problem of buildings with similar uses at adjacent POI points, and the distance can be used to determine whether the nearest building is of the same type. At the same time, even after cropping, blocks with POI points can still affect surrounding blocks without POI point data, making the most of POI data for use classification.

[0153] Step S620: Determine a building use recognition result and a building roof recognition result based on the sample remote sensing image by using a building use recognition model.

[0154] The structure of the building use recognition model is described above. In some embodiments, the building use recognition model has a U-Net network structure, and the building use recognition model includes an encoder and a decoder. Optionally, the encoder is a residual network. Alternatively, the encoder is a transformer.

[0155] In some embodiments, when identifying the use of a building and the roof of a building, not only the remote sensing image but also the social attribute data is referred to, and the sample remote sensing image of the identified area and the social attribute data of the identified area are fused to obtain sample heterogeneous data; the sample heterogeneous data is input into the building use recognition model. The building use recognition model determines the building use recognition result and the building roof recognition result based on the sample heterogeneous data.

[0156] In some implementations of heterogeneous data fusion, the sample remote sensing image and the sample social attribute data are spliced ​​in the channel dimension to obtain sample heterogeneous data.

[0157] The sample remote sensing image includes m channels, the sample social attribute data includes n channels, and the sample heterogeneous data includes m channels of the sample remote sensing image and n channels of the sample social attribute data. For example, the sample heterogeneous data contains a total of (m+n) channels, wherein the first m channels are channels of the sample remote sensing image, and the last n channels are channels of the sample social attribute image. Among them, m is, for example, the number of color channels (or visible light channels) in the sample remote sensing image, and n is the number of social attribute categories in the sample social attribute image. For example, m is 3, and the m channels are the red channel, green channel, and blue channel of the sample remote sensing image. n is 3, the sample social attribute image is a POI image, and the n channels are a residential POI image, a commercial POI image, and an industrial POI image.

[0158] In some embodiments of the building use identification model processing heterogeneous data, the sample heterogeneous data is encoded by the encoder to obtain the fusion features of the sample remote sensing image and the sample social attribute data; the fusion features of the sample remote sensing image and the sample social attribute data are decoded by the decoder to obtain the building use identification result and the building roof identification result of the sample area.

[0159] The encoder includes one or more convolutional layers, and the encoder convolves (forward propagation) the heterogeneous data through the convolutional layers. During the convolution process, the visual features of the buildings in the remote sensing image and the social features of the buildings in the social attribute image are further fused. The fused features finally encoded include the visual features of the buildings in the remote sensing image and the social features of the buildings in the social attribute image.

[0160] In some embodiments, the input image is cropped into small blocks, and then the output results of the model are spliced ​​back into a large block, so as to reduce the risk of the input image exceeding the range that the model can handle and causing the solution to be unavailable, and to achieve batch recognition, thereby improving the efficiency of recognition of large-sized remote sensing images. For example, the sample remote sensing image is divided into blocks according to a preset image size to obtain a plurality of sample remote sensing image blocks; the sample social attribute image is divided into blocks according to the preset image size to obtain a plurality of sample social attribute image blocks; the sample remote sensing image blocks with the same geographical location coordinates and the sample social attribute image blocks are spliced ​​in the channel dimension to obtain a plurality of heterogeneous data blocks, each of which includes m channels of a sample remote sensing image block with the same geographical location coordinates and n channels of a sample social attribute image block. Multiple heterogeneous data blocks are input into the building use recognition model in parallel, and multiple heterogeneous data blocks are equivalent to a batch. The encoder encodes the multiple heterogeneous data blocks in parallel to obtain multiple fusion features, each of which includes a feature of the sample remote sensing image block and a feature of the sample social attribute image block; the decoder decodes the multiple fusion features in parallel to obtain multiple building use data blocks and multiple building roof data blocks, each building use data block includes a building use recognition result in a remote sensing image of a preset image size, and each building roof data block includes a building roof recognition result in a remote sensing image of a preset image size; the multiple building use data blocks are spliced ​​according to geographic location coordinates to obtain a building use recognition result of the sample area; the multiple building roof data blocks are spliced ​​according to geographic location coordinates to obtain a building roof recognition result of the sample area.

[0161] Step S630, determining the building use loss based on the building use identification result and the building use label.

[0162] The building use label is used to indicate the building use of the identified area. In some embodiments, pixel-level annotation is used, that is, the building use corresponding to each pixel is annotated in the building use label. For example, the pixel value of each pixel in the building use label is a first pixel value, a second pixel value, a third pixel value, or a fourth pixel value, the first pixel value represents the background, the second pixel value represents that the building use is residential, the third pixel value represents that the building use is commercial, and the fourth pixel value represents that the building use is industrial. For example, the pixel value of the pixel in the building use label is 0, 1, 2, or 3, 0 represents the background, 1 represents residential, 2 represents commercial, and 3 represents industrial.

[0163] Building label data can be provided by a map-related cooperative supplier. The data format of building label data is a shp surface feature file, covering the core areas of 68 core cities in China, and contains information such as the building location, roof shape, building use, and building height of the building. Among them, the building roof shape and building use are the content that the model needs to learn and output, so this building data is used as the label for model learning. At present, the model outputs a raster image format that is easy to calculate, so it is necessary to convert the building shp surface file into a raster image format, and align the converted raster position and raster size to the remote sensing raster file of the corresponding area. The label layers to be converted include the building use layer and the building roof layer. In the building use layer, use a mask to set the file to single-band data, and use a grayscale image to express the labeled data.

[0164] The building use loss is used to characterize the deviation between the building use recognition result and the building use label. In some embodiments, the building use loss is determined based on the probability that the building use of each pixel in the building use recognition result is identified as residential, the probability that the building use of each pixel is identified as commercial, the probability that the building use of each pixel is identified as industrial, and the pixel value of each pixel in the building use label. Optionally, the DiceLoss loss function is used to determine the building use loss. For example, building use loss = (DiceLoss (the probability that the model identifies the building use of a pixel as residential, whether the building use of the pixel in the label is actually residential) + DiceLoss (the probability that the model identifies the building use of a pixel as commercial, whether the building use of the pixel in the label is actually commercial) + DiceLoss (the probability that the model identifies the building use of a pixel as industrial, whether the building use of the pixel in the label is actually industrial)) / 3.

[0165] Step S640: determining the building roof loss based on the building roof recognition result and the building roof label.

[0166] The building roof tag is used to characterize the building roof of the identified area. In some embodiments, pixel-level annotation is used, that is, the building roof corresponding to each pixel is annotated in the building roof tag. The pixel value of each pixel in the building roof tag is the fifth pixel value or the sixth pixel value, the fifth pixel value represents the background, and the sixth pixel value represents the roof. For example, the pixel value of the pixel in the building roof tag is 0 or 1, 0 represents the background, and 1 represents the roof.

[0167] There is also a problem of roof offset in remote sensing images during the conversion process. The reasons for this offset are as follows: Fig.10 As shown in the figure, in areas where there is an offset angle in satellite photography, the roof of the building photographed will be offset relative to the foundation. At a resolution of 10m, one pixel represents a distance of about 10m, so the problem of satellite photography angle offset is not obvious, and batch conversion can be directly performed using the program. However, when the resolution is increased to 1 meter, the offset problem caused by the satellite tilt shooting angle cannot be avoided. Moreover, the direction and angle of the offset of each image data are different (there are differences in shooting offset angles and building heights), so it is impossible to use a regular method to correct them in batches, so manual annotation is used for correction.

[0168] In the manual annotation process, GIS software is used to align the building vector with the remote sensing roof. First, the annotation method is clarified, and the original data information is retained as much as possible. When annotating, the moving layer is mainly used, and the original elements are rarely deleted. Height recognition is one of the project tasks, so in areas with obvious hierarchical distinction, the roofs of higher houses are first determined, and other areas are moved or reshaped to retain the height information of the original auxiliary annotation layer to the greatest extent.

[0169] Please refer to the attached Fig.11 , attached Fig.11 : This is a schematic diagram of the roof comparison between the shp layer and the remote sensing image layer before and after manual annotation provided by the embodiment of the present application. The annotation results are shown in the attached figure. Fig.11 Attached Fig.11 (a) is a sample before labeling, and the building vector is seriously offset from the remote sensing roof. Fig.11 (b) is the labeled sample, where the building vector and remote sensing roof are well aligned.

[0170] The building roof loss is used to characterize the deviation between the building roof recognition result and the building roof label. The building roof loss is determined based on the probability that the building use of each pixel in the building use recognition result is identified as a roof and the pixel value of each pixel in the building roof label. Optionally, the DiceLoss loss function is used to determine the building roof loss. For example, building roof Loss = DiceLoss (the probability that the model identifies a pixel as a roof, whether the building use of the pixel in the label is actually a roof).

[0171] Step S650: Determine the comprehensive loss based on the building use loss and the building roof loss. The comprehensive loss is used to characterize the overall deviation of the building use recognition result and the building roof recognition result.

[0172] In some embodiments of determining the combined loss, the building use loss and the building roof loss are summed to obtain the combined loss.

[0173] In other embodiments of determining the comprehensive loss, based on the weight corresponding to the building use loss and the weight corresponding to the building roof loss, the building use loss and the building roof loss are weighted summed to obtain the comprehensive loss.

[0174] In some embodiments, the weight corresponding to the building use loss and the weight corresponding to the building roof loss in the loss function are dynamically adjusted, thereby optimizing the model's performance in performing the building use identification task and the building roof identification task. For example, in the i-th round of training, if the deviation of the building use identification result output by the building use identification model is less than the deviation threshold and the deviation of the building roof identification result is greater than the deviation threshold, this indicates that the model performs well in the building use identification task but poorly in the building roof identification task. Then, by reducing the weight corresponding to the building use loss or increasing the weight corresponding to the building roof loss, the model is made to focus more on identifying the building roof in the (i+1) round of training, reducing the deviation of the building roof identification result output by the model in the (i+1) round of training, and improving the convergence speed of the building roof identification task.

[0175] Similarly, in the i-th round of training, if the deviation of the building use identification result output by the building use identification model is greater than the deviation threshold and the deviation of the building roof identification result is less than the deviation threshold, this indicates that the model performs well on the building roof identification task but poorly on the building use identification task. In this way, by reducing the weight corresponding to the building roof loss or increasing the weight corresponding to the building use loss, the model will focus more on identifying the building use in the (i+1) round of training, thereby reducing the deviation of the building use identification result output by the model in the (i+1) round of training and improving the convergence speed of the building use identification task.

[0176] Step S660, adjusting the parameters of the building use identification model based on the comprehensive loss so that the deviation of the building use identification result output by the building use identification model and the deviation of the building roof identification result both meet the conditions.

[0177] For example, the parameters of the building use recognition model are adjusted by back propagation based on the comprehensive loss, so that the deviation of the building use recognition results output by the building use recognition model and the deviation of the building roof recognition results continue to decrease, until the deviation of the building use recognition results output by the building use recognition model is less than the deviation threshold, and the deviation of the building roof recognition results is also less than the deviation threshold, and the training is terminated.

[0178] After the training of the building use recognition model is completed, the building use recognition model can be used to perform building use recognition tasks and building roof recognition tasks for remote sensing images of unknown use. For example, a remote sensing image of the identified area and a social attribute image of the identified area are obtained; the remote sensing image and the social attribute image are spliced ​​in the channel dimension to obtain heterogeneous data, the remote sensing image includes m channels, the social attribute image includes n channels, and the heterogeneous data includes m channels of the remote sensing image and n channels of the social attribute image; the heterogeneous data is input into the building use recognition model, and the building use recognition model includes an encoder and a decoder; the heterogeneous data is encoded by the encoder to obtain fusion features of the remote sensing image and the social attribute image, and the fusion features include visual features of the building in the remote sensing image and social features of the building in the social attribute image; the fusion features are decoded by the decoder to obtain the building use recognition result of the identified area and the building roof recognition result of the identified area.

[0179] In some embodiments, the social attribute image includes point of interest (POI) data, and the POI data includes the location of the POI in the identified area and the category of the POI; multiple POI point maps are created based on the POI data, the size of each POI point map is the same as the size of the remote sensing image, each POI point map corresponds to a category of POI, the pixel value of each pixel point in the POI point map that has a POI of the corresponding category is a first pixel value, and the pixel value of each pixel point in the POI point map that does not have a POI of the corresponding category is a second pixel value; based on the distance between each pixel point in each POI point map and the nearest adjacent POI pixel point, multiple POI color temperature maps are created; accordingly, the remote sensing image and the social attribute image are spliced ​​in the channel dimension, including: splicing the remote sensing image with the multiple POI color temperature maps in the channel dimension.

[0180] In some embodiments, the method of creating multiple POI color temperature maps based on the distance between each pixel point and the nearest adjacent POI pixel point in each of the POI point maps includes: for each pixel point in each of the POI point maps, determining the distance between the pixel point and each POI pixel point with the maximum range of influence in the POI point map to which the pixel point belongs, wherein in each of the POI point maps, the maximum range of influence of the POI point map of the industrial category is greater than the maximum range of influence of the POI point map of the residential category, and greater than the maximum range of influence of the POI point map of the commercial category; selecting the closest distance value between the pixel point and all POI pixel points as the pixel value of the pixel point in the POI color temperature map.

[0181] In some embodiments, the remote sensing image and the social attribute image are spliced ​​in the channel dimension, including: dividing the remote sensing image according to a preset image size to obtain a plurality of remote sensing image blocks; dividing the social attribute image according to the preset image size to obtain a plurality of social attribute image blocks; splicing the remote sensing image blocks and the social attribute image blocks with the same geographical location coordinates in the channel dimension; accordingly, the heterogeneous data includes a plurality of heterogeneous data blocks, each of which includes m channels of a remote sensing image block with the same geographical location coordinates and n channels of a social attribute image block; accordingly, the heterogeneous data is encoded by the encoder , to obtain the fusion features of the remote sensing image and the social attribute image, including: encoding the multiple heterogeneous data blocks in parallel through the encoder to obtain multiple fusion features, each of which includes a feature of the remote sensing image block and a feature of the social attribute image block; decoding the fusion features through the decoder to obtain the building use recognition result of the identified area and the building roof recognition result of the identified area, including: decoding the multiple fusion features in parallel through the decoder to obtain multiple use data blocks and multiple roof data blocks; splicing the multiple use data blocks according to the geographical location coordinates to obtain the building use recognition result of the identified area. Splicing the roof data blocks according to the geographical location coordinates to obtain the building roof recognition result of the identified area.

[0182] In some embodiments, the building use recognition model is based on the sample remote sensing image of the identified area to determine the building use recognition result of each pixel in the sample remote sensing image and the building roof recognition result of each pixel. For example, the building use recognition result is in the form of a mask image, and the pixel value of each pixel in the building use recognition result is used to characterize the building use of the corresponding pixel. For example, if the pixel value of the pixel with coordinates (i, j) in the building use recognition result is 1, it means that the building use of the pixel with coordinates (i, j) in the remote sensing image is residential, if the pixel value of the pixel with coordinates (i, j) in the building use recognition result is 2, it means that the building use of the pixel with coordinates (i, j) in the remote sensing image is commercial, if the pixel value of the pixel with coordinates (i, j) in the building use recognition result is 3, it means that the building use of the pixel with coordinates (i, j) in the remote sensing image is industrial. For another example, the building roof recognition result is in the form of a mask image, and the pixel value of each pixel in the building roof recognition result is used to characterize the building roof of the corresponding pixel. For example, if the pixel value of the pixel point with coordinates (i, j) in the building roof recognition result is 1, it means that the pixel point with coordinates (i, j) in the remote sensing image is the building roof; if the pixel value of the pixel point with coordinates (i, j) in the building roof recognition result is 0, it means that the pixel point with coordinates (i, j) in the remote sensing image is not the building roof.

[0183] In some embodiments, the area of ​​the sample area identified once is between 500 meters * 500 meters and 1 kilometer * 1 kilometer. Specifically, through multiple experiments, it is found that when a single-task architecture is adopted, that is, the model only outputs the building use recognition result without outputting the building roof recognition result, when the size of the input image changes, the building use recognition effect of the model is fluctuating and flat. After the introduction of the multi-task architecture, it is found that the model of the multi-task architecture has a better building use recognition effect on input images of specific sizes. When the area of ​​the sample area identified once is 110 meters * 110 meters, the sample remote sensing image of the sample area contains very few buildings, and the sample remote sensing image may only contain one building. In this way, when the model identifies the building use, it can only refer to the characteristics of one building or a small number of buildings, so it is not easy to identify the building use, resulting in the multi-task architecture model Compared with the single-task architecture model, the building use recognition effect of the sample remote sensing image of the sample area is not significantly improved. When the area of ​​the sample area to be identified at a single time is between 500 meters * 500 meters and 1 kilometer * 1 kilometer, the sample remote sensing image of the sample area will usually contain multiple buildings, and different buildings usually have spatial correlations. In this way, when identifying the use of a building, the model can refer to the characteristics of multiple spatially adjacent buildings. For example, it can identify the use of a building by referring to multiple adjacent buildings in a community, thereby further improving the recognition results of the building use.

[0184] In order to test the help of heterogeneous data fusion and multi-task learning in building use recognition, an ablation experiment was conducted based on the remote sensing image of the identified area of ​​500 meters * 500 meters. The specific experimental results are shown in the following table. In Experiment 1, the input data of the model is the remote sensing image, and the output of the model is the building use recognition result; on the basis of Experiment 1, Experiment 2 fuses the POI image and the remote sensing image into heterogeneous data as the input of the model, and the output of the model is the building use recognition result. On the basis of Experiment 2, the output of Experiment 2 adds the building roof recognition result, and the model is a multi-task framework for building roof recognition and building use recognition. Among them, the F1 index is a binary classification evaluation index that can measure both accuracy and recall.

[0185] Building use identification ablation experiment table

[0186]

[0187] By comparing Experiment 1 and Experiment 2, we can find that the classification effect of each building use is improved after the introduction of POI data, which shows that POI data is beneficial to the task of building use classification and the effectiveness of the currently designed heterogeneous data fusion structure. By comparing Experiment 2 and Experiment 3, we can see that after using the multi-task framework that adds building roof recognition, the indicators of building use have been further improved.

[0188] At the same time, the comparison is as follows Fig.12 , attached Fig.12 : is a comparison chart of the multi-task recognition effect and the single-task recognition effect provided by the embodiment of the present application. Fig.12 It can be seen that the recognition of a single building as a whole in the multi-task framework will be more accurate, thereby improving the accuracy of category judgment. This also shows that the building roof recognition task can be used as an auxiliary task for building use identification, proving the effectiveness of the multi-task framework.

[0189] The effect of interpreting building use is shown in the attached Fig.13 As shown in the figure. The roof is drawn using known building vector data, and the color is filled based on the roof and model inference results. Red, green, and blue represent residential, commercial, and industrial roofs, respectively, and black represents the background. It can be seen that the model has a good recognition effect on the red residential roof; the green and blue commercial and industrial roofs are confused in some areas, but can be accurately recognized overall.

[0190] The national building use identification model framework provided in this embodiment utilizes nationwide remote sensing data and nationwide social attribute data, adopts a partial area training and nationwide reasoning approach, and takes into account both performance and accuracy.

[0191] In the visual feature and social feature fusion model provided in this embodiment, by directly performing channel stitching on the input remote sensing image and POI data, the visual features in the remote sensing image and the social features in the POI data can be effectively fused, thereby improving the accuracy of pixel-level granularity building use recognition.

[0192] This embodiment uses social attribute data in the process of executing the national building use identification task, thereby improving the accuracy of building use identification while ensuring that the identification scope remains unchanged.

[0193] This embodiment adds an auxiliary task of building roof recognition to the building use recognition task to form a multi-task structure to improve the accuracy of the building use recognition task model.

[0194] Fig.14 : is a structural schematic diagram of a training device 600 for a building use recognition model based on multi-task learning provided in an embodiment of the present application, wherein the device 600 comprises:

[0195] An identification unit 610 is used to determine a building use identification result and a building roof identification result based on a sample remote sensing image of a sample area by using a building use identification model;

[0196] The determination unit 620 is used to determine the building use loss based on the building use identification result and the building use label, the building use label is used to indicate the building use of the sample area, and the building use loss is used to characterize the deviation between the building use identification result and the building use label; determine the building roof loss based on the building roof identification result and the building roof label, the building roof label is used to characterize the building roof of the sample area, and the building roof loss is used to characterize the deviation between the building roof identification result and the building roof label; determine the comprehensive loss based on the building use loss and the building roof loss, and the comprehensive loss is used to characterize the deviation of the building use identification result and the building roof identification result as a whole;

[0197] The adjustment unit 630 is used to adjust the parameters of the building use identification model based on the comprehensive loss so that the deviation of the building use identification result and the deviation of the building roof identification result output by the building use identification model both meet conditions.

[0198] In some embodiments, the determination unit 620 is used to sum the loss of the building use and the loss of the building roof to obtain a comprehensive loss; or, based on the weight corresponding to the loss of the building use and the weight corresponding to the loss of the building roof, perform weighted summation of the loss of the building use and the loss of the building roof to obtain a comprehensive loss.

[0199] In some embodiments, the adjustment unit 630 is also used to reduce the weight corresponding to the building use loss or increase the weight corresponding to the building roof loss if the deviation of the building use identification result output by the building use identification model is less than the deviation threshold and the deviation of the building roof identification result is greater than the deviation threshold; if the deviation of the building use identification result output by the building use identification model is greater than the deviation threshold and the deviation of the building roof identification result is less than the deviation threshold, reduce the weight corresponding to the building roof loss or increase the weight corresponding to the building use loss.

[0200] In some embodiments, the determination unit 620 is used to determine the building use recognition result of each pixel in the sample remote sensing image and the building roof recognition result of each pixel based on the sample remote sensing image of the sample area through the building use recognition model; accordingly, the pixel value of each pixel in the building use label is a first pixel value, a second pixel value, a third pixel value or a fourth pixel value, the first pixel value represents the background, the second pixel value represents that the building use is residential, the third pixel value represents that the building use is commercial, the fourth pixel value represents that the building use is industrial, and the pixel value of each pixel in the building roof label is a fifth pixel value or a sixth pixel value, the fifth pixel value represents the background, and the sixth pixel value represents the roof; accordingly, based on the probability that the building use of each pixel in the building use recognition result is recognized as residential, the probability that the building use of each pixel is recognized as commercial, the probability that the building use of each pixel is recognized as industrial, and the pixel value of each pixel in the building use label, the building use loss is determined; based on the probability that the building use of each pixel in the building use recognition result is recognized as roof and the pixel value of each pixel in the building roof label, the building roof loss is determined.

[0201] In some embodiments, the device further comprises: a fusion unit, configured to fuse the sample remote sensing image of the sample area and the social attribute data of the sample area to obtain sample heterogeneous data;

[0202] The identification unit 610 is used to determine a building use identification result and a building roof identification result based on the sample heterogeneous data through a building use identification model.

[0203] In some embodiments, the fusion unit is used to splice the sample remote sensing image and the sample social attribute data in the channel dimension to obtain sample heterogeneous data, the sample remote sensing image includes m channels, the sample social attribute data includes n channels, and the sample heterogeneous data includes the m channels of the sample remote sensing image and the n channels of the sample social attribute data;

[0204] The building use identification model includes an encoder and a decoder. The identification unit 610 is used to encode the sample heterogeneous data through the encoder to obtain the fusion features of the sample remote sensing image and the sample social attribute data; and decode the fusion features of the sample remote sensing image and the sample social attribute data through the decoder to obtain the building use identification result and the building roof identification result of the sample area.

[0205] In some embodiments, the area of ​​the sample region identified at a single time is between 500 meters*500 meters and 1 kilometer*1 kilometer.

[0206] Fig.15 600 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device 600 includes: a processor 601, the processor 601 is coupled to a memory 602, the memory 602 stores at least one computer program instruction, and the at least one computer program instruction is loaded and executed by the processor 601, so that the computer device 600 implements Figure 6 The method provided by the embodiment.

[0207] In some embodiments, a computer-readable storage medium is further provided, wherein the storage medium stores at least one instruction, and when the instruction is executed on a computer, the computer executes the above-mentioned Figure 5 The method provided by the embodiment.

[0208] In some embodiments, a computer program product is also provided. The computer program product includes one or more computer program instructions. When the computer program instructions are loaded and executed by a computer, the computer executes the above Figure 5 The method provided by the embodiment.

[0209] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0210] A refers to B, which means that A is the same as B or A is a simple variant of B.

[0211] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order of objects, and cannot be understood as indicating or implying relative importance. For example, a first pixel value and a second pixel value are used to distinguish different pixel values, rather than to describe a specific order of pixel values, and cannot be understood as meaning that the first pixel value is more important than the second pixel value.

[0212] In the embodiments of the present application, unless otherwise specified, "at least one" means one or more, and "a plurality" means two or more. For example, a plurality of pixel values ​​means two or more pixel values.

[0213] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk SolidState Disk (SSD)), etc.

[0214] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for a building use recognition model based on multi-task learning, characterized in that: The method comprises: Determine building use recognition results and building roof recognition results based on sample remote sensing images of sample areas through a building use recognition model; Determine a building use loss based on the building use identification result and the building use label, wherein the building use label is used to indicate the building use of the sample area, and the building use loss is used to characterize the deviation between the building use identification result and the building use label; Determine a building roof loss based on the building roof recognition result and the building roof label, wherein the building roof label is used to characterize the building roof of the sample area, and the building roof loss is used to characterize the deviation between the building roof recognition result and the building roof label; Determine a comprehensive loss based on the building use loss and the building roof loss, wherein the comprehensive loss is used to characterize the overall deviation of the building use identification result and the building roof identification result; The parameters of the building use identification model are adjusted based on the comprehensive loss so that the deviation of the building use identification result and the deviation of the building roof identification result output by the building use identification model both meet conditions.

2. The method according to claim 1, characterized in that The comprehensive loss determined based on the loss of use of the building and the loss of the roof of the building includes: summing the loss of the building's use and the loss of the building's roof to obtain a combined loss; or, Based on the weight corresponding to the loss of the building's use and the weight corresponding to the loss of the building's roof, a weighted sum is performed on the loss of the building's use and the loss of the building's roof to obtain a comprehensive loss.

3. The method according to claim 2, characterized in that The method further comprises: If the deviation of the building use identification result output by the building use identification model is less than the deviation threshold and the deviation of the building roof identification result is greater than the deviation threshold, reduce the weight corresponding to the building use loss or increase the weight corresponding to the building roof loss; If the deviation of the building use identification result output by the building use identification model is greater than the deviation threshold and the deviation of the building roof identification result is less than the deviation threshold, reduce the weight corresponding to the building roof loss or increase the weight corresponding to the building use loss.

4. The method according to claim 1, characterized in that: The method of determining the building use recognition result and the building roof recognition result based on the sample remote sensing image of the sample area by using the building use recognition model includes: Based on the sample remote sensing image of the sample area, the building use recognition result of each pixel point in the sample remote sensing image and the building roof recognition result of each pixel point are determined by the building use recognition model; accordingly, The pixel value of each pixel point in the building use label is a first pixel value, a second pixel value, a third pixel value or a fourth pixel value, the first pixel value represents the background, the second pixel value represents that the building use is residential, the third pixel value represents that the building use is commercial, and the fourth pixel value represents that the building use is industrial, and the pixel value of each pixel point in the building roof label is a fifth pixel value or a sixth pixel value, the fifth pixel value represents the background, and the sixth pixel value represents the roof; accordingly, The determining of the building use loss based on the building use identification result and the building use label includes: Determine the building use loss based on the probability that the building use of each pixel point in the building use identification result is identified as residential, the probability that the building use of each pixel point is identified as commercial, the probability that the building use of each pixel point is identified as industrial, and the pixel value of each pixel point in the building use label; The determining of the building roof loss based on the building roof recognition result and the building roof label includes: The building roof loss is determined based on the probability that the building use of each pixel point in the building use identification result is identified as a roof and the pixel value of each pixel point in the building roof label.

5. The method according to claim 1, characterized in that: The method of determining the building use recognition result and the building roof recognition result based on the sample remote sensing image of the sample area by using the building use recognition model includes: fusing the sample remote sensing image of the sample area and the social attribute data of the sample area to obtain sample heterogeneous data; A building use recognition result and a building roof recognition result are determined based on the sample heterogeneous data through a building use recognition model.

6. The method according to claim 5, characterized in that The fusing of the sample remote sensing image of the sample area and the social attribute data of the sample area to obtain sample heterogeneous data includes: The sample remote sensing image and the sample social attribute data are spliced ​​in the channel dimension to obtain sample heterogeneous data, wherein the sample remote sensing image includes m channels, the sample social attribute data includes n channels, and the sample heterogeneous data includes the m channels of the sample remote sensing image and the n channels of the sample social attribute data; The building use recognition model includes an encoder and a decoder, and determining a building use recognition result and a building roof recognition result based on the sample heterogeneous data by using the building use recognition model includes: Encoding the sample heterogeneous data by the encoder to obtain fusion features of the sample remote sensing image and the sample social attribute data; The decoder is used to decode the fusion features of the sample remote sensing image and the sample social attribute data to obtain the building use recognition result and the building roof recognition result of the sample area.

7. The method according to claim 1, characterized in that The area of ​​the sample area identified once is between 500 meters*500 meters and 1 kilometer*1 kilometer.

8. A training device for a building use recognition model based on multi-task learning, characterized in that: The device comprises: An identification unit, used to determine a building use identification result and a building roof identification result based on a sample remote sensing image of a sample area through a building use identification model; A determination unit is configured to determine a building use loss based on the building use identification result and the building use label, wherein the building use label is used to indicate the building use of the sample area, and the building use loss is used to characterize the deviation between the building use identification result and the building use label; determine a building roof loss based on the building roof identification result and the building roof label, wherein the building roof label is used to characterize the building roof of the sample area, and the building roof loss is used to characterize the deviation between the building roof identification result and the building roof label; determine a comprehensive loss based on the building use loss and the building roof loss, wherein the comprehensive loss is used to characterize the deviation of the building use identification result and the building roof identification result as a whole; An adjustment unit is used to adjust the parameters of the building use identification model based on the comprehensive loss so that the deviation of the building use identification result and the deviation of the building roof identification result output by the building use identification model both meet conditions.

9. A computer device, characterized in that: The computer device comprises: a processor, the processor is coupled to a memory, the memory stores at least one computer program instruction, and the at least one computer program instruction is loaded and executed by the processor, so that the computer device implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, and when the instruction is executed on a computer, the computer executes the method according to any one of claims 1 to 7.

Images