Roof detection and classification method, apparatus, device, and medium

Abstract

The application provides a roof detection and classification method, device, equipment and medium, relates to the technical field of remote sensing image detection, and the method comprises the following steps: acquiring remote sensing image data and synthetic aperture radar data of a scene to be detected; performing pixel standardization processing and multi-modal data alignment processing on the remote sensing image data and the synthetic aperture radar data; inputting the processed remote sensing image data and the synthetic aperture radar data into a pre-trained target roof detection and classification model, and outputting a building roof category, a roof bounding box and a roof segmentation contour line; wherein the target roof detection and classification model is a double-main-network structure detector pre-trained in a multi-modal self-supervised manner by using remote sensing image data and synthetic aperture radar data. The application improves the detection and classification effect of weak features of a building roof.

Description

Technical Field

[0001] This application relates to the field of remote sensing image detection technology, and in particular to a method, apparatus, equipment and medium for roof detection and classification. Background Technology

[0002] Roof contour extraction and classification based on satellite remote sensing images have significant scientific and industrial implications in the field of computer vision. However, extracting roof contours and performing fine-grained classification on relatively blurry image features remains a considerable challenge. Currently, for pixel-level roof contour extraction and classification, two approaches are employed: firstly, traditional image recognition methods such as line detection are used to identify building roofs, combined with machine learning methods such as support vector machines and random forests for classification; secondly, deep learning methods are used to process remote sensing images to obtain target areas and classification results for building roofs.

[0003] However, current methods cannot extract detailed roof outlines at the pixel level, and they cannot achieve fine-grained roof classification when faced with roof types that do not have obvious features. Summary of the Invention

[0004] The purpose of this application is to provide a method, apparatus, equipment and medium for roof detection and classification, which improves the detection and classification effect of building roofs with weak features.

[0005] In a first aspect, the present invention provides a roof detection and classification method, the method comprising: acquiring remote sensing image data and synthetic aperture radar data of the scene to be detected; performing pixel normalization processing and multimodal data alignment processing on the remote sensing image data and synthetic aperture radar data; inputting the processed remote sensing image data and synthetic aperture radar data into a pre-trained target roof detection and classification model, and outputting the building roof category, the roof bounding rectangle, and the roof segmentation outline; wherein, the target roof detection and classification model is a dual-backbone network structure detector that is pre-trained using remote sensing image data and synthetic aperture radar data through multimodal self-supervised training.

[0006] In an optional implementation, the training steps of the target roof detection classification model include: performing self-supervised training on the modal data input to the initial roof detection classification model to obtain initial weights; wherein, the modal data includes a remote sensing image sample set and a synthetic aperture radar sample set; performing data augmentation processing on the modal data, and supervising training the initial roof detection classification model with the augmented samples in a first preset proportion of training rounds, and supervising training with the modal data in a second preset proportion of training rounds; after each round of training, the detector model determines the corresponding target roof detection classification model based on the target weights corresponding to the optimal mAP50 index.

[0007] In an optional implementation, self-supervised training is performed on the modal data input to the initial roof detection classification model, including: performing self-supervised training on remote sensing image data and synthetic aperture radar data respectively, and determining the weights obtained after completing the self-supervised training as the initial weights of the dual-backbone structure in the initial roof detection classification model; wherein, the self-supervised training involves randomly masking remote sensing image data and synthetic aperture radar data input to the initial roof detection classification model, and recovering the masked parts through the model during the training process, and combining sparse convolution for modeling.

[0008] In an optional implementation, data augmentation processing is performed on the modal data, including: randomly extracting pixel-level instance targets from remote sensing image sample sets and synthetic aperture radar sample sets; performing one or more preset scaling, flipping, and rotation operations on the instance targets; and pasting the processed instance targets onto selected data samples to generate augmented data samples.

[0009] In an optional implementation, the target roof detection classification model includes two sub-backbone networks, wherein the two sub-backbone networks are densely connected; when a single modal input is received, the network structure of the two sub-backbone networks is used to enhance the feature extraction capability of the backbone network; when a multimodal input is received, the network structure of the two sub-backbone networks is used to perform intermodal fusion.

[0010] In an optional implementation, the target roof detection and classification model is based on the instance segmentation framework Cascade Mask R-CNN. The basic network components include a backbone network, an FPN neck, a region generation network (RPN), a segmentation head (FCN head), and a classification head (Cascade R-CNN head). The overall model extracts image features through the backbone. The neck and head parts perform feature classification and regression through the FPN neck of Cascade Mask R-CNN, one segmentation head (FCN head), and three detection heads. Specifically, the classification and regression losses of the RPN are supervised by cross-entropy loss (CE loss) and smooth L1 loss, respectively, for the ROI regions extracted by the RPN. The classification of pixels in the segmentation head uses CE loss. The classification loss is supervised by seesaw loss, and the regression loss of the bounding box is supervised by GIOU loss.

[0011] In an optional implementation, the processed remote sensing image data and synthetic aperture radar (SAR) data are input into a pre-trained target roof detection and classification model, which outputs the building roof category, roof bounding box, and roof segmentation outline. This includes: simultaneously inputting the processed remote sensing image data and SAR data into the backbone network of the pre-trained target roof detection and classification model, and generating pyramid feature maps corresponding to the remote sensing image data and SAR data respectively through two sub-backbone networks; obtaining the region of interest (ROI) on the pyramid feature map through a region generation network (RPN); extracting features from the ROI and performing ROI alignment to obtain the target region features; and inputting the target region features into the segmentation head, classification head, and regression head respectively, outputting the building roof category, roof bounding box, and roof segmentation outline.

[0012] Secondly, the present invention provides a roof detection and classification device, comprising: a data acquisition module for acquiring remote sensing image data and synthetic aperture radar (SAR) data of the scene to be detected; a data preprocessing module for performing pixel normalization and multimodal data alignment processing on the remote sensing image data and SAR data; and a roof detection and classification module for inputting the processed remote sensing image data and SAR data into a pre-trained target roof detection and classification model, and outputting the building roof category, roof bounding rectangle, and roof segmentation outline; wherein the target roof detection and classification model is a dual-backbone network structure detector that has undergone multimodal self-supervised pre-training using remote sensing image data and SAR data.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions that can be executed by the processor, the processor executing the computer-executable instructions to implement the roof detection and classification method of any of the foregoing embodiments.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the roof detection and classification method of any of the foregoing embodiments.

[0015] The roof detection and classification method, apparatus, device, and medium provided in this application improve the understanding of building roofs with subtle features by combining a self-supervised pre-training strategy to model the backbone network weights on the target dataset. The detector with a dual-backbone network structure significantly enhances its classification and segmentation capabilities. The dual-backbone network structure is used as input to dual-modal data (RGB+SAR), and transfer learning is combined to initialize the self-supervised pre-trained backbone network weight model, thereby enhancing the model's ability to detect, segment, and classify building roofs with subtle features. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a roof detection and classification method provided in this application embodiment;

[0018] Figure 2 A structural diagram of a target roof detection and classification model provided in an embodiment of this application;

[0019] Figure 3 A schematic diagram illustrating a dense connection method of two sub-backbone networks provided in an embodiment of this application;

[0020] Figure 4 An example of a synthesized roof RGB image based on modified copy-paste composite data provided in this application embodiment;

[0021] Figure 5 This application provides a specific implementation method for roof detection and classification.

[0022] Figure 6 A schematic diagram illustrating a roof detection classification result provided in an embodiment of this application;

[0023] Figure 7 A structural diagram of a roof detection and classification device provided in an embodiment of this application;

[0024] Figure 8 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0027] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0028] This application provides a method for roof detection and classification. See [link to relevant documentation]. Figure 1 As shown, the method mainly includes the following steps:

[0029] Step S110: Acquire remote sensing image data and synthetic aperture radar data of the scene to be detected;

[0030] Step S120: Perform pixel normalization and multimodal data alignment processing on remote sensing image data and synthetic aperture radar data;

[0031] Step S130: Input the processed remote sensing image data and synthetic aperture radar data into the pre-trained target roof detection and classification model, and output the building roof category, roof bounding rectangle and roof segmentation outline; wherein, the target roof detection and classification model is a dual backbone network structure detector that is pre-trained by multimodal self-supervised training using remote sensing image data and synthetic aperture radar data.

[0032] For ease of understanding, the roof detection and classification method provided in the embodiments of this application will be described in detail below.

[0033] When training a model or detecting and classifying roofs, it is necessary to perform pixel normalization on both the remote sensing image data (remote sensing RGB data) and the synthetic aperture radar (SAR) data, and ensure that the two modal data (i.e., remote sensing image data and SAR data) are aligned, in order to improve the model training effect or improve the accuracy of roof detection and classification.

[0034] The target roof detection classification model used in this application includes two sub-backbone networks. In order to handle RGB+SAR multimodal inputs more flexibly, the two sub-backbone networks are densely connected.

[0035] Based on the instance segmentation framework Cascade Mask R-CNN, the network's basic components include a backbone network, an FPN neck, a region generation network (RPN), a segmentation head (FCN head), and a classification head (Cascade R-CNN head). The overall model extracts image features through the backbone. See also Figure 2 The network structure diagram shown depicts a ConvNeXtV2-base sub-backbone network. These networks enhance the fusion of high-dimensional and low-dimensional information, as well as the fusion of feature information between the two sub-backbone networks, through dense connections. Figure 3 The diagram illustrates a densely connected configuration of two sub-backbone networks. When a single modal input is received, this structure serves as a feature extraction component, enhancing the feature extraction capabilities of the backbone network. When a multimodal input is received, this structure facilitates intermodal feature fusion.

[0036] To enhance the sub-backbone network's modeling capability for remote sensing data, this application employs the ConvNeXt V2-base self-supervised training method FCMAE (Fully Convolutional Masked Autoencoders). This self-supervised training scheme involves randomly masking some regions on the input image, then allowing the model to attempt to recover the masked parts. Sparse convolution is introduced to model the image, mitigating the incompatibility of convolution operations in self-supervised training. The weights in this step are initialized using pre-trained weights from the open-source ConvNeXt V2-base dataset on ImageNet22k. Self-supervised training is performed on both RGB and SAR datasets, and the resulting weights are used as the initial weights for the dual-backbone structure in the detector training.

[0037] After explaining the model structure, the training of the target roof detection and classification model will be described in detail below. In one embodiment, the training steps of the target roof detection and classification model may include steps 1 to 3:

[0038] Step 1: Perform self-supervised training on the modal data input to the initial roof detection classification model to obtain the initial weights; wherein, the modal data includes remote sensing image sample set and synthetic aperture radar sample set.

[0039] In practice, self-supervised training can be performed on remote sensing image data and synthetic aperture radar data respectively. The weights obtained after completing the self-supervised training are determined as the initial weights of the dual-backbone structure in the initial roof detection and classification model. The self-supervised training involves randomly masking the remote sensing image data and synthetic aperture radar data input to the initial roof detection and classification model, and recovering the masked parts through the model during the training process, combined with sparse convolution for modeling.

[0040] Step 2: Perform data augmentation on the modal data, and conduct supervised training on the initial roof detection classification model using the augmented samples in the first preset proportion of training rounds, and conduct supervised training using the modal data in the second preset proportion of training rounds.

[0041] In practical implementation, data augmentation processing of modal data may include the following steps 2.1 and 2.2:

[0042] Step 2.1: Randomly extract pixel-level instance targets from the remote sensing image sample set and the synthetic aperture radar sample set;

[0043] Step 2.2: Perform one or more preset scaling, flipping, and rotation operations on the instance target, and paste the processed instance target into the selected data sample to generate the augmented data sample.

[0044] Optionally, to avoid data distribution shift caused by data augmentation, the first preset ratio can be 90%, and the second preset ratio can be 10%. That is, steps 2.1 and 2.2 are repeated for the first 90% of training rounds, i.e., data augmentation is performed using modified copy paste; modified copy paste is discontinued for the last 10% of training rounds. In one example, Figure 4 An example of a synthesized roof RGB image is shown, based on modified copy-paste composite data.

[0045] Furthermore, during model training, the model's neck and head parts are classified and regressed using the FPN neck of cascade mask R-CNN, a segmentation head FCN head, and three detection heads. Specifically, the classification and regression losses of the RPN are supervised by cross-entropy loss (CE loss) and smooth L1 loss, respectively, for the ROI regions extracted by the RPN. The classification of pixels in the segmentation head uses CE loss. For the classification loss, seesaw loss is used for supervision. For the regression loss of the bounding box, GIOU loss is used for supervision.

[0046] Step 3: After each round of training, the detector model determines the corresponding target roof detection classification model based on the target weights corresponding to the optimal mAP50 index.

[0047] In one implementation, after each training round of the detector model, which has undergone data augmentation and supervised loss processing, the model weights with the best mAP50 index selected after evaluation by the validation set can be used as the weights for subsequent detector inference. Optionally, the target weights can be determined by averaging or variance processing of a predetermined number of model weights with good mAP50 indexes. In practical applications, the two methods of determining the target weights can be compared, and the target weight with the smaller error can be selected to determine the target roof detection classification model.

[0048] After the target roof detection and classification model is trained as described above, the processed remote sensing image data and synthetic aperture radar data are input into the pre-trained target roof detection and classification model to output the building roof category, roof bounding rectangle, and roof segmentation outline. In specific implementation, this may include the following steps A to D:

[0049] Step A: The backbone network of the target roof detection classification model is used, and pyramid feature maps corresponding to remote sensing image data and synthetic aperture radar data are generated through two sub-backbone networks.

[0050] Step B: The region of interest on the pyramid feature map is obtained by passing the region generation network (RPN).

[0051] Step C involves extracting features from the region of interest and performing ROI alignment to obtain the target region features.

[0052] Step D involves inputting the target region features into the segmentation head, classification head, and regression head, respectively, and outputting the building roof category, roof bounding rectangle, and roof segmentation outline.

[0053] The above-mentioned target roof detection and classification model is used to perform roof detection and classification on multimodal data (RGB+SAR). The feature extractors of the two modalities are densely connected at the feature level, which greatly improves the robustness and accuracy of the model.

[0054] Furthermore, this application also provides another specific implementation method for roof detection and classification, see [link to relevant documentation]. Figure 5 As shown, this method mainly includes the following steps one through five:

[0055] Step 1: Data preprocessing.

[0056] The roof detection and classification method used in this application uses remote sensing RGB data (remote sensing image data) and SAR (synthetic aperture radar) data. Before model development and inference, pixel normalization processing needs to be performed on the two modal data separately, and it needs to be ensured that the two modal data have been aligned.

[0057] Step 2: Self-supervised training of RGB and SAR modal data.

[0058] The backbone network for feature extraction by the detector provided in this embodiment (i.e., the roof detection classification model) consists of two sub-backbone networks (see...). Figure 2 Each sub-backbone network is ConvNeXt V2-base. To improve the sub-backbone network's modeling ability for remote sensing data, this scheme adopts the self-supervised training method FCMAE (Fully Convolutional Masked Autoencoders) of ConvNeXt V2-base. The idea of ​​this self-supervised training scheme is to randomly mask some regions on the input image, and then let the model try to recover the masked parts. Sparse convolution is introduced to model the image, alleviating the incompatibility problem of convolution operations in self-supervised training. The weights in this step are initialized with the pre-trained weights of the open-source ConvNeXtV2-base on the ImageNet22k dataset. Self-supervised training is performed on RGB and SAR datasets respectively, and the completed weights are used as the initial weights of the dual-backbone structure in the detector training.

[0059] Step 3: Training the roof detector.

[0060] The detector model training uses the classic gradient descent and backpropagation algorithms. The training process mainly includes three important components: 1) data augmentation strategy; 2) detector model; 3) supervised loss combination.

[0061] 1) Data Augmentation Strategy: Modified copy paste: During detector training, effective data augmentation strategies are a strong guarantee of detector robustness. Simple copy paste is one of the important data augmentation methods for instance segmentation detectors. This application proposes a modified copy paste for remote sensing scenes, specifically for overhead shots of remote sensing images, to enrich the dataset to the greatest extent possible without destroying the semantic information of the data. The data augmentation process is as follows:

[0062] a) Randomly extract pixel-level instance targets from the dataset;

[0063] b) Randomly scale, flip, and rotate the instance target;

[0064] c) Paste the instance target processed in step b) onto the selected data sample to form a new augmented data sample;

[0065] d) Repeat steps 1) through 3) for the first 90% of rounds of training, and stop using modified copy paste for the last 10% of rounds of training.

[0066] 2) Model Structure: Dual-backbone: This approach uses the classic two-stage instance segmentation framework Cascade Mask R-CNN as its foundation. The basic network components include the backbone, FPN neck, RPN, FCN head, and Cascade R-CNN head. The overall model extracts image features through the backbone. To more flexibly handle RGB+SAR multimodal inputs, this approach references the CBNet network structure and designs two densely connected dual-backbone structures, as shown below. Figure 3 As shown, both sub-backbone networks are ConvNeXt V2-base networks. They enhance the fusion of high-dimensional and low-dimensional information and the fusion of feature information between the two sub-backbone networks through dense connections. When there is a single-modal input, this structure serves as a feature extraction component to enhance the feature extraction capability of the backbone network. When there is a multimodal input, this structure can effectively perform intermodal feature fusion.

[0067] 3) Supervision Loss Combination: The neck and head parts utilize the FPN neck, one segmentation head (FCN head), and three detection heads from cascade mask R-CNN for feature classification and regression. The classification and regression losses for the RPN use cross-entropy loss (CE loss) and smooth L1 loss to supervise the ROI regions extracted by the RPN, respectively. The classification of pixels in the segmentation head uses CE loss. To address the long-tailed distribution of data in real-world scenarios, this scheme uses seesaw loss for supervision in classification, and GIOU loss for supervision in bounding box regression.

[0068] Step 4: Model evaluation.

[0069] After each round of training in step three, the detector model is evaluated using a validation set, and the model weights with the best mAP50 index are selected as the weights to be used for detector inference in the next step.

[0070] Step 5, Model Reasoning.

[0071] The RGB and SAR data of the scene to be detected are simultaneously input into the detector model, which has been trained and filtered through the above steps. Through end-to-end model inference, the detected roof bounding box, category, and pixel-level contour are directly output. The details of the inference process are as follows:

[0072] 1) Load the model weights trained in the second step to initialize the detector model;

[0073] 2) Standardize the RGB and SAR data separately;

[0074] 3) Input the two modal data processed in 2) into the detector backbone network simultaneously. After passing through the two backbone networks, the data forms a pyramid-shaped feature map.

[0075] 4) The feature map is processed by RPN to obtain the Region of Interest (ROI) on the feature map;

[0076] 5) The ROI region features are obtained by using the feature extraction and alignment structure ROI Align;

[0077] 6) Finally, the target region features are fed into the segmentation head, classification head, and regression head respectively to obtain the segmentation mask, category information, and bounding rectangle of the roof region.

[0078] 7) The bounding rectangle output from step 6) is processed by NMS post-processing to obtain the final roof detection and classification results, as shown below. Figure 6 As shown.

[0079] In summary, this application introduces an end-to-end multimodal roof detection and classification scheme based on RGB and SAR, and densely connects the feature extractors of the two modalities at the feature level, greatly improving the robustness and accuracy of the model. To further improve the model's accuracy, this scheme uses a self-supervised pre-training paradigm and adopts effective data augmentation combinations and fine-tuning strategies during training to enhance the model's segmentation and classification accuracy without changing the data distribution. The combination of losses at each stage during training can well cope with the long-tail data distribution and effectively supervise each classification and regression branch.

[0080] In terms of model structure, this application adopts the classic end-to-end instance segmentation algorithm Cascade Mask R-CNN as its basic framework, which can simultaneously output the detected outer bounding box of the building roof, the roof category, and the accurate roof contour segmentation line. For satellite remote sensing scenarios with RGB, SAR multimodal image input or RGB single-modal image input, a flexible dual-backbone structure is innovatively proposed for feature extraction, improving the model's feature extraction capability and enhancing its robustness.

[0081] During model training, this application proposes a more effective data augmentation strategy for satellite remote sensing scenarios: modified copy paste. This strategy expands the existing dataset while maintaining the data distribution, thereby enhancing the model's generalization ability. For model loss, a combination of seesaw loss and GIOU loss is used to enhance the model's resistance to long-tailed distributions, thus significantly improving model accuracy.

[0082] Based on the above method embodiments, this application also provides a roof detection and classification device, see [link to relevant documentation]. Figure 7 As shown, the device mainly includes the following parts:

[0083] The data acquisition module 710 is used to acquire remote sensing image data and synthetic aperture radar data of the scene to be detected;

[0084] The data preprocessing module 720 is used to perform pixel normalization and multimodal data alignment processing on remote sensing image data and synthetic aperture radar data.

[0085] The roof detection and classification module 730 is used to input the processed remote sensing image data and synthetic aperture radar data into the pre-trained target roof detection and classification model, and output the building roof category, roof bounding rectangle and roof segmentation outline; wherein, the target roof detection and classification model is a dual backbone network structure detector that is pre-trained in multiple modes using remote sensing image data and synthetic aperture radar data.

[0086] In an optional embodiment, the above apparatus further includes: a model training module, used for:

[0087] The modal data input to the initial roof detection and classification model are subjected to self-supervised training to obtain the initial weights; the modal data includes remote sensing image sample sets and synthetic aperture radar sample sets.

[0088] The modal data is augmented, and the initial roof detection classification model is trained in a supervised manner using the augmented samples in the first preset proportion of training rounds, and in a supervised manner using the modal data in the second preset proportion of training rounds.

[0089] After each round of training, the detector model determines the corresponding target roof detection classification model based on the target weights corresponding to the optimal mAP50 index.

[0090] In an optional implementation, the model training module described above is further used for:

[0091] Self-supervised training was performed on remote sensing image data and synthetic aperture radar data respectively. The weights obtained after the self-supervised training were determined as the initial weights of the dual-backbone structure in the initial roof detection classification model.

[0092] In this process, self-supervised training involves randomly masking remote sensing image data and synthetic aperture radar data input to the initial roof detection and classification model, and then recovering the masked parts through the model during training, combined with sparse convolution for modeling.

[0093] In an optional implementation, the model training module described above is further used for:

[0094] Pixel-level instance targets are randomly extracted from remote sensing image sample sets and synthetic aperture radar sample sets;

[0095] Perform one or more preset scaling, flipping, and rotation operations on the instance target, and paste the processed instance target into the selected data sample to generate the augmented data sample.

[0096] In an optional implementation, the target roof detection classification model includes two sub-backbone networks, wherein the two sub-backbone networks are densely connected.

[0097] When there is a single-modal input, the network structure of two sub-backbone networks is used to increase the enhanced feature extraction capability of the backbone network;

[0098] When there is a multimodal input, the network structure of the two sub-backbone networks is used for intermodal fusion.

[0099] In an optional implementation, the target roof detection classification model is based on the instance segmentation framework cascade mask R-CNN. The basic network components include a backbone network, FPNneck, region generation network RPN, segmentation head FCNhead, and classification head cascade R-CNN head. The overall model extracts image features through the backbone.

[0100] The neck and head parts are used for feature classification and regression through the FPN neck of cascade mask R-CNN, a segmentation head FCN head, and three detection heads;

[0101] Specifically, the classification and regression losses of RPN were supervised by cross-entropy loss (CE loss) and smooth L1 loss, respectively, for the ROI regions extracted by RPN; the classification of the segmented head pixels was supervised by CE loss; the classification loss was supervised by seesaw loss; and the regression loss of the bounding box was supervised by GIOU loss.

[0102] In an optional implementation, the roof detection and classification module 730 is further used for:

[0103] The processed remote sensing image data and synthetic aperture radar data are simultaneously input into the backbone network of the pre-trained target roof detection and classification model, and pyramid feature maps corresponding to the remote sensing image data and synthetic aperture radar data are generated through two sub-backbone networks.

[0104] The region of interest on the pyramid feature map is obtained by passing the region generation network RPN.

[0105] The target region features are obtained by performing feature extraction on the region of interest and aligning the ROI structure using ROI Align.

[0106] The target region features are input into the segmentation head, classification head, and regression head respectively, and the output is the building roof category, the roof bounding rectangle, and the roof segmentation outline.

[0107] The roof detection and classification device provided in this application has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts of the roof detection and classification device not mentioned in the embodiment can be referred to the corresponding content in the aforementioned roof detection and classification method embodiment.

[0108] This application also provides an electronic device, such as... Figure 8 The diagram shows the structure of the electronic device 100, which includes a processor 81 and a memory 80. The memory 80 stores computer-executable instructions that can be executed by the processor 81. The processor 81 executes the computer-executable instructions to implement any of the above-mentioned roof detection and classification methods.

[0109] exist Figure 8 In the illustrated embodiment, the electronic device further includes a bus 82 and a communication interface 83, wherein the processor 81, the communication interface 83, and the memory 80 are connected via the bus 82.

[0110] The memory 80 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 83 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 82 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 82 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0111] Processor 81 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 81 or by software instructions. The processor 81 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory. The processor 81 reads the information in the memory and, in conjunction with its hardware, completes the steps of the roof detection and classification method of the aforementioned embodiment.

[0112] This application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the above-described roof detection and classification method. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0113] The computer program product of the roof detection and classification method, apparatus, equipment and medium provided in the embodiments of this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0114] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0115] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. In addition, the terms "first," "second," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A roof detection and classification method, characterized by, The method comprises: acquiring remote sensing image data and synthetic aperture radar data of a scene to be detected; performing pixel standardization processing and multi-modal data alignment processing on the remote sensing image data and the synthetic aperture radar data; inputting the processed remote sensing image data and the synthetic aperture radar data into a pre-trained target roof detection classification model to output a building roof category, a roof bounding box and a roof segmentation contour line; wherein the target roof detection classification model is a dual-main network structure detector pre-trained in a multi-modal self-supervised manner using remote sensing image data and synthetic aperture radar data; The training steps of the target roof detection classification model include: performing self-supervised training on the modal data input into the initial roof detection classification model to obtain initialization weights; wherein the modal data includes a remote sensing image sample set and a synthetic aperture radar sample set; performing data enhancement processing on the modal data, and performing supervised training on the initial roof detection classification model using the enhanced samples in a first preset proportion of training rounds, and performing supervised training using the modal data in a second preset proportion of training rounds; the detector model after each round of training is determined based on the target weights corresponding to the optimal mAP50 index. The self-supervised training of the modal data input into the initial roof detection classification model includes: performing self-supervised training on the remote sensing image data and the synthetic aperture radar data respectively, and determining the weights obtained after the self-supervised training as the initialization weights of the dual-backbone structure in the initial roof detection classification model; wherein the self-supervised training is random area masking of the remote sensing image data and the synthetic aperture radar data input into the initial roof detection classification model, and the masked part is recovered through the model during the training process, and modeling is performed in combination with sparse convolution.

2. The roof detection and classification method of claim 1, wherein, The data enhancement processing of the modal data comprises: randomly extracting pixel-level instance targets from the remote sensing image sample set and the synthetic aperture radar sample set; performing one or more of a preset scaling operation, a flipping operation and a rotation operation on the instance targets, and pasting the processed instance targets to selected data samples to generate augmented data samples.

3. The roof detection and classification method according to claim 1 or 2, characterized in that, The target roof detection classification model comprises two sub-main networks, wherein the two sub-main networks are densely connected; When a single modality is input, the network structure of the two sub-main networks is used to increase the enhancement feature extraction capability of the main network; When multiple modalities are input, the network structure of the two sub-main networks is used for inter-modal fusion.

4. The roof detection and classification method of claim 3, wherein, The target roof detection classification model is based on the instance segmentation framework cascade mask rcnn, and the network basic components include a backbone network backbone, an FPN neck, a region generation network RPN, a segmentation head FCN head and a classification head cascade rcnn head, and the overall model extracts image features through the backbone. The neck and head part are classified and regressed by the FPN neck of the cascade mask rcnn and a segmentation head FCN head and three detection heads; The classification and regression losses of the RPN are respectively a cross-entropy loss (CE loss) and a smooth L1 loss, which are used to supervise the ROI regions extracted by the RPN. The classification of the pixel points of the segmentation head adopts a CE loss; the classification loss is supervised by a seesaw loss; and the regression loss of the bbox is supervised by a GIOU loss.

5. The roof detection and classification method of claim 1, wherein, The processed remote sensing image data and the synthetic aperture radar data are input into a pre-trained target roof detection and classification model, and building roof categories, roof bounding boxes and roof segmentation contour lines are output. The processed remote sensing image data and the synthetic aperture radar data are simultaneously input into a main network of the pre-trained target roof detection and classification model, and pyramid feature maps corresponding to the remote sensing image data and the synthetic aperture radar data are generated through two sub-main networks. The pyramid feature maps are input into a region proposal network (RPN) to obtain regions of interest (ROIs) on the pyramid feature maps. The ROIs are subjected to feature extraction and structure alignment ROI Align to obtain target region features. The target region features are input into a segmentation head, a classification head and a regression head, and building roof categories, roof bounding boxes and roof segmentation contour lines are output.

6. A roof detection and classification apparatus, characterized by, The device comprises: a data acquisition module configured to acquire remote sensing image data and synthetic aperture radar data of a scene to be detected; a data preprocessing module configured to perform pixel standardization processing and multi-modal data alignment processing on the remote sensing image data and the synthetic aperture radar data; a roof detection and classification module configured to input the processed remote sensing image data and the synthetic aperture radar data into a pre-trained target roof detection and classification model, and output building roof categories, roof bounding boxes and roof segmentation contour lines; wherein the target roof detection and classification model is a dual-main network structure detector pre-trained in a multi-modal self-supervised manner using remote sensing image data and synthetic aperture radar data; The device further comprises a model training module configured to perform self-supervised training on modal data input into an initial roof detection and classification model to obtain initialization weights; wherein the modal data comprises a remote sensing image sample set and a synthetic aperture radar sample set; the modal data is subjected to data enhancement processing, and the initial roof detection and classification model is supervised and trained using the enhanced samples in a first preset proportion of training rounds and supervised and trained using the modal data in a second preset proportion of training rounds; the detector model after each round of training is determined based on a target weight corresponding to an optimal mAP50 index to obtain a corresponding target roof detection and classification model. The model training module is further configured to perform self-supervised training on the remote sensing image data and the synthetic aperture radar data respectively, and determine the weight obtained after the self-supervised training as the initial weight of the dual backbone structure of the roof detection classification model; wherein the self-supervised training is to randomly mask the remote sensing image data and the synthetic aperture radar data input to the initial roof detection classification model, and recover the masked part through the model during the training process, and model in combination with sparse convolution.

7. An electronic device, comprising: A computer readable storage medium storing computer executable instructions that, when invoked and executed by a processor, cause the processor to implement the roof detection and classification method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions that, when invoked and executed by a processor, cause the processor to implement the roof detection and classification method of any one of claims 1 to 5.

Images