Building roof recognition method and device based on multi-source heterogeneous deep network fusion

Abstract

This application relates to a method, apparatus, computer equipment, storage medium, and computer program product for building roof recognition based on multi-source heterogeneous deep network fusion, which can be used in the field of image recognition technology. This application can improve the efficiency and accuracy of building roof recognition. The method includes: acquiring an optical image and a remote sensing image of a target area; the target area contains a building roof; inputting the optical image into an optical image segmentation model to obtain an optically segmented image of the building roof; inputting the remote sensing image into a remote sensing image segmentation model to obtain a remotely sensed segmented image of the building roof; fusing the optical and remotely sensed segmented images to obtain a fused segmented image of the building roof; and recognizing the fused segmented image to obtain the area information of the building roof.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for building roof recognition based on multi-source heterogeneous deep network fusion. Background Technology

[0002] With the development of power technology, building roof identification has important applications in many fields such as power system planning and management. By analyzing the characteristics of different types of building roofs, we can understand their differences in shape, size, and other aspects, which can be used for subsequent power plant planning and design. Therefore, how to efficiently identify building roofs has become an important research direction.

[0003] Traditional technology typically involves manually identifying building rooftops from images; however, this method requires significant manual processing time, resulting in low efficiency in building rooftop identification. Summary of the Invention

[0004] Therefore, it is necessary to provide a building roof recognition method, device, computer equipment, computer-readable storage medium, and computer program product based on multi-source heterogeneous deep network fusion that can improve the efficiency of building roof recognition, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a method for identifying building rooftops based on multi-source heterogeneous deep network fusion. The method includes:

[0006] Acquire optical images and remote sensing images of the target area; the target area includes building rooftops;

[0007] The optical image is input into the optical image segmentation model to obtain the optical segmentation image of the building roof;

[0008] The remote sensing image is input into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof;

[0009] The optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmentation image of the building roof.

[0010] The area information of the building's roof is obtained by recognizing the fused segmented image.

[0011] In one embodiment, the step of fusing the optical segmentation image and the remote sensing segmentation image to obtain a fused segmentation image of the building roof includes:

[0012] Based on the maximum pixel value of the pixel in the optical segmentation image, the relative pixel value of each pixel in the optical segmentation image is determined;

[0013] The relative pixel value of each pixel in the remote sensing segmented image is determined based on the maximum pixel value of the pixel in the remote sensing segmented image.

[0014] Based on the relative pixel values ​​of each pixel in the optical segmentation image and the relative pixel values ​​of each pixel in the remote sensing segmentation image, the optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmentation image of the building roof.

[0015] In one embodiment, the step of fusing the optical segmentation image and the remote sensing segmentation image based on the relative pixel values ​​of each pixel in the optical segmentation image and the relative pixel values ​​of each pixel in the remote sensing segmentation image to obtain a fused segmented image of the building roof includes:

[0016] Based on the position information of each pixel in the optical segmentation image and the position information of each pixel in the remote sensing segmentation image, the matching relationship between each pixel in the optical segmentation image and each pixel in the remote sensing segmentation image is determined.

[0017] Based on the matching relationship, pixel matching pairs between the optical segmentation image and the remote sensing segmentation image are determined;

[0018] For each pixel matching pair, the pixel with the largest relative pixel value is selected from the pixel matching pair and used as the target pixel in the generated fused segmentation image of the building roof.

[0019] A fused segmented image of the building roof is generated based on each of the target pixels.

[0020] In one embodiment, the step of identifying the fused segmented image to obtain the area information of the building's roof includes:

[0021] Edge detection processing is performed on the fused segmented image to obtain the edge detection image of the building roof;

[0022] The area information of the building's roof is obtained by identifying the edge detection image.

[0023] In one embodiment, the step of identifying the edge detection image to obtain the area information of the building's roof includes:

[0024] The edge detection image is identified to determine the edge feature points in the edge detection image;

[0025] The area enclosed by the edge feature points is identified to obtain the area value of the enclosed area;

[0026] The area value of the enclosed region is identified as the area information of the building's roof.

[0027] In one embodiment, after identifying the fused segmented image to obtain the area information of the building's roof, the method further includes:

[0028] Based on the geographical location information of the building's roof, the corresponding solar altitude angle and solar azimuth angle information are determined;

[0029] Based on the area information, the solar altitude angle information, and the solar azimuth angle information, the photovoltaic panel installation area corresponding to the building roof is determined.

[0030] Secondly, this application also provides a building roof identification device based on multi-source heterogeneous deep network fusion. The device includes:

[0031] The image acquisition module is used to acquire optical images and remote sensing images of the target area; the target area includes building rooftops;

[0032] The first input module is used to input the optical image into the optical image segmentation model to obtain the optical segmentation image of the building roof;

[0033] The second input module is used to input the remote sensing image into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof;

[0034] An image fusion module is used to fuse the optical segmentation image and the remote sensing segmentation image to obtain a fused segmentation image of the building roof;

[0035] An image recognition module is used to recognize the fused and segmented image to obtain the area information of the building's roof.

[0036] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0037] Acquire optical images and remote sensing images of the target area; the target area includes building rooftops;

[0038] The optical image is input into the optical image segmentation model to obtain the optical segmentation image of the building roof;

[0039] The remote sensing image is input into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof;

[0040] The optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmentation image of the building roof.

[0041] The area information of the building's roof is obtained by recognizing the fused segmented image.

[0042] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0043] Acquire optical images and remote sensing images of the target area; the target area includes building rooftops;

[0044] The optical image is input into the optical image segmentation model to obtain the optical segmentation image of the building roof;

[0045] The remote sensing image is input into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof;

[0046] The optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmentation image of the building roof.

[0047] The area information of the building's roof is obtained by recognizing the fused segmented image.

[0048] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0049] Acquire optical images and remote sensing images of the target area; the target area includes building rooftops;

[0050] The optical image is input into the optical image segmentation model to obtain the optical segmentation image of the building roof;

[0051] The remote sensing image is input into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof;

[0052] The optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmentation image of the building roof.

[0053] The area information of the building's roof is obtained by recognizing the fused segmented image.

[0054] The aforementioned method, apparatus, computer equipment, storage medium, and computer program product for building roof identification based on multi-source heterogeneous deep network fusion acquires an optical image and a remote sensing image of a target area; the target area includes building roofs; the optical image is input into an optical image segmentation model to obtain an optical segmented image of the building roof; the remote sensing image is input into a remote sensing image segmentation model to obtain a remote sensing segmented image of the building roof; the optical segmented image and the remote sensing segmented image are fused to obtain a fused segmented image of the building roof; the fused segmented image is then identified to obtain the area information of the building roof. This scheme acquires optical and remote sensing images of the target area, which contain information about building roofs. The optical images are then input into an optical image segmentation model to obtain an optically segmented image of the building roof. Simultaneously, the remote sensing images are input into a remotely sensing image segmentation model to obtain a remotely sensing segmented image of the building roof. Next, the optical and remotely sensing segmented images are fused to obtain a fused segmented image of the building roof. Finally, the fused segmented image is used for recognition to obtain the area information of the building roof, thus improving the efficiency and accuracy of building roof recognition. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating a building roof identification method based on multi-source heterogeneous deep network fusion in one embodiment.

[0057] Figure 2 This is a flowchart illustrating a building roof identification method based on multi-source heterogeneous deep network fusion in another embodiment;

[0058] Figure 3 This is a schematic diagram of the framework for visual transformation image segmentation in one embodiment;

[0059] Figure 4 This is a schematic diagram of the framework of a residual convolution module in one embodiment;

[0060] Figure 5 This is a schematic diagram of the framework of a deterministic network module in one embodiment;

[0061] Figure 6This is a schematic diagram of the framework of a two-layer attention fusion module in one embodiment;

[0062] Figure 7 This is a schematic diagram of the frame of the conversion encoder in one embodiment;

[0063] Figure 8 This is a schematic diagram of the conversion decoder framework in one embodiment;

[0064] Figure 9 This is a schematic diagram of the framework of a shuffling fully convolutional neural network in one embodiment;

[0065] Figure 10 This is a structural block diagram of a building roof recognition device based on multi-source heterogeneous deep network fusion in one embodiment;

[0066] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0069] In one exemplary embodiment, such as Figure 1 As shown, a method for building roof recognition based on multi-source heterogeneous deep network fusion is provided. This embodiment illustrates the application of this method to a terminal; it is understood that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc.; the server can be a standalone server or a server cluster composed of multiple servers. In this embodiment, the method includes the following steps:

[0070] Step S101: Acquire optical images and remote sensing images of the target area; the target area includes building rooftops.

[0071] In this step, the optical image can be an image imaged using light within the visible light range; the remote sensing image can be an image of the Earth's surface acquired using remote sensing technology, and can include information in multiple bands, such as infrared, thermal infrared, etc. For example, the remote sensing image can be a remote sensing satellite image; the target area can be a certain geographical area.

[0072] Optionally, the terminal acquires optical image databases and remote sensing image databases through a geographic information system, and selects optical images and remote sensing images of a specified area from them, wherein these images include building roofs.

[0073] Step S102: Input the optical image into the optical image segmentation model to obtain the optical segmentation image of the building roof.

[0074] In this step, the optical image segmentation model can be a deep learning model that can segment building roofs in an image by learning the features and contextual information of the image. For example, the optical image segmentation model can be a model based on the visual transformation Mask2former (image segmentation) framework based on a residual deterministic network module and a dual attention fusion network module. The segmented image can be an image in which pixels in the image are assigned to different categories or regions to label different targets in the image. The optical segmented image can be the segmented image after optical image segmentation.

[0075] Optionally, the terminal inputs the optical image into the optical image segmentation model. Based on the features and contextual information of the learned image, the optical image segmentation model segments the building roof in the image. In this way, the terminal can obtain the optical segmentation image of the building roof output by the optical image segmentation model.

[0076] Step S103: Input the remote sensing image into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof.

[0077] In this step, the remote sensing image segmentation model can also be a deep learning model, which can segment building roofs in the image by learning the features and contextual information of the image. For example, the remote sensing image segmentation model can be a shuffled fully connected network model (shuffled two-dimensional fully interconnected multilayer neural network model); the remote sensing segmented image can be the segmented image after remote sensing image segmentation.

[0078] Optionally, the terminal inputs the remote sensing image into the remote sensing image segmentation model. Based on the features and contextual information of the learned image, the remote sensing image segmentation model segments the building roofs in the image. In this way, the terminal can obtain the remote sensing segmented image of the building roofs output by the remote sensing image segmentation model.

[0079] Step S104: The optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmentation image of the building roof.

[0080] Optionally, the terminal performs a fusion process on the optical segmentation image and the remote sensing segmentation image. This process can use a pixel-level fusion method, such as comparing the pixels of the two segmentation images and selecting the pixel with the larger pixel value as the pixel of the fused segmentation image of the building roof, thereby obtaining the fused segmentation image of the building roof.

[0081] Step S105: Recognize the fused segmented image to obtain the area information of the building roof.

[0082] In this step, the area information can be the size of the building's roof area, which can be used to assess the usability of the building's roof or for other relevant analyses.

[0083] Optionally, the terminal performs recognition on the fused segmented image to obtain the area information of the building roof. This process can use computer vision algorithms, such as calculating the number of pixels on the building roof or calculating the area of ​​the building roof region.

[0084] The aforementioned building roof recognition method based on multi-source heterogeneous deep network fusion involves acquiring optical and remote sensing images of the target region; the target region contains building roofs; the optical image is input into an optical image segmentation model to obtain an optically segmented image of the building roof; the remote sensing image is input into a remotely sensing image segmentation model to obtain a remotely sensing segmented image of the building roof; the optical and remotely sensing segmented images are fused to obtain a fused segmented image of the building roof; and the fused segmented image is then recognized to obtain the area information of the building roof. This scheme acquires optical and remote sensing images of the target region, which contain information about building roofs. The optical image is input into an optical image segmentation model to obtain an optically segmented image of the building roof, and the remote sensing image is input into a remotely sensing image segmentation model to obtain a remotely sensing segmented image of the building roof. The optical and remotely sensing segmented images are then fused to obtain a fused segmented image of the building roof. Finally, the fused segmented image is recognized to obtain the area information of the building roof, thus improving the efficiency and accuracy of building roof recognition.

[0085] In an exemplary embodiment, in step S104, the optical segmentation image and the remote sensing segmentation image are fused to obtain a fused segmented image of the building roof. Specifically, this includes: determining the relative pixel value of each pixel in the optical segmentation image based on the maximum pixel value of the pixel in the optical segmentation image; determining the relative pixel value of each pixel in the remote sensing segmentation image based on the maximum pixel value of the pixel in the remote sensing segmentation image; and fusing the optical segmentation image and the remote sensing segmentation image based on the relative pixel values ​​of each pixel in the optical segmentation image and the relative pixel values ​​of each pixel in the remote sensing segmentation image to obtain a fused segmented image of the building roof.

[0086] In this embodiment, the relative pixel value can refer to a value that is normalized to a specific range, or it can be the pixel value divided by the maximum pixel value, so that the pixel value is between 0 and 1.

[0087] Optionally, the terminal determines the relative pixel value of each pixel in the optical segmentation image based on the maximum pixel value of the pixels in the optical segmentation image. This can be achieved by dividing the pixel value of each pixel by the maximum pixel value, so that the pixel value of each pixel is between 0 and 1. Similarly, the terminal determines the relative pixel value of each pixel in the remote sensing segmentation image based on the maximum pixel value of the pixels in the remote sensing segmentation image. This can also be achieved by dividing the pixel value of each pixel by the maximum pixel value, so that the pixel value of each pixel is between 0 and 1. The optical segmentation image and the remote sensing segmentation image are fused using the relative pixel values ​​of the optical segmentation image and the remote sensing segmentation image. This process can use a pixel-level fusion method, such as weighted averaging of the relative pixel values ​​of the two segmentation images, to obtain a fused segmentation image of the building roof.

[0088] The technical solution provided in this embodiment, by using the relative pixel values ​​of each pixel in the optical segmentation image and the relative pixel values ​​of each pixel in the remote sensing segmentation image, is conducive to obtaining the fused segmentation image of the building roof more efficiently and accurately, thereby improving the efficiency and accuracy of building roof recognition.

[0089] In an exemplary embodiment, the above steps involve fusing the optical segmentation image and the remote sensing segmentation image based on the relative pixel values ​​of each pixel in the optical segmentation image and the relative pixel values ​​of each pixel in the remote sensing segmentation image to obtain a fused segmented image of the building roof. Specifically, this includes: determining the matching relationship between each pixel in the optical segmentation image and each pixel in the remote sensing segmentation image based on their respective positional information; determining pixel matching pairs between the optical segmentation image and the remote sensing segmentation image based on the matching relationship; selecting the pixel with the largest relative pixel value from each pixel matching pair as the target pixel in the fused segmented image of the building roof to be generated; and generating the fused segmented image of the building roof based on each target pixel.

[0090] In this embodiment, the position information of a pixel can be its position in the image, such as its coordinates. The matching relationship can refer to the correspondence between pixels in the optical segmentation image and pixels in the remote sensing segmentation image. By comparing the position coordinates of pixels, corresponding pixels in the optical segmentation image and the remote sensing segmentation image can be determined. A pixel matching pair can refer to each pair of matched pixels, where each pixel matching pair includes two matched pixels. The target pixel can refer to the pixel with a higher relative pixel value selected from the matching pairs. When generating the fused segmentation image, the selected target pixel will be placed in the new image to form the fused segmentation image of the building roof.

[0091] Optionally, the terminal determines the matching relationship between each pixel in the optical segmentation image and each pixel in the remote sensing segmentation image based on the position information of each pixel in the optical segmentation image and the position information of each pixel in the remote sensing segmentation image. This can be achieved by comparing the position coordinates of the pixels to find the corresponding pixels in the optical segmentation image and the remote sensing segmentation image. Based on the matching relationship, pixel matching pairs between the optical segmentation image and the remote sensing segmentation image are determined, which means finding the corresponding pixels in the optical segmentation image and the remote sensing segmentation image for subsequent fusion processing. For each pixel matching pair, the pixel with the largest relative pixel value is selected as the target pixel of the building roof, which means selecting the pixel with the higher relative pixel value as the target pixel in each matching pair. Based on the selected target pixel, a fused segmented image of the building roof is generated, which can be achieved by placing the selected target pixel in a new image to form the fused segmented image of the building roof.

[0092] The technical solution provided in this embodiment, by using the position information of each pixel in the optical segmentation image and the position information of each pixel in the remote sensing segmentation image, can more accurately obtain the fused segmented image of the building roof, thereby improving the accuracy of building roof recognition.

[0093] In an exemplary embodiment, in step S105, the fused segmented image is identified to obtain the area information of the building roof, which specifically includes the following: edge detection processing is performed on the fused segmented image to obtain the edge detection image of the building roof; the edge detection image is identified to obtain the area information of the building roof.

[0094] In this embodiment, edge detection processing can refer to using computer vision algorithms to process an image in order to identify edge information in the image. Edge detection processing can help find the boundaries of objects in the image, including the edges of building roofs. An edge detection image can refer to an image obtained after edge detection processing. An edge detection image displays the edge information of objects in the image, including the edges of building roofs.

[0095] Optionally, the terminal uses computer vision algorithms (such as edge detection algorithms) to perform edge detection processing on the fused segmented image, which will identify the edges of the building roof and generate an edge detection image; and uses computer vision algorithms (such as image analysis and pattern recognition algorithms) to recognize the edge detection image, which will analyze the edge information in the image and identify the area information of the building roof.

[0096] The technical solution provided in this embodiment, by performing edge detection processing on the fused segmented image, helps to obtain the area information of the building roof more efficiently and accurately, thereby improving the efficiency and accuracy of building roof recognition.

[0097] In an exemplary embodiment, the above steps of identifying the edge detection image to obtain the area information of the building roof specifically include the following: identifying the edge detection image to determine the edge feature points in the edge detection image; identifying the area enclosing the edge feature points to obtain the area value of the enclosing area; and identifying the area value of the enclosing area as the area information of the building roof.

[0098] In this embodiment, edge feature points can be key points of the edge determined in the edge detection image. These points can include the endpoints or corners of the edge, which helps to more accurately identify the shape and boundary of the building roof. The enclosing region can refer to the area around the edge feature points. This region can be determined by connecting the edge feature points or by using a region growing algorithm. The area of ​​the enclosing region can be used to calculate the actual area of ​​the building roof. The area value can refer to the size of the building roof or its enclosing region. In this embodiment, by identifying the enclosing region of the edge feature points, the area value of the building roof can be calculated, which will provide quantitative data on the size of the building roof.

[0099] Optionally, the terminal uses computer vision algorithms (such as corner detection algorithms or edge detection algorithms) to process the edge detection image to determine edge feature points in the image; it uses computer vision algorithms (region growing algorithms or contour analysis algorithms) to identify the region around the edge feature points to determine the actual shape of the building roof and calculate the area value of the region; the calculated area value of the surrounding region is used as the area information of the building roof, which will help determine the usable area of ​​the building roof.

[0100] The technical solution provided in this embodiment, by using edge feature points in the edge detection image, helps to obtain the area information of the building roof more efficiently and accurately, thereby improving the efficiency and accuracy of building roof recognition.

[0101] In an exemplary embodiment, after identifying the fused segmented image and obtaining the area information of the building roof, step S105 further includes the following: determining the corresponding solar altitude angle information and solar azimuth angle information based on the geographical location information of the building roof; and determining the photovoltaic panel installation area corresponding to the building roof based on the area information, solar altitude angle information, and solar azimuth angle information.

[0102] In this embodiment, geographical location information can refer to the geographical coordinates of the building, including latitude and longitude. This information is used to determine the geographical location of the building in order to calculate the solar altitude angle and solar azimuth angle. The solar altitude angle can refer to the angle of the sun relative to the horizon. The solar azimuth angle can refer to the angle of the sun relative to due south. A photovoltaic panel, also known as a solar panel, is a device that converts solar energy into electrical energy. A photovoltaic panel is usually composed of multiple solar cells and can be installed on the roof of a building to collect solar energy and convert it into usable electrical energy. The installation area can refer to the area occupied by the photovoltaic panel on the roof of the building. Based on the roof area information, solar altitude angle information, and solar azimuth angle information, the area of ​​the roof suitable for installing photovoltaic panels can be determined, which will help determine the layout and installation method of the photovoltaic panels.

[0103] Optionally, the terminal calculates the corresponding solar altitude angle and solar azimuth angle based on the geographical location information of the building roof; and determines the area of ​​the building roof suitable for installing photovoltaic panels based on the area information, solar altitude angle information and solar azimuth angle information of the building roof.

[0104] The technical solution provided in this embodiment, by using area information, solar altitude angle information, and solar azimuth angle information, is conducive to efficiently and accurately determining the photovoltaic panel installation area corresponding to the building roof. The photovoltaic panel installation area is helpful for subsequent photovoltaic power station planning and design.

[0105] The following application example illustrates the building roof recognition method based on multi-source heterogeneous deep network fusion provided in this application. This application example demonstrates the application of this method to a terminal. Figure 2 As shown, the main steps include:

[0106] The first step involves the terminal acquiring publicly available optical image databases and remote sensing image databases through a geographic information system (GIS) and selecting optical and remote sensing images of a specified area. The acquired optical images are then cropped. The center coordinates of the optical image are determined as (x_enter, y_center), where x_enter represents the horizontal coordinate and y_center represents the vertical coordinate. The top-left corner coordinates of the cropped area are set to (x_enter-100, y_center-100), and the bottom-right corner coordinates are set to (x_center+100, y_center+100) for image cropping. This results in an optical image with a pixel size of 200×200. This 200×200 image is then cropped again. The top-left corner coordinates of the cropped area are set to (x, y), where x and y represent the column and row indices of the cropped area, respectively. The cropped area size can be set to 20 pixels. The number of cropping operations is set to 10. Finally, the 200×200 optical image is cropped into 10 optical images of size 20×20 pixels each.

[0107] The second step involves the terminal performing rooftop image segmentation on 10 optical images, each 20×20 pixels in size, using a visual transformation (Mask2former) image segmentation framework comprised of a residual deterministic network module and a dual attention fusion network module; for example... Figure 3 As shown, the framework for visual transformation image segmentation based on a residual deterministic network module and a dual attention fusion network module (which can be divided into a visual transformation framework and an image segmentation framework) is as follows: The input layer is a 7×7 convolutional layer; the input layer is the first residual convolutional module; the input layer is the second residual convolutional module; the input layer is the third residual convolutional module; the input layer is the first deterministic network module; the input layer is the first transformation encoder module; the process is divided into two paths. The first path passes through a dual attention fusion module, a global pooling layer, a fully connected layer, and then enters the second deterministic network module. The fully connected layer in the first path is followed by a Softmax layer (normalization layer). The global pooling layer, fully connected layer, and Softmax layer in the first path... The input to the transpose encoder is merged; it then enters the transposed convolutional layer; it enters the Softmax layer; the second path directly enters the second deterministic network module; it enters the second deterministic network module; it enters the second transpose encoder module; it splits into two paths, the first path goes through a two-layer attention fusion module, a global pooling layer, a fully connected layer, and then enters the third deterministic network module; the fully connected layer of the first path is followed by the Softmax layer; the global pooling layer, fully connected layer, and Softmax layer in the first path are merged into the input to the transpose encoder; it then enters the transposed convolutional layer; it enters the Softmax layer; the second path directly enters the third deterministic network module; it enters the third deterministic network module.

[0108] The specific content of each module is as follows: Figure 4As shown, the input to the residual convolution module is split into two paths. The first path goes through a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer. The first path is then merged with the second path. This process is repeated, with the first path going through a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer. The first path is then merged with the second path. This process is repeated, with the first path going through a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer. The first path is then merged with the second path. (The text repeats itself here.) Figure 5 As shown, entering the deterministic network module, it splits into two paths. The first path goes through a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer; the second path goes through a 1×1 convolutional layer. The first and second paths are then merged. It then splits into two paths again, with the first path going through a 1×1 convolutional layer, a 3×3 convolutional layer, and a 1×1 convolutional layer; the first and second paths are then merged. This process is repeated three times in the original text. For example... Figure 6 As shown, the process enters the two-layer attention fusion module, which is divided into two paths. The first path is further divided into four paths. The first path of the four paths goes through a 3×3 convolutional layer and then a transposed convolutional layer. The second path of the four paths goes through a 3×3 convolutional layer and then a transposed convolutional layer. The third path of the four paths goes through a 3×3 convolutional layer and then a transposed convolutional layer. The first and second paths of the four paths are merged, and then merged with the third path of the four paths after passing through a Softmax layer (normalization layer). After passing through a transposed convolutional layer, it is merged with the fourth path of the four paths. The second path is further divided into four paths. The first path of the four paths goes through a transposed convolutional layer. The second path of the four paths goes through a transposed convolutional layer. The third path of the four paths goes through a transposed convolutional layer. The second and third paths of the four paths are merged, and then merged with the first path of the four paths after passing through a Softmax layer, and then merged with the fourth path of the four paths. The first and second paths are then merged. Figure 7 As shown, the input to the transformation encoder is split into two paths. The first path goes through a normalization layer, then splits into three paths, and finally passes through a multi-head attention layer. The first path is then merged with the second path. Alternatively, the first path can be split into two paths: it goes through a normalization layer, a fully connected layer, and a Softmax layer (normalization layer); the first path is then merged with the second path. (The text repeats itself here.) Figure 8 As shown, the image enters the converter decoder and is divided into two paths. The first path passes through a mask attention layer and a normalization layer. The first path is then merged with the second path. The image is then divided into four paths. The first three paths pass through a self-attention layer, a normalization layer, and are then merged with the fourth path. The image is then divided into two paths. The first path passes through a fully connected layer and a normalization layer, and is then merged with the second path. Finally, two optical building roof segmentation images are obtained (e.g., optical building segmentation image 1 and optical building segmentation image 2).

[0109] The third step involves the terminal acquiring a 200×200 pixel remote sensing satellite image and inputting it into a shuffled fully connected network for image segmentation. For example... Figure 9 As shown, the framework of the shuffling fully convolutional neural network is as follows: Entering a 3×3 convolutional layer; entering the first 3×3 max pooling layer; splitting into two paths, the first path goes through a 1×1 grouped convolutional layer, a channel shuffling layer, a 3×3 depthwise convolutional layer, and another 1×1 grouped convolutional layer; the second path goes through a 3×3 average pooling layer; the first and second paths are merged; entering the second 3×3 max pooling layer; splitting into two paths again, the first path goes through a 1×1 grouped convolutional layer, a channel shuffling layer, a 3×3 depthwise convolutional layer, and another 1×1 grouped convolutional layer; the second path goes through a 3×3 average pooling layer; the first and second paths are merged; entering the third 3×3 max pooling layer... A 3×3 max pooling layer is then used; the layers are split into two paths. The first path goes through a 1×1 grouped convolutional layer, a channel shuffling layer, a 3×3 depthwise convolutional layer, and a 1×1 grouped convolutional layer. The second path goes through an average pooling layer. The first and second paths are merged. The layers then enter a fourth 3×3 max pooling layer. The layers then enter a first transposed convolutional layer. The transposed convolutional layer is merged with the third 3×3 max pooling layer. The layers then enter a second transposed convolutional layer. The transposed convolutional layer is merged with the second 3×3 max pooling layer. The layers then enter a third transposed convolutional layer. Finally, a 3×3 global pooling layer is used. The layers then enter a 3×3 convolutional layer. This process ultimately yields a remotely sensed building roof segmentation image (remotely sensed building segmentation image).

[0110] The fourth step involves the terminal determining each pixel in the optical building roof segmentation image. Each pixel in the optical building roof segmentation image is a building roof probability value obtained through a visual transformation (Mask2former) image segmentation method based on a residual deterministic network module and a dual attention fusion network module. The maximum absolute value of all pixels is found, and the absolute value of each pixel is taken and then divided by this maximum value, ensuring that the pixel value range for each pixel is [0, 1].

[0111] The fifth step is to determine each pixel of the remote-sensed building roof segmentation image. The pixel value of each pixel in the remote-sensed building roof segmentation image is the building roof probability value obtained through a shuffled fully connected network method. The maximum absolute value of all pixels is found, and the absolute value of each pixel is taken and then divided by this maximum value, so that the pixel value range of each pixel is [0, 1].

[0112] The sixth step involves the terminal comparing each pixel of the processed optical building roof segmentation image with each pixel of the remote sensing building roof segmentation image; the pixel with the larger pixel is used as the pixel of the building roof segmentation fusion image, thereby obtaining the building roof segmentation fusion image (which can be simply referred to as the building segmentation fusion image).

[0113] Step 7: The terminal performs a Gaussian filter function to convolve each pixel of the building roof segmentation and fusion image with the Gaussian filter function to obtain a Gaussian-filtered building roof segmentation and fusion image. Then, a Laplacian operation is performed on each pixel of the Gaussian-filtered building roof segmentation and fusion image to obtain an edge detection image. When the pixel of the edge detection image is 0, it is identified as an edge marker point.

[0114] Step 8: The terminal finds the maximum and minimum values ​​of the row and column indices within the edge markers, and uses Green's formula to calculate the area enclosed by the edge markers.

[0115] The ninth step involves the terminal determining the area where photovoltaic panels can be installed based on the installation tilt angle, solar altitude angle, and solar azimuth angle, ultimately obtaining the area of ​​the building roof within the image area that can be covered with photovoltaic panels (the photovoltaic area).

[0116] This method consists of two paths. The first path uses a visual transformation (Mask2former) image segmentation method based on a residual deterministic network module and a dual attention fusion network module to obtain a segmented image of the building roof. The second path uses a shuffled fully convolutional neural network method to obtain a segmented image of the building roof from remote sensing satellite images. The edges of the building roof are obtained through a pixel maximum value image fusion method and a Laplacian edge detection method (edge ​​detection algorithm), thereby calculating the usable area of ​​the building roof. Finally, the area of ​​photovoltaic panels that can be installed is estimated based on the usable roof area. This method can improve the accuracy of building roof recognition in complex terrain and reduce the probability of false recognition. In other words, it combines a visual transformation (Mask2former) image segmentation method based on a residual deterministic network module and a dual attention fusion network module, a shuffled fully convolutional neural network method, a pixel maximum value image fusion method, and a Laplacian edge detection method for building roof recognition in complex terrain, thereby improving the accuracy of building roof recognition in complex terrain and optimizing the data acquisition method for the usable area of ​​building roofs.

[0117] The technical solutions provided in this application example achieve the following: First, they segment building roofs in optical images using a visual transformation image segmentation method based on a residual deterministic network module and a dual attention fusion network module. This method understands the contextual information between pixels, improving segmentation accuracy. Second, they segment building roofs in remote sensing images using a shuffling fully convolutional neural network method. This method fuses feature maps from different levels, improving image segmentation accuracy. Third, they fuse optical and remote sensing building roof segmentation images using a pixel maximum value image fusion method. This fully utilizes the building roof segmentation images from both data sources, enabling the acquisition of comprehensive building roof features in complex terrain environments. Fourth, they perform building roof edge detection on the fused building roof segmentation images using a Laplacian edge detection method. This method adapts to the edge features of building roofs of different shapes and has high computational efficiency. These improvements enhance the efficiency and accuracy of building roof recognition.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides a building roof identification device based on multi-source heterogeneous deep network fusion for implementing the building roof identification method based on multi-source heterogeneous deep network fusion described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the building roof identification device based on multi-source heterogeneous deep network fusion provided below can be found in the limitations of the building roof identification method based on multi-source heterogeneous deep network fusion described above, and will not be repeated here.

[0120] In one exemplary embodiment, such as Figure 10 As shown, a building roof identification device based on multi-source heterogeneous deep network fusion is provided. The device 1000 may include:

[0121] Image acquisition module 1001 is used to acquire optical images and remote sensing images of the target area; the target area includes building rooftops;

[0122] The first input module 1002 is used to input the optical image into the optical image segmentation model to obtain the optical segmentation image of the building roof;

[0123] The second input module 1003 is used to input remote sensing images into the remote sensing image segmentation model to obtain remote sensing segmented images of building roofs;

[0124] The image fusion module 1004 is used to fuse optical segmentation images and remote sensing segmentation images to obtain a fused segmentation image of the building roof.

[0125] The image recognition module 1005 is used to recognize the fused segmented image to obtain the area information of the building roof.

[0126] In an exemplary embodiment, the image fusion module 1004 is further configured to determine the relative pixel value of each pixel in the optical segmentation image based on the maximum pixel value of the pixel in the optical segmentation image; determine the relative pixel value of each pixel in the remote sensing segmentation image based on the maximum pixel value of the pixel in the remote sensing segmentation image; and perform fusion processing on the optical segmentation image and the remote sensing segmentation image based on the relative pixel values ​​of each pixel in the optical segmentation image and the relative pixel values ​​of each pixel in the remote sensing segmentation image to obtain a fused segmented image of the building roof.

[0127] In an exemplary embodiment, the image fusion module 1004 is further configured to: determine the matching relationship between each pixel in the optical segmentation image and each pixel in the remote sensing segmentation image based on the position information of each pixel in the optical segmentation image and the position information of each pixel in the remote sensing segmentation image; determine pixel matching pairs between the optical segmentation image and the remote sensing segmentation image based on the matching relationship; select the pixel with the largest relative pixel value from each pixel matching pair as the target pixel in the fused segmentation image to be generated for the building roof; and generate the fused segmentation image of the building roof based on each target pixel.

[0128] In an exemplary embodiment, the image recognition module 1005 is further configured to perform edge detection processing on the fused segmented image to obtain an edge detection image of the building roof; and to recognize the edge detection image to obtain the area information of the building roof.

[0129] In an exemplary embodiment, the image recognition module 1005 is further configured to recognize the edge detection image, determine the edge feature points in the edge detection image; recognize the area enclosing the edge feature points to obtain the area value of the enclosing area; and recognize the area value of the enclosing area as the area information of the building roof.

[0130] In an exemplary embodiment, the device 1000 further includes: an information determination module, configured to determine the corresponding solar altitude angle information and solar azimuth angle information based on the geographical location information of the building roof; and to determine the photovoltaic panel installation area corresponding to the building roof based on the area information, solar altitude angle information, and solar azimuth angle information.

[0131] The modules in the aforementioned building roof recognition device based on multi-source heterogeneous deep network fusion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0132] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, input / output interface, communication interface, display unit, and input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input / output interface is used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a building roof recognition method based on multi-source heterogeneous deep network fusion. The display unit is used to form a visually visible image and can be a display screen, projection device, or virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0133] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0135] In one exemplary embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above-described method embodiments.

[0136] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0137] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0138] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0139] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for identifying building rooftops based on multi-source heterogeneous deep network fusion, characterized in that, The method includes: Acquire optical images and remote sensing images of the target area; the target area includes building rooftops; The optical image is input into the optical image segmentation model to obtain the optical segmentation image of the building roof; The remote sensing image is input into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof; Based on the maximum pixel value of the pixel in the optical segmentation image, the relative pixel value of each pixel in the optical segmentation image is determined; The relative pixel value of each pixel in the remote sensing segmented image is determined based on the maximum pixel value of the pixel in the remote sensing segmented image. Based on the position information of each pixel in the optical segmentation image and the position information of each pixel in the remote sensing segmentation image, the matching relationship between each pixel in the optical segmentation image and each pixel in the remote sensing segmentation image is determined. Based on the matching relationship, pixel matching pairs between the optical segmentation image and the remote sensing segmentation image are determined; For each pixel matching pair, the pixel with the largest relative pixel value is selected from the pixel matching pair and used as the target pixel in the generated fused segmentation image of the building roof. Based on each of the target pixels, a fused segmented image of the building roof is generated; The area information of the building's roof is obtained by recognizing the fused segmented image.

2. The method according to claim 1, characterized in that, The step of identifying the fused segmented image to obtain the area information of the building's roof includes: Edge detection processing is performed on the fused segmented image to obtain the edge detection image of the building roof; The area information of the building's roof is obtained by identifying the edge detection image.

3. The method according to claim 2, characterized in that, The step of identifying the edge detection image to obtain the area information of the building's roof includes: The edge detection image is identified to determine the edge feature points in the edge detection image; The area enclosed by the edge feature points is identified to obtain the area value of the enclosed area; The area value of the enclosed region is identified as the area information of the building's roof.

4. The method according to claim 1, characterized in that, After identifying the fused segmented image to obtain the area information of the building's roof, the process further includes: Based on the geographical location information of the building's roof, the corresponding solar altitude angle and solar azimuth angle information are determined; Based on the area information, the solar altitude angle information, and the solar azimuth angle information, the photovoltaic panel installation area corresponding to the building roof is determined.

5. A building roof recognition device based on multi-source heterogeneous deep network fusion, characterized in that, The device includes: The image acquisition module is used to acquire optical images and remote sensing images of the target area; the target area includes building rooftops; The first input module is used to input the optical image into the optical image segmentation model to obtain the optical segmentation image of the building roof; The second input module is used to input the remote sensing image into the remote sensing image segmentation model to obtain the remote sensing segmentation image of the building roof; The image fusion module is used to: determine the relative pixel value of each pixel in the optical segmentation image based on the maximum pixel value of each pixel in the optical segmentation image; determine the relative pixel value of each pixel in the remote sensing segmentation image based on the maximum pixel value of each pixel in the remote sensing segmentation image; determine the matching relationship between each pixel in the optical segmentation image and each pixel in the remote sensing segmentation image based on the position information of each pixel in the optical segmentation image and the position information of each pixel in the remote sensing segmentation image; determine pixel matching pairs between the optical segmentation image and the remote sensing segmentation image based on the matching relationship; select the pixel with the largest relative pixel value from the pixel matching pair as the target pixel in the fused segmentation image to be generated for the building roof; and generate the fused segmentation image of the building roof based on each target pixel. An image recognition module is used to recognize the fused and segmented image to obtain the area information of the building's roof.

6. The apparatus according to claim 5, characterized in that, The image recognition module is further configured to perform edge detection processing on the fused segmented image to obtain an edge detection image of the building roof; and to recognize the edge detection image to obtain the area information of the building roof.

7. The apparatus according to claim 6, characterized in that, The image recognition module is further configured to recognize the edge detection image, determine the edge feature points in the edge detection image; recognize the area enclosing the edge feature points to obtain the area value of the enclosing area; and recognize the area value of the enclosing area as the area information of the building roof.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 4.

Images