Building segmentation method, device, system and equipment in orthoimage and medium
By processing orthophotos with coding units of different convolutional kernel sizes in the building segmentation model and combining them with a decoding module, the problem of insufficient accuracy in dense building segmentation in existing technologies is solved, achieving higher accuracy in building segmentation and 3D reconstruction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN JIANSHAN TECH CO LTD
- Filing Date
- 2024-10-21
- Publication Date
- 2026-04-21
AI Technical Summary
Existing image segmentation algorithms are not accurate enough in segmenting dense buildings in orthophotos, which leads to reduced accuracy in the reconstruction of 3D building models.
A building segmentation model is adopted, and the first and second coding units with different sizes of feature convolution kernels are used to process the orthophoto. The feature information is merged and decoded by the decoding module to improve the accuracy of building segmentation mask.
By determining building segmentation masks using coding information of varying precision, the accuracy of building segmentation in orthophotos is significantly improved, thereby enhancing the precision of 3D building model reconstruction.
Smart Images
Figure CN121904065A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to methods, apparatus, systems, electronic devices and media for building segmentation in orthophotos. Background Technology
[0002] With the development of science and technology and the increasing demands for quality of life, the need for urban informatization is also growing. The nation has emphasized the importance of building "smart cities" in several medium- and long-term planning outlines. Three-dimensional building models are an indispensable part of smart city construction and the most important foundational data for city models. Among these, building segmentation masks from two-dimensional images are one of the most fundamental features in building reconstruction. Segmentation masks directly reflect the relative and spatial relationships between buildings. Spatial relationships are particularly important for large-scale building reconstruction. Therefore, the quality of building segmentation in orthophotos directly determines the accuracy and precision of large-scale building reconstruction.
[0003] Currently, building segmentation in 2D images typically relies on existing image segmentation algorithms. However, these algorithms are primarily applied to scenarios with small target buildings and limited dense building data. In scenarios with dense, occluded small target building data, these algorithms are prone to false positives and false negatives, leading to reduced accuracy in building segmentation and consequently affecting the reconstruction accuracy of subsequent 3D building models.
[0004] Therefore, improving the accuracy of building segmentation in orthophotos has become a critical issue that urgently needs to be addressed, which is of great significance for the construction of smart cities. Summary of the Invention
[0005] In view of this, the purpose of the embodiments of this application is to provide a method, apparatus, system, electronic device and medium for building segmentation in orthophotos, so as to improve the accuracy of building segmentation in orthophotos.
[0006] In a first aspect, embodiments of this application provide a method for building segmentation in orthophotos, the method comprising:
[0007] Obtain orthophotos of the target city;
[0008] The orthophoto is input into the building segmentation model to obtain the building segmentation mask in the orthophoto. The network structure of the building segmentation model includes a first coding unit and a second coding unit. The kernel of the feature convolutional layer in the first coding unit is smaller than the kernel of the feature convolutional layer in the second coding unit.
[0009] In one embodiment, the building segmentation model further includes a decoding module, wherein inputting the orthophoto into the building segmentation model to obtain a building segmentation mask in the orthophoto includes:
[0010] The orthophoto is input into the first encoding unit to obtain the first encoding result;
[0011] The orthophoto is input into the second encoding unit to obtain the second encoding result;
[0012] The first encoding result and the second encoding result are merged to obtain the merged feature information;
[0013] The merged feature information is decoded in the decoding module to obtain the building segmentation mask in the orthophoto.
[0014] In one embodiment, the second encoding unit includes a second feature extraction network and a second feature processing network; the step of inputting the orthophoto image into the second encoding unit to obtain a second encoding result includes:
[0015] The orthophoto is input into the second feature extraction network to obtain the second feature information;
[0016] The second feature information is input into the second feature processing network to obtain the second encoding result.
[0017] In one embodiment, the second feature processing network includes a second standard convolutional layer, four second feature convolutional layers with different convolution rates, a second global pooling layer, and a second channel convolutional layer.
[0018] The step of inputting the second feature information into the second feature processing network to obtain the second encoding result includes:
[0019] The second feature information is input into the first standard convolutional layer, the four second feature convolutional layers with different convolution rates, and the first global pooling layer, respectively, to obtain the corresponding second output results.
[0020] All second output results are fused to obtain the second fused features;
[0021] The second fusion feature is input into the second channel convolutional layer for channel dimensionality reduction to obtain the second encoding result.
[0022] In one embodiment, the first encoding unit includes a first feature extraction network and a first feature processing network; the step of inputting the orthophoto image into the first encoding unit to obtain a first encoding result includes:
[0023] The orthophoto is input into the first feature extraction network to obtain the first feature information;
[0024] The first feature information is processed based on the first feature processing network to obtain the first encoding result.
[0025] In one embodiment, the first feature processing network includes a first standard convolutional layer, four first feature convolutional layers with different convolution rates, a first global pooling layer, and a first channel convolutional layer.
[0026] The second encoding unit includes a second feature processing network, which includes four second feature convolutional layers with different convolution rates; the kernel size of each of the four first feature convolutional layers with different convolution rates is smaller than the kernel size of each of the four second feature convolutional layers with different convolution rates.
[0027] The step of processing the first feature information based on the first feature processing network to obtain the first encoding result includes:
[0028] The first feature information is respectively input into the first standard convolutional layer, the four first feature convolutional layers with different convolution rates, and the first global pooling layer to obtain the corresponding first output results.
[0029] The first output results are fused to obtain the first fused feature;
[0030] The first fusion feature is input into the first channel convolutional layer for channel dimensionality reduction to obtain the first encoding result;
[0031] The step of merging the first encoding result and the second encoding result to obtain merged feature information includes:
[0032] The first encoding result, the second encoding result, and the first feature information are merged to obtain the merged feature information.
[0033] In one embodiment, the decoding module further includes a channel dimensionality reduction convolutional layer, a first upsampling unit, and a second upsampling unit;
[0034] The step of merging the first encoding result, the second encoding result, and the first feature information to obtain the merged feature information includes:
[0035] The first encoding result is input into the first upsampling unit for upsampling to obtain the first upsampling information;
[0036] The second encoding result is input into the second upsampling unit for upsampling to obtain the second upsampling information;
[0037] The first feature information is input into the channel dimensionality reduction convolutional layer to perform channel dimensionality reduction, and the first feature information after dimensionality reduction is obtained.
[0038] The first upsampling information, the second upsampling information, and the dimensionality-reduced first feature information are merged to obtain the merged feature information.
[0039] In one embodiment, the decoding module includes a channel-reverting convolutional layer and a third upsampling unit;
[0040] The step of decoding the merged feature information in the decoding module to obtain the building segmentation mask in the orthophoto image includes:
[0041] The merged feature information is input into the channel-reduction convolutional layer for channel restoration to obtain the merged feature information after convolution.
[0042] The merged feature information after convolution is input into the third upsampling unit for upsampling to obtain the building segmentation mask in the orthophoto image.
[0043] Secondly, embodiments of this application provide a building segmentation device in orthophotos, the device comprising:
[0044] The acquisition module is used to acquire orthophotos of the target city;
[0045] The building segmentation module is used to input the orthophoto into the building segmentation model to obtain the building segmentation mask of the orthophoto. The building segmentation model includes a first encoding unit and a second encoding unit, wherein the kernel of the feature convolutional layer in the first encoding unit is smaller than the kernel of the feature convolutional layer in the second encoding unit.
[0046] Thirdly, embodiments of this application also provide a building segmentation system in orthophotos, the building segmentation system in orthophotos comprising:
[0047] The image acquisition module is used to acquire orthophotos of the target city and a training set of orthophotos of the city.
[0048] The model training module is used to input the orthophoto training set into the initial building segmentation model for training to obtain the building segmentation model. The network structure of the initial building segmentation model includes a first coding unit and a second coding unit. The kernel of the feature convolutional layer in the first coding unit is smaller than the kernel of the feature convolutional layer in the second coding unit.
[0049] The building segmentation module in the orthophoto is used to input the orthophoto into the building segmentation model to obtain the building segmentation mask in the orthophoto.
[0050] Fourthly, embodiments of this application also provide an electronic device, including: a memory and a processor, the memory storing a computer program executable by the processor, the computer program being executed by the processor to perform the building segmentation method in orthophotos as described in the first aspect above.
[0051] Fifthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions that, when executed by a processor, perform the building segmentation method in an orthophoto as described in the first aspect above.
[0052] In the above implementation process, when determining the building segmentation mask in the orthophoto, the building segmentation model processes the orthophoto through the first coding unit and the second coding unit with different sizes of convolution kernels in the building segmentation model network structure, which can obtain coding information with different precision. Then, the building segmentation mask in the orthophoto is determined based on the coding information with different precision, which effectively improves the accuracy of building segmentation in the orthophoto. Attached Figure Description
[0053] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0054] Figure 1 This is a flowchart illustrating a method for segmenting buildings in orthophotos provided in an embodiment of this application;
[0055] Figure 2 This is a schematic diagram of the structure of a building segmentation model provided in an embodiment of this application;
[0056] Figure 3 This is a schematic diagram of a building segmentation mask provided in an embodiment of this application;
[0057] Figure 4 This is a schematic diagram of a building segmentation device in an orthophoto provided in an embodiment of this application;
[0058] Figure 5 This is a schematic diagram of a building segmentation system in an orthophoto provided in an embodiment of this application;
[0059] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0060] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.
[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this application.
[0062] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" and "a variety" mean two or more, unless otherwise explicitly defined.
[0063] With the development of science and technology and the increasing demands for quality of life, the need for urban informatization is also growing. The nation has emphasized the importance of building "smart cities" in several medium- and long-term planning outlines. Three-dimensional building models are an indispensable part of smart city construction and the most important foundational data for city models. Among these, the segmentation mask of buildings in two-dimensional images is one of the most fundamental features in building reconstruction. The segmentation mask directly reflects the relative and spatial relationships between buildings. Spatial relationships are particularly important for large-scale building reconstruction; therefore, the quality of building segmentation in orthophotos directly determines the accuracy and precision of building reconstruction.
[0064] Currently, building segmentation in 2D images typically relies on existing image segmentation algorithms. However, these algorithms are primarily applied to scenarios with small target buildings and limited dense building data. In scenarios with dense, occluded small target building data, these algorithms are prone to false positives and false negatives, leading to reduced accuracy in building segmentation and consequently affecting the reconstruction accuracy of subsequent 3D building models.
[0065] Therefore, improving the accuracy of building segmentation in orthophotos has become a critical issue that urgently needs to be addressed, which is of great significance for the construction of smart cities.
[0066] Please see Figure 1 , Figure 1This is a flowchart illustrating a method for segmenting buildings in orthophotos according to an embodiment of this application. The method for segmenting buildings in orthophotos may include the following steps, such as steps S101-S102:
[0067] Step S101: Obtain orthophotos of the target city.
[0068] For example, orthophotos are remote sensing images with orthogonal projection properties. Orthophotos of a target city can be acquired by drones, and these orthophotos are typically two-dimensional images that include urban landscapes such as buildings, roads, and vegetation.
[0069] Step S102: Input the orthophoto into the building segmentation model to obtain the building segmentation mask in the orthophoto.
[0070] The network structure of the building segmentation model includes a first encoding unit and a second encoding unit, wherein the kernel of the feature convolutional layer in the first encoding unit is smaller than the kernel of the feature convolutional layer in the second encoding unit.
[0071] It should be noted that the step numbers in the embodiments of this application are only used to distinguish different steps, and are not intended to limit the order or priority of the steps. In some embodiments, certain steps can be processed simultaneously or in different orders.
[0072] For example, the building segmentation model is obtained by training an initial segmentation model. Specifically, before using the building segmentation model to segment buildings in orthophotos, remote sensing images for training, validation, and test sets can be collected and created. The resolution of the remote sensing images is generally 8192*8192 pixels. In the early stage, the images need to be preprocessed to adjust the image size to 1024*1024 pixels. Alternatively, publicly available building segmentation datasets such as WHU Building Dataset, Buildings2Vec, and ChesapeakRSC can be used to train and validate the initial segmentation model, thereby obtaining the building segmentation model. The model structure of the initial building segmentation model can be obtained by improving the DeepLabV3+ network.
[0073] Furthermore, the orthophoto is input into the building segmentation model, which then performs image segmentation on the buildings in the orthophoto to obtain the building segmentation mask in the orthophoto.
[0074] Because the network structure of the building segmentation model includes a first coding unit and a second coding unit with different coding ranges, and the kernel of the feature convolutional layer in the first coding unit is smaller than that in the feature convolutional layer in the second coding unit, the building segmentation model can process the orthophoto with coding units of different kernel sizes when determining the building segmentation mask in the orthophoto, thereby obtaining coding information of different precision. Then, the building segmentation mask in the orthophoto is determined based on the coding information of different precision, which improves the accuracy of building segmentation mask extraction.
[0075] In the above implementation process, when extracting the building segmentation mask from the orthophoto, the building segmentation model processes the orthophoto by using the first coding unit and the second coding unit with different convolution kernel sizes in the building segmentation model, which can obtain coding information with different precision. Then, the building segmentation mask in the orthophoto is determined based on the coding information with different precision, which effectively improves the accuracy of determining the building segmentation mask in the orthophoto.
[0076] In one embodiment, the building segmentation model further includes a decoding module. Step S102, inputting the orthophoto into the building segmentation model to obtain the building segmentation mask in the orthophoto, may include the following steps:
[0077] Step S1021: Input the orthophoto into the first coding unit to obtain the first coding result.
[0078] Step S1022: Input the orthophoto into the second coding unit to obtain the second coding result.
[0079] Step S1023: Merge the first encoding result and the second encoding result to obtain merged feature information.
[0080] Step S1024: Decode the merged feature information in the decoding module to obtain the building segmentation mask in the orthophoto image.
[0081] For example, when inputting an orthophoto into a building segmentation model, the orthophoto can be input into a first encoding unit to obtain a first encoding result. The first encoding result is used to describe the building segmentation features of a first precision in the orthophoto. The orthophoto can be input into a second encoding unit to obtain a second encoding result. The first encoding result can be used to describe the building segmentation features of a second precision in the orthophoto. Further, the first encoding result and the second encoding result are merged to obtain merged feature information. Specifically, the Merge module can be used to merge the first encoding result and the second encoding result. The merged feature information is used to describe the building segmentation features in the orthophoto after the first encoding result and the second encoding result are merged.
[0082] The building segmentation model may also include a decoding module, which can then input the merged feature information into the decoding module for decoding, thereby outputting the feature information describing the building segmentation in the form of an image, i.e., obtaining the building segmentation mask in the orthophoto.
[0083] In the above implementation process, orthophotos are input into the first encoding unit and the second encoding unit respectively, resulting in first and second encoding results describing building segmentation features of different precisions. The first and second encoding results are then merged to obtain merged feature information, which includes building segmentation features of two different precisions. Further, the merged feature information is input into a value decoding module for decoding, thereby causing the building segmentation model to output a building segmentation mask.
[0084] In one embodiment, the second encoding unit includes a second feature extraction network and a second feature processing network; step S1022: inputting the orthophoto image into the second encoding unit to obtain a second encoding result may include the following steps:
[0085] Step a: Input the orthophoto into the second feature extraction network to obtain the second feature information.
[0086] Step b: Input the second feature information into the second feature processing network to obtain the second encoding result.
[0087] For example, the second encoding unit may include a second feature extraction network and a second feature processing network.
[0088] Specifically, the second feature extraction network can be a deep convolutional neural network (DCNN). The orthophoto is input into the second feature extraction network for feature extraction, thereby obtaining the second feature information. The second feature processing network can be an atrous spatial pyramid pooling module (ASPP). The second feature information is input into the second feature processing network for atrous convolution, thereby obtaining the second encoding result.
[0089] Specifically, the ASPP module is a key component for improving the receptive field and multi-scale feature extraction capabilities of Convolutional Neural Networks (CNNs) in semantic segmentation tasks. It primarily captures multi-scale contextual information in images by using atrous convolutions with varying dilation rates. The main function of the ASPP module is to expand the receptive field, extract multi-scale features, and integrate global contextual information through atrous convolutions and global average pooling. These characteristics enable ASPP to more accurately segment targets of different scales in semantic segmentation tasks, improving the model's segmentation accuracy.
[0090] In the above implementation process, the orthophoto is input into the second feature extraction network for feature extraction to obtain the second feature information. Furthermore, the second feature information is input into the second feature processing network for dilated convolution to obtain the second encoding result, thus realizing the feature extraction and feature processing of buildings in the orthophoto.
[0091] In one embodiment, the second feature processing network includes a second standard convolutional layer, four second feature convolutional layers with different convolution rates, a second global pooling layer, and a second channel convolutional layer.
[0092] Step b: Input the second feature information into the second feature processing network to obtain the second encoding result, which may include the following steps:
[0093] Step b1: Input the second feature information into a second standard convolutional layer, four second feature convolutional layers with different convolution rates, and a second global pooling layer to obtain the corresponding second output results.
[0094] Step b2: Perform feature fusion on all second output results to obtain the second fused features.
[0095] Step b3: Input the second fused feature into a second-channel convolutional layer for channel dimensionality reduction to obtain the second encoding result.
[0096] For example, the second feature processing network may include a second standard convolutional layer, four second feature convolutional layers with different convolution rates, a second global pooling layer, and a second channel convolutional layer. The second standard convolutional layer may be a 1*1 dilated convolution. The kernel sizes of the four second feature convolutional layers with different convolution rates may be the same or different. In this embodiment, taking the kernel size of the four second feature convolutional layers with different convolution rates as an example, for instance, the four second feature convolutional layers with different convolution rates may all be dilated convolutions, the kernel size may all be 5*5, and the convolution rates may be rate3, rate6, rate12, and rate18, respectively. The second channel convolutional layer may be a 1*1 standard convolution.
[0097] Specifically, the second extracted features are input into a 1*1 second standard convolutional layer with dilated convolution, a 5*5 kernel, and convolution rates of rate3, rate6, rate12, and rate18, respectively. This is followed by a second feature convolutional layer with dilated convolution and a second global pooling layer with dilated convolution, yielding corresponding second output results. Further, all second output results are fused to obtain second fused features. This can be achieved using the Merge module. The second fused features are then input into a 1*1 second channel convolutional layer for channel dimensionality reduction, reducing the number of channels to increase the image size, thus obtaining the second encoding result.
[0098] In the above implementation process, the second extracted features are convolved and pooled through a second standard convolutional layer, four second feature convolutional layers with different convolution rates, and a second global pooling layer, thereby realizing the feature processing of the second extracted features. All the processed second output results are merged, and then the merged second fused features are channel-reduced through a second channel convolutional layer, thereby realizing the feature processing of the second feature information and finally obtaining the second encoding result with second precision.
[0099] In one embodiment, the first encoding unit includes a first feature extraction network and a first feature processing network; step S1021: inputting the orthophoto image into the first encoding unit to obtain a first encoding result may include the following steps:
[0100] Step A: Input the orthophoto into the first feature extraction network to obtain the first feature information.
[0101] Step B: Process the first feature information based on the first feature processing network to obtain the first encoding result.
[0102] For example, the first encoding unit may include a first feature extraction network and a first feature processing network.
[0103] Specifically, the first feature extraction network can be a DCNN network. The orthophoto is input into the first feature extraction network for feature extraction, thereby obtaining the first feature information. The first feature processing network can be an ASPP with dilated convolutions. The first feature information is input into the first feature processing network for dilated convolutions, thereby obtaining the first encoding result. The kernel size of the first feature processing network is smaller than the kernel size of the second feature processing network, thus making the accuracy of the first feature information greater than the accuracy of the second feature information.
[0104] In the above implementation process, the first feature information is obtained by inputting the orthophoto into the first feature extraction network for feature extraction. Furthermore, the first feature information is input into the first feature processing network for dilated convolution to obtain the first encoding result, thereby realizing feature extraction and feature processing of building outline information in the orthophoto.
[0105] In one embodiment, the first feature processing network includes a first standard convolutional layer, four first feature convolutional layers with different convolution rates, a first global pooling layer, and a first channel convolutional layer.
[0106] The second coding unit includes a second feature processing network, which includes four second feature convolutional layers with different convolution rates; the kernel size of each of the four first feature convolutional layers with different convolution rates is smaller than the kernel size of the four second feature convolutional layers with different convolution rates.
[0107] Step B: Process the first feature information based on the first feature processing network to obtain the first encoding result, which may include the following steps:
[0108] Step B1: Input the first feature information into a first standard convolutional layer, four first feature convolutional layers with different convolution rates, and a first global pooling layer to obtain the corresponding first output results.
[0109] Step B2: Perform feature fusion on all the first output results to obtain the first fused feature.
[0110] Step B3: Input the first fused feature into a first-channel convolutional layer for channel dimensionality reduction to obtain the first encoding result.
[0111] Then, step S1023: merging the first encoding result and the second encoding result to obtain merged feature information may include: merging the first encoding result, the second encoding result and the first feature information to obtain merged feature information.
[0112] For example, Figure 2 This is a structural schematic diagram of a building segmentation model provided in an embodiment of this application, such as... Figure 2 The network structure of the building segmentation model shown includes a first coding unit and a second coding unit. The first coding unit includes a first feature extraction network and a first feature processing network, and the second coding unit may include a second feature extraction network and a second feature processing network.
[0113] It should be noted that the first feature extraction network and the second feature extraction network can be a single DCNN network or two DCNN networks with the same structure. In this embodiment, in order to clearly represent the data flow, the first feature extraction network and the second feature extraction network in the building segmentation model of this embodiment are set as two DCNN networks with the same structure, but this application is not limited to this.
[0114] The first feature processing network may include a first standard convolutional layer, four first feature convolutional layers with different convolution rates, a first global pooling layer, and a first channel convolutional layer. The first standard convolutional layer may be a 1*1 dilated convolution. The kernel sizes of the four first feature convolutional layers with different convolution rates may be the same or different. In this embodiment, taking the kernel size of the four first feature convolutional layers with different convolution rates as an example, for example, the four first feature convolutional layers with different convolution rates may all be dilated convolutional layers, the kernel size may all be 3*3, and the convolution rates may be rate3, rate6, rate12, and rate18, respectively. The first channel convolutional layer may also be a 1*1 standard convolution.
[0115] The kernel sizes of the four first-feature convolutional layers with different convolutional rates are all smaller than the kernel sizes of the four second-feature convolutional layers with different convolutional rates. For example, if the kernel size of the four first-feature convolutional layers with different convolutional rates in the first-feature processing network is 3*3, the kernel size of the four second-feature convolutional layers with different convolutional rates in the second-feature processing network can all be 5*5; if the kernel size of the four first-feature convolutional layers with different convolutional rates in the first-feature processing network is 5*5, the kernel size of the four second-feature convolutional layers with different convolutional rates in the second-feature processing network can all be 7*7, without any restrictions. In deep learning, smaller convolutional kernels (such as 3*3) generally have fewer parameters and less computation than larger convolutional kernels (such as 5*5 or larger). This helps to speed up model training while reducing the demand for memory and computing resources. However, a small convolution kernel can limit the expressive power of the model to some extent. Therefore, when processing orthophotos using a building segmentation model that includes two sizes of convolution kernels, not only can the building features obtained by the small convolution kernel be preserved, but the orthophotos can also be processed by the large convolution kernel, which improves the expressive power of the building segmentation model and further increases the accuracy of building segmentation in orthophotos.
[0116] Specifically, the first feature information is input into a 1*1 first standard convolutional layer with dilated convolution, a first feature convolutional layer with dilated convolution and a first global pooling layer with dilated convolution, all with a kernel size of 3*3 and convolution rates of rate3, rate6, rate12 and rate18, respectively, to obtain the corresponding first output results.
[0117] Furthermore, all the first output results are fused to obtain the first fused feature. As an example, the Merge module can be used to fuse all the first output results to obtain the first fused feature. Then, the first fused feature is input into a 1*1 first channel convolutional layer for channel dimensionality reduction, which reduces the number of channels to increase the image size, thereby obtaining the first encoding result.
[0118] Furthermore, in the decoding module, the first encoding result, the second encoding result, and the first feature information are merged to obtain merged feature information.
[0119] In the above implementation process, the first feature information is convolved and processed through a first standard convolutional layer, four first feature convolutional layers with different convolutional rates, and a first global pooling layer, thereby realizing the feature processing of the first feature information. Then, all the first output results after feature processing are fused to obtain the first fused feature. The first fused feature is then passed through a first channel convolutional layer for channel dimensionality reduction to obtain the first encoding result. Finally, the first feature information, the first encoding result, and the second encoding result are merged in the decoding module to obtain the merged feature information. This allows the decoding module to output a building segmentation mask based on the merged feature information. As a result, the output building segmentation mask includes not only the feature information obtained after processing by two encoding units with different convolutional kernel sizes, but also the first feature information, effectively improving the accuracy of the output building segmentation mask.
[0120] In one embodiment, the decoding module further includes a channel dimensionality reduction convolutional layer, a first upsampling unit, and a second upsampling unit.
[0121] The first encoding result, the second encoding result, and the first feature information are merged to obtain the merged feature information, which may include the following steps:
[0122] Step S1: Input the first encoding result into the first upsampling unit for upsampling to obtain the first upsampling information.
[0123] Step S2: Input the second encoding result into the second upsampling unit for upsampling to obtain the second upsampling information.
[0124] Step S3: Input the first feature information into the channel dimensionality reduction convolutional layer for channel dimensionality reduction to obtain the first feature information after dimensionality reduction.
[0125] Step S4: Merge the first upsampled information, the second upsampled information, and the dimensionality-reduced first feature information to obtain merged feature information.
[0126] For example, such as Figure 2 As shown, the decoding module may further include a channel-reduction convolutional layer, a first upsampling unit, and a second upsampling unit. The channel-reduction convolutional layer can be a standard 1x1 convolutional layer. Both the first and second upsampling units can be 4x upsampling units. The first encoding result is input into the first upsampling unit for upsampling to obtain first upsampling information; the second encoding result is input into the second upsampling unit for upsampling to obtain second upsampling information, thereby adjusting the number of channels in the first and second encoding information to be consistent. Furthermore, the first feature information is input into the channel-reduction convolutional layer for channel-reduction to obtain the dimensionality-reduced first feature information.
[0127] It should be noted that the first upsampling unit and the second upsampling unit in this embodiment can be the same upsampling unit. In this embodiment, in order to improve the running efficiency of the model, two identical first upsampling units and second upsampling units are set in the building segmentation model to upsample the output results of the corresponding channel convolutional layers.
[0128] Finally, the first upsampled information, the second upsampled information, and the dimensionality-reduced first feature information are merged to obtain the merged feature information, thus increasing the feature dimension of the merged feature information. Specifically, the concat module can be used to merge features to obtain the merged feature information.
[0129] In the above implementation process, the first encoding result and the second encoding result are upsampled respectively to adjust the channels of the first encoding result and the second encoding result to be consistent. Furthermore, the first feature information is reduced in dimensionality through a channel dimensionality reduction convolutional layer to avoid the influence of too many channels of the first feature information on the first encoding result and the second encoding result. Then, the first upsampled information, the second upsampled information, and the dimensionality-reduced first feature information are merged to increase the resolution and feature dimension of the merged feature information.
[0130] In one embodiment, the decoding module includes a channel-reverting convolutional layer and a third upsampling unit.
[0131] Step S1024: Decode the merged feature information in the decoding module to obtain the building segmentation mask in the orthophoto image, which may include the following steps:
[0132] Step S10241: Input the merged feature information into the channel restoration convolutional layer to perform channel restoration and obtain the merged feature information after convolution.
[0133] Step S10242: Input the merged feature information after convolution into the third upsampling unit for upsampling to obtain the building segmentation mask in the orthophoto.
[0134] For example, such as Figure 2 The decoding module in the building segmentation model shown also includes a channel decomposition convolutional layer and a third upsampling unit. The channel decomposition convolutional layer can be a 3x3 convolutional layer used to restore the number of channels for merging feature information, and the third upsampling unit can be a 4x upsampling unit.
[0135] In the decoding module, the channel restoration convolution layer is used to restore the input channel of the merged feature information to obtain the merged feature information after convolution.
[0136] Finally, the merged feature information after convolution is input into the third upsampling unit for upsampling to obtain the building segmentation mask in the orthophoto, so that the size of the building segmentation mask is consistent with the size of the orthophoto.
[0137] Figure 3 This is a schematic diagram of a building segmentation mask provided in an embodiment of this application, such as... Figure 3 As shown, Figure 3 Each white area in the image corresponds to a segmentation mask for a building.
[0138] In the above implementation process, the channel restoration of the merged feature information is realized through the channel restoration convolutional layer of the decoding module. The merged feature information after convolution is further upsampled to obtain a building segmentation mask with the same size as the orthophoto image.
[0139] Figure 4 This is a schematic diagram of a building segmentation device in an orthophoto provided in an embodiment of this application, as shown below. Figure 4 The building segmentation device shown in the orthophoto image includes:
[0140] The acquisition module 401 is used to acquire orthophotos of the target city.
[0141] The building segmentation module 402 is used to input orthophotos into the building segmentation model to obtain building segmentation masks of the orthophotos. The network structure of the building segmentation model includes a first coding unit and a second coding unit. The convolutional kernel of the feature convolutional layer in the first coding unit is smaller than the convolutional kernel of the feature convolutional layer in the second coding unit.
[0142] In one embodiment, the building segmentation model further includes a decoding module, the building segmentation module 402 being specifically used for:
[0143] The orthophoto is input into the first coding unit to obtain the first coding result;
[0144] The orthophoto is input into the second coding unit to obtain the second coding result;
[0145] The first encoding result and the second encoding result are merged to obtain the merged feature information;
[0146] The merged feature information is decoded in the decoding module to obtain the building segmentation mask in the orthophoto.
[0147] In one embodiment, the second encoding unit includes a second feature extraction network and a second feature processing network; the building segmentation module 402 is specifically used for:
[0148] The orthophoto is input into the second feature extraction network to obtain the second feature information;
[0149] The second feature information is input into the second feature processing network to obtain the second encoding result.
[0150] In one embodiment, the second feature processing network includes a second standard convolutional layer, four second feature convolutional layers with different convolution rates, a second global pooling layer, and a second channel convolutional layer; the building segmentation module 402 is specifically used for:
[0151] The second feature information is input into a second standard convolutional layer, four second feature convolutional layers with different convolution rates, and a second global pooling layer, respectively, to obtain the corresponding second output results.
[0152] All second output results are fused to obtain the second fused features;
[0153] The second fusion feature is input into a second-channel convolutional layer for channel dimensionality reduction to obtain the second encoding result.
[0154] In one embodiment, the first encoding unit includes a first feature extraction network and a first feature processing network; the building segmentation module 402 is specifically used for:
[0155] The orthophoto is input into the first feature extraction network to obtain the first feature information;
[0156] The first feature information is processed by the first feature processing network to obtain the first encoding result.
[0157] In one embodiment, the first feature processing network includes a first standard convolutional layer, four first feature convolutional layers with different convolution rates, a first global pooling layer, and a first channel convolutional layer.
[0158] The second coding unit includes a second feature processing network, which includes four second feature convolutional layers with different convolution rates; the kernel size of each of the four first feature convolutional layers with different convolution rates is smaller than the kernel size of the four second feature convolutional layers with different convolution rates.
[0159] Building segmentation module 402 is specifically used for:
[0160] The first feature information is input into a first standard convolutional layer, four first feature convolutional layers with different convolution rates, and a first global pooling layer, respectively, to obtain the corresponding first output results.
[0161] The first output results are fused to obtain the first fused feature;
[0162] The first fused feature is input into a first-channel convolutional layer for channel dimensionality reduction to obtain the first encoding result;
[0163] The first encoding result, the second encoding result, and the first feature information are merged to obtain the merged feature information.
[0164] In one embodiment, the decoding module further includes a channel dimensionality reduction convolutional layer, a first upsampling unit, and a second upsampling unit; the building segmentation module 402 is specifically used for:
[0165] The first encoding result is input into the first upsampling unit for upsampling to obtain the first upsampling information;
[0166] The second encoding result is input into the second upsampling unit for upsampling to obtain the second upsampling information;
[0167] The first feature information is input into the channel dimensionality reduction convolutional layer for channel dimensionality reduction to obtain the first feature information after dimensionality reduction.
[0168] The first upsampled information, the second upsampled information, and the first feature information after dimensionality reduction are merged to obtain the merged feature information.
[0169] In one embodiment, the decoding module includes a channel-restored convolutional layer and a third upsampling unit; the building segmentation module 402 is specifically used for:
[0170] The merged feature information is input into the channel-reduction convolutional layer for channel reduction, and the merged feature information after convolution is obtained.
[0171] The merged feature information after convolution is input into the third upsampling unit for upsampling to obtain the building segmentation mask in the orthophoto.
[0172] It should be understood that the building segmentation device in the orthophoto corresponds to the above-described method embodiment for building segmentation in orthophoto and is capable of performing the various steps involved in the above method embodiment. The specific functions of the building segmentation device in the orthophoto can be found in the description above; to avoid repetition, detailed descriptions are omitted here. The building segmentation device in the orthophoto includes at least one software function module that can be stored in a memory or embedded in the device's operating system in the form of software or firmware.
[0173] Please refer to Figure 5 , Figure 5 This is a schematic diagram of a building segmentation system in an orthophoto provided in an embodiment of this application. The system can be used to perform the various steps involved in the above-described method embodiments. The system includes:
[0174] Image acquisition module 501 is used to acquire orthophotos of the target city and a training set of orthophotos of the city.
[0175] The model training module 502 is used to input the orthophoto training set into the initial building segmentation model for training to obtain the building segmentation model. The initial building segmentation model includes a first coding unit and a second coding unit. The kernel of the feature convolutional layer in the first coding unit is smaller than the kernel of the feature convolutional layer in the second coding unit.
[0176] The building segmentation module 503 in orthophotos is used to input the orthophotos into the building segmentation model to obtain the building segmentation mask in the orthophotos.
[0177] Please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. An electronic device 600 provided in this application includes a processor 601 and a memory 602. These components are interconnected and communicate with each other via a communication bus 603 and / or other forms of connection mechanisms (not shown). The memory 602 stores a computer program executable by the processor 601. When executed by the processor 601, the computer program performs the building segmentation method in orthophotos described in the first aspect above or the image generation method described in the second aspect.
[0178] This application also provides a computer-readable storage medium storing computer program instructions, which, when executed by a processor 601, perform the building segmentation method in an orthophoto as described in the first aspect above.
[0179] The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.
[0180] It should be understood that the disclosed apparatus / systems and methods can also be implemented in other ways, as provided in the embodiments of this application. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0181] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.
[0182] The above description is only an optional implementation of the embodiments of this application, but the protection scope of the embodiments of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the embodiments of this application should be covered within the protection scope of the embodiments of this application.
Claims
1. A method for segmenting buildings in orthophotos, characterized in that, The method includes: Obtain orthophotos of the target city; The orthophoto is input into the building segmentation model to obtain the building segmentation mask in the orthophoto. The network structure of the building segmentation model includes a first coding unit and a second coding unit. The kernel of the feature convolutional layer in the first coding unit is smaller than the kernel of the feature convolutional layer in the second coding unit.
2. The method according to claim 1, characterized in that, The building segmentation model further includes a decoding module, wherein inputting the orthophoto into the building segmentation model to obtain the building segmentation mask in the orthophoto includes: The orthophoto is input into the first encoding unit to obtain the first encoding result; The orthophoto is input into the second encoding unit to obtain the second encoding result; The first encoding result and the second encoding result are merged to obtain the merged feature information; The merged feature information is decoded in the decoding module to obtain the building segmentation mask in the orthophoto.
3. The method according to claim 2, characterized in that, The second encoding unit includes a second feature extraction network and a second feature processing network; The step of inputting the orthophoto image into the second encoding unit to obtain the second encoding result includes: The orthophoto is input into the second feature extraction network to obtain the second feature information; The second feature information is input into the second feature processing network to obtain the second encoding result.
4. The method according to claim 3, characterized in that, The second feature processing network includes a second standard convolutional layer, four second feature convolutional layers with different convolution rates, a second global pooling layer, and a second channel convolutional layer; The step of inputting the second feature information into the second feature processing network to obtain the second encoding result includes: The second feature information is input into the first standard convolutional layer, the four second feature convolutional layers with different convolution rates, and the first global pooling layer, respectively, to obtain the corresponding second output results. All second output results are fused to obtain the second fused features; The second fusion feature is input into the second channel convolutional layer for channel dimensionality reduction to obtain the second encoding result.
5. The method according to any one of claims 2 to 4, characterized in that, The first encoding unit includes a first feature extraction network and a first feature processing network; The step of inputting the orthophoto image into the first encoding unit to obtain the first encoding result includes: The orthophoto is input into the first feature extraction network to obtain the first feature information; The first feature information is processed based on the first feature processing network to obtain the first encoding result.
6. The method according to claim 5, characterized in that, The first feature processing network includes a first standard convolutional layer, four first feature convolutional layers with different convolution rates, a first global pooling layer, and a first channel convolutional layer; The second encoding unit includes a second feature processing network, which includes four second feature convolutional layers with different convolution rates; the kernel size of each of the four first feature convolutional layers with different convolution rates is smaller than the kernel size of each of the four second feature convolutional layers with different convolution rates. The step of processing the first feature information based on the first feature processing network to obtain the first encoding result includes: The first feature information is respectively input into the first standard convolutional layer, the four first feature convolutional layers with different convolution rates, and the first global pooling layer to obtain the corresponding first output results. The first output results are fused to obtain the first fused feature; The first fusion feature is input into the first channel convolutional layer for channel dimensionality reduction to obtain the first encoding result; The step of merging the first encoding result and the second encoding result to obtain merged feature information includes: The first encoding result, the second encoding result, and the first feature information are merged to obtain the merged feature information.
7. The method according to claim 6, characterized in that, The decoding module also includes a channel dimensionality reduction convolutional layer, a first upsampling unit, and a second upsampling unit; The step of merging the first encoding result, the second encoding result, and the first feature information to obtain the merged feature information includes: The first encoding result is input into the first upsampling unit for upsampling to obtain the first upsampling information; The second encoding result is input into the second upsampling unit for upsampling to obtain the second upsampling information; The first feature information is input into the channel dimensionality reduction convolutional layer to perform channel dimensionality reduction, and the first feature information after dimensionality reduction is obtained. The first upsampling information, the second upsampling information, and the dimensionality-reduced first feature information are merged to obtain the merged feature information.
8. The method according to claim 2, characterized in that, The decoding module includes a channel-restored convolutional layer and a third upsampling unit; The step of decoding the merged feature information in the decoding module to obtain the building segmentation mask in the orthophoto image includes: The merged feature information is input into the channel-reduction convolutional layer for channel restoration to obtain the merged feature information after convolution. The merged feature information after convolution is input into the third upsampling unit for upsampling to obtain the building segmentation mask in the orthophoto.
9. A device for segmenting buildings in an orthophoto, characterized in that, The device includes: The acquisition module is used to acquire orthophotos of the target city; The building segmentation module is used to input the orthophoto into the building segmentation model to obtain the building segmentation mask of the orthophoto. The network structure of the building segmentation model includes a first encoding unit and a second encoding unit. The kernel of the feature convolutional layer in the first encoding unit is smaller than the kernel of the feature convolutional layer in the second encoding unit.
10. A building segmentation system in orthophotos, characterized in that, The building segmentation system in the orthophoto image includes: The image acquisition module is used to acquire orthophotos of the target city and a training set of orthophotos of the city. The model training module is used to input the orthophoto training set into the initial building segmentation model for training to obtain the building segmentation model. The network structure of the initial building segmentation model includes a first coding unit and a second coding unit. The kernel of the feature convolutional layer in the first coding unit is smaller than the kernel of the feature convolutional layer in the second coding unit. The building segmentation module in the orthophoto is used to input the orthophoto into the building segmentation model to obtain the building segmentation mask in the orthophoto.
11. An electronic device, characterized in that, The electronic device includes: Memory; processor; The memory stores a computer program executable by the processor, which, when executed by the processor, performs the building segmentation method in orthophotos according to any one of claims 1-8.
12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, perform the building segmentation method in an orthophoto as described in any one of claims 1-8.