Semantic segmentation method and device of three-dimensional model, electronic equipment and storage medium

Abstract

The invention relates to a semantic segmentation method and device of a three-dimensional model, electronic equipment and a storage medium. The method comprises the following steps: projecting a to-be-segmented three-dimensional model to a two-dimensional plane by adopting orthogonal projection to obtain an orthographic image corresponding to the three-dimensional model; performing semantic segmentation on the orthographic image to obtain a semantic segmentation result of the orthographic image; based on a semantic segmentation result of the orthographic image, determining a target area in the orthographic image, wherein the semantic segmentation quality of the target area meets a preset low-quality condition; obtaining annotation data of the target area; based on the annotation data of the target area, updating a semantic segmentation result of the orthographic image; and determining a semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthographic image and a corresponding relationship between pixels in the orthographic image and three-dimensional primitives in the three-dimensional model.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a semantic segmentation method, device, electronic device and storage medium for a three-dimensional model. Background Art

[0002] As the global urbanization process continues to accelerate and the scale of cities continues to expand, urban management and planning are facing more and more challenges. In order to effectively respond to these challenges and provide intelligent urban solutions, 3D semantic segmentation of large-scale aerial scenes has become an important research and application field. Although traditional city maps and remote sensing images can provide some basic information about the city, they have limitations in understanding the specific scenes, structures and semantics in the city. However, with the development of aerial photography technology and the maturity of 3D geometric reconstruction algorithms, it has become possible to obtain high-precision geometric models of large-scale urban aerial scenes. 3D semantic segmentation of large-scale aerial scenes aims to classify the semantic categories of each point in urban aerial data. By accurately segmenting different semantic categories such as buildings, roads, pedestrians, and vehicles, a detailed understanding and modeling of urban scenes can be achieved. This understanding and modeling can provide important information and support for urban planning, traffic management, building safety, environmental monitoring and other fields, thereby promoting the sustainable development of cities and providing more intelligent urban services. Summary of the invention

[0003] The present disclosure provides a technical solution for semantic segmentation of three-dimensional models.

[0004] According to one aspect of the present disclosure, a semantic segmentation method for a three-dimensional model is provided, comprising:

[0005] Projecting the three-dimensional model to be segmented onto a two-dimensional plane using orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model;

[0006] Performing semantic segmentation on the orthophoto image to obtain a semantic segmentation result of the orthophoto image;

[0007] Based on the semantic segmentation result of the orthoimage, determining a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition;

[0008] Obtaining the labeling data of the target area;

[0009] Based on the annotation data of the target area, updating the semantic segmentation result of the orthoimage;

[0010] Based on the updated semantic segmentation result of the orthoimage and the correspondence between pixels in the orthoimage and three-dimensional primitives in the three-dimensional model, a semantic segmentation result of the three-dimensional model is determined.

[0011] In one possible implementation,

[0012] The performing semantic segmentation on the orthophoto image to obtain the semantic segmentation result of the orthophoto image comprises: performing semantic segmentation on the orthophoto image through a semantic segmentation network to obtain the semantic segmentation result of the orthophoto image;

[0013] The updating of the semantic segmentation result of the orthophoto image based on the annotation data of the target area includes: updating the parameters of the semantic segmentation network based on the annotation data of the target area and the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image; performing semantic segmentation on the orthophoto image through the semantic segmentation network with updated parameters to obtain an updated semantic segmentation result of the orthophoto image.

[0014] In a possible implementation, the determining, based on the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality satisfies a preset low-quality condition includes:

[0015] For each pixel in the orthoimage,

[0016] Obtaining, from the semantic segmentation result of the orthoimage, a probability that the pixel belongs to each of N preset semantic categories, where N is an integer greater than 1;

[0017] Determining the entropy of the pixel according to the probability that the pixel belongs to each of the N preset semantic categories;

[0018] According to the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

[0019] In a possible implementation, determining, according to the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality satisfies a preset low-quality condition includes:

[0020] Based on the entropy of each pixel in the orthoimage, generating a cross entropy map corresponding to the orthoimage;

[0021] Binarizing the cross entropy map to obtain a binary cross entropy map corresponding to the orthoimage;

[0022] Determining a connected domain in the binary cross entropy graph;

[0023] Determining a semantic segmentation quality metric value of the connected domain;

[0024] According to the semantic segmentation quality metric value of the connected domain, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

[0025] In a possible implementation, binarizing the cross entropy map to obtain a binarized cross entropy map corresponding to the orthoimage includes:

[0026] Determining a binarization threshold based on the entropy of pixels in the orthoimage;

[0027] Based on the binarization threshold, the cross entropy map is binarized to obtain a binarized cross entropy map corresponding to the orthoimage.

[0028] In a possible implementation manner, determining a binarization threshold based on entropy of pixels in the orthoimage includes:

[0029] The pixels in the orthoimage are sorted in descending order of the entropy of the pixels in the orthoimage, and the entropy of the qth pixel in the sorted pixels is determined as the binarization threshold; wherein q is a value calculated based on the total number of pixels of the orthoimage and a preset proportional coefficient.

[0030] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes an uncertainty metric value of the connected domain;

[0031] The determining of the semantic segmentation quality metric value of the connected domain includes:

[0032] An uncertainty measure value of the connected domain is determined according to entropy of pixels in the connected domain.

[0033] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes a diversity metric value of the connected domain;

[0034] The determining of the semantic segmentation quality metric value of the connected domain includes:

[0035] Determining the probability that the connected domain belongs to each of the N preset semantic categories according to the probability that the pixels in the connected domain belong to each of the N preset semantic categories;

[0036] The diversity metric value of the connected domain is determined according to the probability that the connected domain belongs to each of the N preset semantic categories.

[0037] In one possible implementation,

[0038] The method further comprises: projecting the three-dimensional model onto a two-dimensional plane using orthogonal projection to obtain an elevation image corresponding to the three-dimensional model;

[0039] The performing semantic segmentation on the orthophoto image to obtain the semantic segmentation result of the orthophoto image includes: performing semantic segmentation on the orthophoto image based on the elevation image to obtain the semantic segmentation result of the orthophoto image.

[0040] In a possible implementation, the performing semantic segmentation on the orthoimage based on the elevation image to obtain the semantic segmentation result of the orthoimage includes:

[0041] Performing feature extraction on the orthophoto image to obtain a first feature corresponding to the orthophoto image;

[0042] Performing feature extraction on the elevation image to obtain a second feature corresponding to the elevation image;

[0043] fusing the first feature and the second feature to obtain a fused feature;

[0044] Based on the fusion features, a semantic segmentation result of the orthoimage is obtained.

[0045] In a possible implementation manner, the fusing the first feature and the second feature to obtain a fused feature includes:

[0046] Based on the first feature and the second feature, obtaining a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image;

[0047] Based on the first feature and the first attention weight map, obtaining a first feature to be fused corresponding to the orthoimage;

[0048] Based on the second feature and the second attention weight map, obtaining a second feature to be fused corresponding to the elevation image;

[0049] A fusion feature is obtained based on the first feature to be fused and the second feature to be fused.

[0050] In a possible implementation, obtaining a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image based on the first feature and the second feature includes:

[0051] Performing attention learning based on the first feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image;

[0052] Performing attention learning based on the second feature through a second attention module to obtain a second transformation feature corresponding to the elevation image;

[0053] Splicing the first transformed feature and the second transformed feature to obtain a first spliced ​​feature;

[0054] Performing attention learning based on the first splicing feature through a third attention module to obtain a first attention weight map corresponding to the orthoimage;

[0055] Attention learning is performed based on the first splicing features through a fourth attention module to obtain a second attention weight map corresponding to the elevation image.

[0056] In one possible implementation,

[0057] The obtaining, based on the first feature and the first attention weight map, a first feature to be fused corresponding to the orthophoto image, comprises: obtaining, based on the first feature, the first attention weight map and a learnable first scale coefficient, a first feature to be fused corresponding to the orthophoto image;

[0058] The obtaining of the second feature to be fused corresponding to the elevation image based on the second feature and the second attention weight map includes: obtaining the second feature to be fused corresponding to the elevation image based on the second feature, the second attention weight map and a learnable second scale coefficient.

[0059] In one possible implementation,

[0060] The step of extracting features from the orthoimage to obtain a first feature corresponding to the orthoimage includes: extracting multiple levels of features from the orthoimage to obtain multiple levels of first features corresponding to the orthoimage;

[0061] The extracting features from the elevation image to obtain the second features corresponding to the elevation image comprises: extracting multiple levels of features from the elevation image to obtain multiple levels of second features corresponding to the elevation image;

[0062] The fusing the first feature and the second feature to obtain a fused feature comprises: fusing the multi-level first feature and the multi-level second feature respectively to obtain a multi-level fused feature;

[0063] The obtaining of the semantic segmentation result of the orthophoto image based on the fusion features includes: obtaining the semantic segmentation result of the orthophoto image based on the multi-level fusion features.

[0064] In a possible implementation, obtaining a semantic segmentation result of the orthoimage based on the multi-level fusion features includes:

[0065] Concatenate the first-level fusion features and the second-level fusion features to obtain the second-level concatenated features;

[0066] In the case of including three or more levels of fusion features, the k-1th level splicing feature and the kth level fusion feature are spliced ​​to obtain the kth level splicing feature, where k is an integer greater than or equal to 3;

[0067] Based on the last level of splicing features, a semantic segmentation result of the orthophoto image is obtained.

[0068] In a possible implementation, obtaining a semantic segmentation result of the orthoimage based on the fusion feature includes:

[0069] Based on the fusion feature, obtaining a probability that a pixel in the orthoimage belongs to each of the N preset semantic categories, where N is an integer greater than 1;

[0070] Based on the pixels in the orthoimage whose maximum probability is greater than the confidence threshold and the fusion features, determine the N category features corresponding to the N preset semantic categories; wherein the maximum probability corresponding to any pixel represents the maximum probability of the pixel belonging to each preset semantic category in the N preset semantic categories;

[0071] Based on the fusion features and the N category features, a semantic segmentation result of the orthoimage is obtained.

[0072] In a possible implementation manner, obtaining a semantic segmentation result of the orthoimage based on the fusion feature and the N category features includes:

[0073] Calculate similarity based on the fusion feature and the N category features to obtain a similarity matrix corresponding to the orthoimage;

[0074] Based on the similarity matrix and the N category features, obtaining enhanced features corresponding to the orthoimage;

[0075] Based on the fusion feature and the enhancement feature, an enhanced fusion feature corresponding to the orthoimage is obtained.

[0076] In a possible implementation, determining the semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthoimage and the correspondence between pixels in the orthoimage and three-dimensional primitives in the three-dimensional model includes:

[0077] For each pixel in the orthoimage, determining a set of three-dimensional primitives corresponding to the pixel in the three-dimensional model;

[0078] The three-dimensional primitives in the three-dimensional primitive set are traversed in order of height values ​​from large to small, and in response to the height difference between the traversed target three-dimensional primitive and the previous three-dimensional primitive of the target three-dimensional primitive being greater than a preset height threshold, the semantic label of the pixel in the updated semantic segmentation result of the orthophoto image is determined as the semantic label of the previous three-dimensional primitive and the three-dimensional primitive before the previous three-dimensional primitive, and the semantic label of the target three-dimensional primitive and the three-dimensional primitive after the target three-dimensional primitive is determined based on geometric consistency.

[0079] According to one aspect of the present disclosure, a semantic segmentation device for a three-dimensional model is provided, comprising:

[0080] A first projection module is used to project the three-dimensional model to be segmented onto a two-dimensional plane by orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model;

[0081] A semantic segmentation module, used to perform semantic segmentation on the orthophoto image to obtain a semantic segmentation result of the orthophoto image;

[0082] A first determination module is used to determine, based on the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition;

[0083] An acquisition module, used for acquiring the labeling data of the target area;

[0084] An updating module, used for updating the semantic segmentation result of the orthoimage based on the annotation data of the target area;

[0085] The second determination module is used to determine the semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthoimage and the correspondence between the pixels in the orthoimage and the three-dimensional primitives in the three-dimensional model.

[0086] In one possible implementation,

[0087] The semantic segmentation module is used to: perform semantic segmentation on the orthophoto image through a semantic segmentation network to obtain a semantic segmentation result of the orthophoto image;

[0088] The updating module is used to: update the parameters of the semantic segmentation network based on the annotation data of the target area and the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image; perform semantic segmentation on the orthophoto image through the semantic segmentation network with updated parameters to obtain an updated semantic segmentation result of the orthophoto image.

[0089] In a possible implementation manner, the first determining module is used to:

[0090] For each pixel in the orthoimage,

[0091] Obtaining, from the semantic segmentation result of the orthoimage, a probability that the pixel belongs to each of N preset semantic categories, where N is an integer greater than 1;

[0092] Determining the entropy of the pixel according to the probability that the pixel belongs to each of the N preset semantic categories;

[0093] According to the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

[0094] In a possible implementation manner, the first determining module is used to:

[0095] Based on the entropy of each pixel in the orthoimage, generating a cross entropy map corresponding to the orthoimage;

[0096] Binarizing the cross entropy map to obtain a binary cross entropy map corresponding to the orthoimage;

[0097] Determining a connected domain in the binary cross entropy graph;

[0098] Determining a semantic segmentation quality metric value of the connected domain;

[0099] According to the semantic segmentation quality metric value of the connected domain, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

[0100] In a possible implementation manner, the first determining module is used to:

[0101] Determining a binarization threshold based on the entropy of pixels in the orthoimage;

[0102] Based on the binarization threshold, the cross entropy map is binarized to obtain a binarized cross entropy map corresponding to the orthoimage.

[0103] In a possible implementation manner, the first determining module is used to:

[0104] The pixels in the orthoimage are sorted in descending order of the entropy of the pixels in the orthoimage, and the entropy of the qth pixel in the sorted pixels is determined as the binarization threshold; wherein q is a value calculated based on the total number of pixels of the orthoimage and a preset proportional coefficient.

[0105] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes an uncertainty metric value of the connected domain;

[0106] The first determining module is used for:

[0107] An uncertainty measure value of the connected domain is determined according to entropy of pixels in the connected domain.

[0108] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes a diversity metric value of the connected domain;

[0109] The first determining module is used for:

[0110] Determining the probability that the connected domain belongs to each of the N preset semantic categories according to the probability that the pixels in the connected domain belong to each of the N preset semantic categories;

[0111] The diversity metric value of the connected domain is determined according to the probability that the connected domain belongs to each of the N preset semantic categories.

[0112] In one possible implementation,

[0113] The device further comprises: a second projection module, configured to project the three-dimensional model onto a two-dimensional plane using orthogonal projection to obtain an elevation image corresponding to the three-dimensional model;

[0114] The semantic segmentation module is used to perform semantic segmentation on the orthoimage based on the elevation image to obtain a semantic segmentation result of the orthoimage.

[0115] In a possible implementation, the semantic segmentation module is used to:

[0116] Performing feature extraction on the orthophoto image to obtain a first feature corresponding to the orthophoto image;

[0117] Performing feature extraction on the elevation image to obtain a second feature corresponding to the elevation image;

[0118] fusing the first feature and the second feature to obtain a fused feature;

[0119] Based on the fusion features, a semantic segmentation result of the orthoimage is obtained.

[0120] In a possible implementation, the semantic segmentation module is used to:

[0121] Based on the first feature and the second feature, obtaining a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image;

[0122] Based on the first feature and the first attention weight map, obtaining a first feature to be fused corresponding to the orthoimage;

[0123] Based on the second feature and the second attention weight map, obtaining a second feature to be fused corresponding to the elevation image;

[0124] A fusion feature is obtained based on the first feature to be fused and the second feature to be fused.

[0125] In a possible implementation, the semantic segmentation module is used to:

[0126] Performing attention learning based on the first feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image;

[0127] Performing attention learning based on the second feature through a second attention module to obtain a second transformation feature corresponding to the elevation image;

[0128] Splicing the first transformed feature and the second transformed feature to obtain a first spliced ​​feature;

[0129] Performing attention learning based on the first splicing feature through a third attention module to obtain a first attention weight map corresponding to the orthoimage;

[0130] Attention learning is performed based on the first splicing features through a fourth attention module to obtain a second attention weight map corresponding to the elevation image.

[0131] In a possible implementation, the semantic segmentation module is used to:

[0132] Based on the first feature, the first attention weight map and a learnable first scale coefficient, obtaining a first feature to be fused corresponding to the orthoimage;

[0133] Based on the second feature, the second attention weight map and the learnable second scale coefficient, a second feature to be fused corresponding to the elevation image is obtained.

[0134] In a possible implementation, the semantic segmentation module is used to:

[0135] Performing multi-level feature extraction on the orthoimage to obtain multi-level first features corresponding to the orthoimage;

[0136] Performing multi-level feature extraction on the elevation image to obtain multi-level second features corresponding to the elevation image;

[0137] fusing the multi-level first features and the multi-level second features respectively to obtain multi-level fused features;

[0138] Based on the multi-level fusion features, a semantic segmentation result of the orthophoto is obtained.

[0139] In a possible implementation, the semantic segmentation module is used to:

[0140] The first-level fusion features and the second-level fusion features are spliced ​​to obtain the second-level splicing features;

[0141] In the case of including three or more levels of fusion features, the k-1th level splicing feature and the kth level fusion feature are spliced ​​to obtain the kth level splicing feature, where k is an integer greater than or equal to 3;

[0142] Based on the last level of splicing features, a semantic segmentation result of the orthophoto image is obtained.

[0143] In a possible implementation, the semantic segmentation module is used to:

[0144] Based on the fusion feature, obtaining a probability that a pixel in the orthoimage belongs to each of the N preset semantic categories, where N is an integer greater than 1;

[0145] Based on the pixels in the orthoimage whose maximum probability is greater than the confidence threshold and the fusion features, determine the N category features corresponding to the N preset semantic categories; wherein the maximum probability corresponding to any pixel represents the maximum probability of the pixel belonging to each preset semantic category in the N preset semantic categories;

[0146] Based on the fusion features and the N category features, a semantic segmentation result of the orthoimage is obtained.

[0147] In a possible implementation, the semantic segmentation module is used to:

[0148] Calculate similarity based on the fusion feature and the N category features to obtain a similarity matrix corresponding to the orthoimage;

[0149] Based on the similarity matrix and the N category features, obtaining enhanced features corresponding to the orthoimage;

[0150] Based on the fusion feature and the enhancement feature, an enhanced fusion feature corresponding to the orthoimage is obtained.

[0151] In a possible implementation manner, the second determining module is used to:

[0152] For each pixel in the orthoimage, determining a set of three-dimensional primitives corresponding to the pixel in the three-dimensional model;

[0153] The three-dimensional primitives in the three-dimensional primitive set are traversed in order of height values ​​from large to small, and in response to the height difference between the traversed target three-dimensional primitive and the previous three-dimensional primitive of the target three-dimensional primitive being greater than a preset height threshold, the semantic label of the pixel in the updated semantic segmentation result of the orthophoto image is determined as the semantic label of the previous three-dimensional primitive and the three-dimensional primitive before the previous three-dimensional primitive, and the semantic label of the target three-dimensional primitive and the three-dimensional primitive after the target three-dimensional primitive is determined based on geometric consistency.

[0154] According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0155] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

[0156] According to one aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0157] In the embodiment of the present disclosure, the three-dimensional model to be segmented is projected onto a two-dimensional plane by orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model, the orthophoto image is semantically segmented to obtain a semantic segmentation result of the orthophoto image, and based on the semantic segmentation result of the orthophoto image, a target area in the orthophoto image whose semantic segmentation quality meets a preset low-quality condition is determined, and the annotation data of the target area is obtained. Based on the annotation data of the target area, the semantic segmentation result of the orthophoto image is updated, and based on the updated semantic segmentation result of the orthophoto image and the correspondence between the pixels in the orthophoto image and the three-dimensional primitives in the three-dimensional model, the semantic segmentation result of the three-dimensional model is determined, thereby using the orthophoto image to perform semantic segmentation of the three-dimensional scene, replacing the traditional multi-view perspective projection, reducing the number of images to be processed, thereby improving the speed of semantic segmentation of the three-dimensional model, and realizing fast semantic segmentation of large-scale three-dimensional scenes. In addition, the embodiment of the present disclosure does not need to set complex internal and external parameters of the camera, providing greater convenience and improving operation efficiency.

[0158] In urban aerial photography scenes, 3D (3D) city scenes cover a wide area with many tall and dense buildings. Under perspective projection, these scenes are prone to serious occlusion problems, and usually require careful selection of camera angles. In contrast, orthographic projection images can only see the roofs of buildings and their surroundings, which can effectively avoid occlusion by tall buildings and other objects, and there is no need to set complex camera parameters. In this case, only a small number of orthographic images are needed to show the overall appearance of the entire scene.

[0159] In addition, experiments show that not every area in the high-resolution images of the aerial scene needs to be labeled. In the disclosed embodiment, an accurate area to be labeled (i.e., the target area) is generated based on the semantic segmentation result of the orthophoto image, and the semantic segmentation result of the orthophoto image is updated based on the labeled data, thereby improving the accuracy of the semantic segmentation result of the orthophoto image, thereby improving the accuracy of the semantic segmentation result of the three-dimensional model. Moreover, in the disclosed embodiment, it is only necessary to obtain the labeled data of the target area with poor semantic segmentation quality in the orthophoto image, and there is no need to obtain the labeled data of the entire orthophoto image, that is, the labeling personnel only need to specifically label the areas with poor semantic segmentation quality, thereby reducing the amount of labeled data, reducing the burden of labeling work, and saving time and labor costs.

[0160] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

[0161] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0162] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0163] Figure 1 A flowchart of a semantic segmentation method for a three-dimensional model provided by an embodiment of the present disclosure is shown.

[0164] Figure 2 A schematic diagram of a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided in an embodiment of the present disclosure is shown.

[0165] Figure 3 A schematic diagram of a second feature extraction module in an elevation image branch in a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided in an embodiment of the present disclosure is shown.

[0166] Figure 4A schematic diagram of a cross-modal feature aggregation module based on an attention mechanism in a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided in an embodiment of the present disclosure is shown.

[0167] Figure 5 A schematic diagram of a category-level context guidance module in a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided in an embodiment of the present disclosure is shown.

[0168] Figure 6 A schematic diagram showing the intermediate results of the active learning method in the semantic segmentation method of the three-dimensional model provided by an embodiment of the present disclosure.

[0169] Figure 7 A schematic diagram showing a method for determining semantic labels of three-dimensional primitives in a semantic segmentation method for a three-dimensional model provided by an embodiment of the present disclosure.

[0170] Figure 8 A block diagram of a semantic segmentation device for a three-dimensional model provided by an embodiment of the present disclosure is shown.

[0171] Fig. 9 A block diagram of an electronic device 1900 provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0172] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0173] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0174] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0175] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0176] In the field of 3D semantic segmentation of large-scale aerial photography scenes, methods based on deep neural networks are the current mainstream research direction. These methods can be divided into two processing ideas: based on 3D deep neural networks and based on multi-view 2D images. The method based on 3D deep neural networks directly performs semantic classification on each 3D point. Since it can make full use of the geometric structure information of the 3D points, it has achieved relatively more accurate results. However, this method requires a large amount of reliable and effective 3D annotation data for supervision and training, and obtaining these 3D annotation data is usually very time-consuming and labor-intensive. Another processing idea is the method based on multi-view 2D images. This type of method first uses the image semantic segmentation network to perform semantic segmentation on the 2D image, and then back-projects these segmentation results onto the 3D model for fusion. When the original acquired images of the 3D scene are provided, researchers usually process them on these ready-made images. However, due to factors such as occlusion, illumination changes, and the presence of dynamic objects, the 3D model and some areas of the 2D image are not strictly corresponding, resulting in a certain gap in the performance of this type of method compared with the current benchmark method. Therefore, many scholars turn to using perspective projection to generate new multi-view images from the 3D model. However, in order to make the image collection cover the entire three-dimensional scene as much as possible, they all render a large number of two-dimensional multi-view images from multiple positions, which leads to data redundancy and increases the amount of calculation and processing time required.

[0177] In order to solve the technical problems similar to those described above, the embodiment of the present disclosure provides a semantic segmentation method for a three-dimensional model, by using orthogonal projection to project the three-dimensional model to be segmented onto a two-dimensional plane, obtaining an orthophoto image corresponding to the three-dimensional model, performing semantic segmentation on the orthophoto image, obtaining a semantic segmentation result of the orthophoto image, determining a target area in the orthophoto image whose semantic segmentation quality meets a preset low-quality condition based on the semantic segmentation result of the orthophoto image, obtaining the annotation data of the target area, updating the semantic segmentation result of the orthophoto image based on the annotation data of the target area, and determining the semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthophoto image and the correspondence between the pixels in the orthophoto image and the three-dimensional primitives in the three-dimensional model, thereby using the orthophoto image to perform semantic segmentation of the three-dimensional scene, replacing the traditional multi-view perspective projection, reducing the number of images to be processed, thereby improving the speed of semantic segmentation of the three-dimensional model, and realizing fast semantic segmentation of large-scale three-dimensional scenes. In addition, the embodiment of the present disclosure does not need to set complex internal and external parameters of the camera, providing greater convenience and improving operation efficiency.

[0178] In urban aerial photography scenes, 3D (3D) city scenes cover a wide area with many tall and dense buildings. Under perspective projection, these scenes are prone to serious occlusion problems, and usually require careful selection of camera angles. In contrast, orthographic projection images can only see the roofs of buildings and their surroundings, which can effectively avoid occlusion by tall buildings and other objects, and there is no need to set complex camera parameters. In this case, only a small number of orthographic images are needed to show the overall appearance of the entire scene.

[0179] In addition, experiments show that not every area in the high-resolution images of the aerial scene needs to be labeled. In the disclosed embodiment, an accurate area to be labeled (i.e., the target area) is generated based on the semantic segmentation result of the orthophoto image, and the semantic segmentation result of the orthophoto image is updated based on the labeled data, thereby improving the accuracy of the semantic segmentation result of the orthophoto image, thereby improving the accuracy of the semantic segmentation result of the three-dimensional model. Moreover, in the disclosed embodiment, it is only necessary to obtain the labeled data of the target area with poor semantic segmentation quality in the orthophoto image, and there is no need to obtain the labeled data of the entire orthophoto image, that is, the labeling personnel only need to specifically label the areas with poor semantic segmentation quality, thereby reducing the amount of labeled data, reducing the burden of labeling work, and saving time and labor costs.

[0180] The semantic segmentation method of the three-dimensional model provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

[0181] Figure 1 A flow chart of a method for semantic segmentation of a three-dimensional model provided in an embodiment of the present disclosure is shown. In one possible implementation, the executor of the method for semantic segmentation of a three-dimensional model may be a semantic segmentation apparatus for a three-dimensional model. For example, the method for semantic segmentation of a three-dimensional model may be executed by a terminal device or a server or other electronic device. The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the method for semantic segmentation of a three-dimensional model may be implemented by a processor calling computer-readable instructions stored in a memory. As Figure 1 As shown, the semantic segmentation method of the three-dimensional model includes steps S11 to S16.

[0182] In step S11, the three-dimensional model to be segmented is projected onto a two-dimensional plane using orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model.

[0183] In step S12, semantic segmentation is performed on the orthoimage to obtain a semantic segmentation result of the orthoimage.

[0184] In step S13, based on the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined;

[0185] In step S14, obtaining the labeling data of the target area;

[0186] In step S15, based on the annotation data of the target area, the semantic segmentation result of the orthoimage is updated;

[0187] In step S16, based on the updated semantic segmentation result of the orthoimage and the correspondence between the pixels in the orthoimage and the three-dimensional primitives in the three-dimensional model, the semantic segmentation result of the three-dimensional model is determined.

[0188] In the embodiments of the present disclosure, the 3D model to be segmented may be any 3D model to be subjected to 3D semantic segmentation. The 3D model may be a 3D model with texture. For example, the 3D model may be a textured mesh model; for another example, the 3D model may be a point cloud model with color; and so on. 3D semantic segmentation may refer to assigning a semantic category with practical significance to each 3D primitive (such as a 3D point or a triangular facet) in the 3D model.

[0189] In the embodiment of the present disclosure, for the three-dimensional model to be segmented, the orthogonal projection method can be used to project the three-dimensional model onto a two-dimensional plane to obtain an orthophoto image corresponding to the three-dimensional model. The orthophoto image is a two-dimensional image. For each three-dimensional model sub-block, regardless of its size, orthogonal projection can be used to compress it onto a two-dimensional plane. Compared with the multi-view image method in the related art, the number of projected images in the embodiment of the present disclosure is significantly reduced, thereby improving the efficiency of semantic segmentation.

[0190] In one example, the 3D model to be segmented can be represented by M, and the bounding box of the 3D model can be represented by the minimum point p min =(x min ,y min ,z min ) and the maximum point p max =(x max ,y max ,z max ) to represent it.

[0191] In a possible implementation, the orthophoto image may be an RGB (Red-Green-Blue) image.

[0192] In a possible implementation, the pixel value of any pixel in the orthoimage may be determined according to the pixel value of the 3D primitive with the largest height among the corresponding 3D primitives. That is, among all the 3D primitives projected to the same image plane coordinates, only the color information of the 3D primitive with the largest height may be retained in the orthoimage.

[0193] In one example, the orthoimage can be denoted by I∈R 3×H×W . Where H and W can represent the height and width of the orthoimage respectively.

[0194] In one example, the three-dimensional model is a point cloud model. In the case where the three-dimensional model is a point cloud model, for a point p in the three-dimensional model i ∈M, its coordinates (row, col) on the image plane can be determined according to Formula 1:

[0195]

[0196] Among them, x i Represents a 3D point p i x coordinates, y coordinates in three-dimensional space i Represents a 3D point p i The y coordinate in the three-dimensional space, s represents the actual physical width of a single pixel in the orthoimage in the three-dimensional space.

[0197] In one example, the 3D model is a texture mesh model, which is composed of tightly connected triangular facets. In the case where the 3D model is a texture mesh model, the projection of each triangular facet on the image plane is a triangular area formed by the aggregation of pixels. For each image pixel, a ray parallel to the z-axis can be emitted from the pixel, and the intersection point with the 3D model can be calculated using the CGAL tool library, and finally the information of the intersection point is assigned to the current image pixel.

[0198] In the embodiment of the present disclosure, when the three-dimensional model is a point cloud model, the three-dimensional primitives in the three-dimensional model are three-dimensional points; when the three-dimensional model is a textured mesh model, the three-dimensional primitives in the three-dimensional model are triangular faces.

[0199] In an embodiment of the present disclosure, semantic segmentation may be performed on the orthoimage to obtain a semantic segmentation result of the orthoimage. The semantic segmentation result of the orthoimage may include a probability that each pixel in the orthoimage belongs to each of the N preset semantic categories, where N is an integer greater than 1. For example, the N preset semantic categories may include buildings, roads, pedestrians, vehicles, and the like.

[0200] In a possible implementation, the method further includes: projecting the three-dimensional model onto a two-dimensional plane using orthogonal projection to obtain an elevation image corresponding to the three-dimensional model; performing semantic segmentation on the orthophoto image to obtain a semantic segmentation result of the orthophoto image, including: performing semantic segmentation on the orthophoto image based on the elevation image to obtain a semantic segmentation result of the orthophoto image.

[0201] In this implementation, an elevation image corresponding to the three-dimensional model can also be obtained through orthogonal projection. In one example, the elevation image can be recorded as E∈R H×W Wherein, H and W may represent the height and width of the elevation image, respectively. The height H of the elevation image may be equal to the height H of the orthophoto image, and the width W of the elevation image may be equal to the width W of the orthophoto image.

[0202] For elevation images, it is not necessary to keep the exact height values ​​in 3D space, but only to understand the relative height differences. For example, the roof of a building is higher than the ground, and the green space on the ground is lower than the tall vegetation. When the entire scene cannot be processed at once due to computing power limitations, each sub-block in the 3D model can be processed separately. For example, the height of each sub-block in the 3D model can be normalized and saved as an 8-bit image. In an elevation image, the height is z i The gray value gv of the three-dimensional point on the elevation image i It can be determined using Formula 2:

[0203]

[0204] The height value of any pixel in the elevation image can be determined according to the height value of the three-dimensional primitive with the largest height among the corresponding three-dimensional primitives. That is, among all the three-dimensional primitives projected to the same image plane coordinates, only the height value of the three-dimensional primitive with the largest height can be retained in the elevation image.

[0205] As an example of this implementation, the orthophoto image and the elevation image can be input into a semantic segmentation network, and the semantic segmentation network performs semantic segmentation based on the orthophoto image and the elevation image to obtain a semantic segmentation result of the orthophoto image. The semantic segmentation result of the orthophoto image can also be referred to as a semantic segmentation result of the orthophoto image and the elevation image or a semantic segmentation result of the two-dimensional image.

[0206] The elevation image carries complementary information of the RGB texture. In this implementation, the elevation image corresponding to the three-dimensional model is obtained by projecting the three-dimensional model onto a two-dimensional plane using orthogonal projection, and the orthophoto image is semantically segmented based on the elevation image to obtain the semantic segmentation result of the orthophoto image. Thus, the elevation information is used as a supplement when the feature difference of the orthophoto image is not significant, and the semantic segmentation network is guided to learn more discriminative features of different categories, so that objects with similar texture appearance can be distinguished more accurately, and the accuracy of semantic segmentation can be further improved. For example, the segmentation accuracy of the ground and building roofs, green areas on the ground and tall trees can be improved.

[0207] In a possible implementation, when the 3D model is a point cloud model and the point cloud model is sparse, there may be many blank pixels in the generated orthoimage and elevation image, and these blank pixels have no corresponding 3D points in the 3D model. In this case, the pixel values ​​and height values ​​of these pixels in the orthoimage and elevation image can be initialized to 0, and their semantic labels can be set to "ignore".

[0208] In a possible implementation, the orthophoto image and the elevation image can be processed by a semantic segmentation network to obtain a semantic segmentation result of the orthophoto image. When the sizes of the orthophoto image and the elevation image are large, the orthophoto image and the elevation image can be cropped respectively. The cropped orthophoto images do not overlap each other, and the cropped orthophoto images can be spliced ​​to obtain the orthophoto image before cropping. The size (width and height) of the cropped orthophoto image is smaller than the size of the orthophoto image before cropping, and the PPI (Pixels Per Inch) of the orthophoto images before and after cropping is the same. The cropped elevation images do not overlap each other, and the cropped elevation images can be spliced ​​to obtain the elevation image before cropping. The size (width and height) of the cropped elevation image is smaller than the size of the elevation image before cropping, and the PPI of the elevation images before and after cropping is the same. The width of the image can represent the number of pixels in the horizontal direction of the image, and the height of the image represents the number of pixels in the vertical direction of the image. The cropping can be performed by uniform equal division, and the whole process does not involve downsampling or information loss. After the cropped image is semantically segmented by the semantic segmentation network, it can be restored to its original size by stitching.

[0209] In a possible implementation, the semantic segmentation of the orthophoto image is performed based on the elevation image to obtain the semantic segmentation result of the orthophoto image, including: extracting features from the orthophoto image to obtain a first feature corresponding to the orthophoto image; extracting features from the elevation image to obtain a second feature corresponding to the elevation image; fusing the first feature and the second feature to obtain a fused feature; and obtaining the semantic segmentation result of the orthophoto image based on the fused feature.

[0210] In this implementation, the first feature may represent a feature obtained by extracting features from the orthoimage, and the second feature may represent a feature obtained by extracting features from the elevation image. That is, the first feature may represent a feature corresponding to the orthoimage, and the second feature may represent a feature corresponding to the elevation image.

[0211] The fused feature may refer to a feature obtained by fusing the first feature and the second feature.

[0212] As an example of this implementation, the first feature corresponding to the orthoimage and the second feature corresponding to the elevation image can be fused through a cross-modality-feature aggregation module (CM-FAM) in the semantic segmentation network.

[0213] In this implementation, feature extraction is performed on the orthoimage to obtain a first feature corresponding to the orthoimage, and feature extraction is performed on the elevation image to obtain a second feature corresponding to the elevation image; the first feature and the second feature are fused to obtain a fused feature, and based on the fused feature, a semantic segmentation result of the orthoimage is obtained. Thus, the features of the two modalities (the first feature corresponding to the orthoimage and the second feature corresponding to the elevation image) are fused, which can assist the semantic segmentation network in learning more discriminative features.

[0214] In a possible implementation, the extracting features from the orthoimage to obtain a first feature corresponding to the orthoimage includes: performing multi-level feature extraction on the orthoimage to obtain multi-level first features corresponding to the orthoimage; the extracting features from the elevation image to obtain a second feature corresponding to the elevation image includes: performing multi-level feature extraction on the elevation image to obtain multi-level second features corresponding to the elevation image; the fusing the first feature and the second feature to obtain a fused feature includes: fusing the multi-level first feature and the multi-level second feature respectively to obtain a multi-level fused feature; and obtaining a semantic segmentation result of the orthoimage based on the fused feature includes: obtaining a semantic segmentation result of the orthoimage based on the multi-level fused feature.

[0215] Figure 2 A schematic diagram of a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided by an embodiment of the present disclosure is shown. Figure 2 As shown, the input of the semantic segmentation network can include orthophotos (in Figure 2 denoted as “RGB image”) and elevation image.

[0216] exist Figure 2 In the example shown, the four first feature extraction modules corresponding to the orthophoto image ( Figure 2 Layer_1~4) above performs 4-level feature extraction on the orthophoto image to obtain the 4-level first features corresponding to the orthophoto image. Among them, the 4-level first features are Figure 2 Output of Layer_1~4 above. Figure 2 The output of Layer_1 above can be recorded as the first level first feature, Figure 2 The output of Layer_2 above can be recorded as the first feature of the second level. Figure 2 The output of Layer_3 above can be recorded as the first feature of the third level. Figure 2 The output of Layer_4 above can be recorded as the first feature of the fourth level.

[0217] The four second feature extraction modules corresponding to the elevation image can be used ( Figure 2 The Layer_1~4 below performs 4-level feature extraction on the elevation image to obtain 4-level second features corresponding to the elevation image. Among them, the 4-level second features are Figure 2 The output of Layer_1 to 4 below. Figure 2 The output of Layer_1 below can be recorded as the first-level second feature. Figure 2 The output of Layer_2 below can be recorded as the second level second feature, Figure 2 The output of Layer_3 below can be recorded as the second feature of the third level. Figure 2 The output of Layer_4 below can be recorded as the second feature of the fourth level.

[0218] The first level first feature and the first level second feature can be fused to obtain the first level fusion feature (i.e. Figure 2 The output of the first CM-FAM in the first CM-FAM); the second-level first feature and the second-level second feature can be fused to obtain the second-level fusion feature (i.e. Figure 2 The output of the second CM-FAM in the third level); the third level first feature and the third level second feature can be fused to obtain the third level fusion feature (i.e. Figure 2 The output of the third CM-FAM in the fourth level); the fourth level first feature and the fourth level second feature can be fused to obtain the fourth level fusion feature (i.e. Figure 2 output of the fourth CM-FAM in ).

[0219] The j-th level fusion feature may represent the feature obtained by fusing the j-th level first feature and the j-th level second feature, 1≤j≤J, and J represents the total level of the fusion feature. In one example, the total level of the first feature = the total level of the second feature = the total level of the fusion feature. Figure 2 , J=4.

[0220] In this implementation, multi-level feature extraction is performed on the orthoimage to obtain multi-level first features corresponding to the orthoimage, and multi-level feature extraction is performed on the elevation image to obtain multi-level second features corresponding to the elevation image. The multi-level first features and the multi-level second features are fused respectively to obtain multi-level fused features, and based on the multi-level fused features, the semantic segmentation result of the orthoimage is obtained, thereby making full use of the shallow and deep visual information of the orthoimage and the elevation image for semantic segmentation, which is conducive to obtaining more accurate semantic segmentation results.

[0221] exist Figure 2 In the example shown, the feature extraction backbone network in the semantic segmentation network includes an orthophoto branch and an elevation image branch. The orthophoto branch may include five first feature extraction modules, and the elevation image branch may include five second feature extraction modules. In addition to the five first feature extraction modules, Figure 2 Layer_1~4 above can also include settings Figure 2 Layer_0 of Layer_1 above (in Figure 2 The 5 second feature extraction modules include Figure 2 Layer_1 to Layer_4 below can also include Figure 2 Layer_0 of Layer_1 below (in Figure 2 (not shown in FIG. 1 ). In this example, the total number of levels of the first features = the total number of levels of the second features > the total number of levels of the fused features.

[0222] In another example, the total number of levels of the first features>the total number of levels of the second features>the total number of levels of the fused features.

[0223] In one example, the first feature extraction module for feature extraction of the orthophoto image can adopt the ResNet50 architecture. The first feature extraction module can follow the original design of the ResNet50 architecture, including a 7×7 convolution layer, and then reduce the input image size to 1 / 4 of the original size through a maximum pooling layer with a stride of 2. The ResNet50 network can be composed of four residual blocks, each of which contains multiple convolution layers and jump connections. Except for the first block, the feature output sizes of the following three blocks are reduced to half of the original size.

[0224] In one example, the second feature extraction module for extracting features from the elevation image may adopt a relatively simplified residual structure to reduce the computational complexity of the semantic segmentation network. Figure 3 A schematic diagram of a second feature extraction module in an elevation image branch in a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided by an embodiment of the present disclosure is shown. Figure 3 In , ConvA can be a 3×3 convolution with a stride of 2, and ConvB can be a 3×3 convolution with a stride of 1. The outputs of both convolutional layers are processed by batch normalization (BN) and ReLU activation function. Therefore, the size of the output features of each second feature extraction module is reduced by half. Finally, the second to fifth second feature extraction modules ( Figure 2 The feature maps output by Layer_1~4) below are fused with the output of the orthophoto branch.

[0225] In a possible implementation, obtaining the semantic segmentation result of the orthophoto image based on the multi-level fusion features includes: splicing the first-level fusion features and the second-level fusion features to obtain the second-level splicing features; in the case of including three or more levels of fusion features, splicing the k-1-th level splicing features and the k-th level fusion features to obtain the k-th level splicing features, where k is an integer greater than or equal to 3; and obtaining the semantic segmentation result of the orthophoto image based on the last level splicing features.

[0226] As an example of this implementation method, before the k-1th level splicing feature and the kth level fusion feature are spliced, the kth level fusion feature can be upsampled so that the dimension of the upsampled kth level fusion feature is the same as the dimension of the k-1th level splicing feature.

[0227] In some application scenarios, the concatenation operation may also be referred to as a cascade operation or a merge operation, which is not limited here.

[0228] For example, in Figure 2 In the example shown, the first splicing operation module ( Figure 2 The first C in the first level fusion feature (i.e. Figure 2 The output of the first CM-FAM in the Figure 2 The output of the second CM-FAM in the Figure 2 The output of the first C in ). The second splicing operation module ( Figure 2 The second C in the second level concatenation feature and the third level fusion feature (i.e. Figure 2 The output of the third CM-FAM in the above example is concatenated to obtain the third-level concatenated features (i.e. Figure 2 The output of the second C in the example). The third splicing operation module ( Figure 2 The third C in the third level concatenation feature and the fourth level fusion feature (i.e. Figure 2 The output of the fourth CM-FAM in is concatenated to obtain the fourth-level concatenated features (i.e. Figure 2 The output of the third C in ). Based on the fourth-level splicing features, the semantic segmentation result of the orthophoto image can be obtained.

[0229] In this implementation, the second-level splicing features are obtained by splicing the first-level fusion features and the second-level fusion features. In the case of three or more levels of fusion features, the k-1-level splicing features and the k-level fusion features are spliced ​​to obtain the k-level splicing features, wherein k is an integer greater than or equal to 3, and based on the last-level splicing features, the semantic segmentation result of the orthophoto image is obtained, thereby making it possible to utilize different levels of features of the two modalities (orthophoto image and elevation image) to obtain a more accurate semantic segmentation result.

[0230] In a possible implementation, the first feature and the second feature are fused to obtain a fused feature, including: based on the first feature and the second feature, obtaining a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image; based on the first feature and the first attention weight map, obtaining a first feature to be fused corresponding to the orthophoto image; based on the second feature and the second attention weight map, obtaining a second feature to be fused corresponding to the elevation image; and obtaining a fused feature based on the first feature to be fused and the second feature to be fused.

[0231] As an example of this implementation, a cross-modal feature aggregation module based on an attention mechanism (see Figure 2CM-FAM in the method, based on the first feature and the second feature, a first attention weight map corresponding to the orthoimage and a second attention weight map corresponding to the elevation image are obtained, based on the first feature and the first attention weight map, a first feature to be fused corresponding to the orthoimage is obtained, based on the second feature and the second attention weight map, a second feature to be fused corresponding to the elevation image is obtained, and based on the first feature to be fused and the second feature to be fused, a fused feature is obtained.

[0232] Figure 4 A schematic diagram of a cross-modal feature aggregation module based on an attention mechanism in a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided by an embodiment of the present disclosure is shown. Figure 4 Medium, F rgb It can represent the first feature, F ele The second feature, F fuse The first feature F can be expressed as rgb and the second feature F ele fusion characteristics.

[0233] In one example, the first attention weight map can be W rgb ∈R 1×H×W Indicates that the second attention weight map can be adopted W ele ∈R 1×H×W express.

[0234] As an example of this implementation method, the first feature can be multiplied by the first attention weight map to obtain the first feature to be fused corresponding to the orthophoto image; the second feature can be multiplied by the second attention weight map to obtain the second feature to be fused corresponding to the elevation image.

[0235] In an example, the first feature to be fused may also be referred to as an enhanced first feature, and the second feature to be fused may also be referred to as an enhanced second feature, which is not limited here.

[0236] In this implementation, a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image are obtained based on the first feature and the second feature, a first feature to be fused corresponding to the orthophoto image is obtained based on the first feature and the first attention weight map, a second feature to be fused corresponding to the elevation image is obtained based on the second feature and the second attention weight map, and a fused feature is obtained based on the first feature to be fused and the second feature to be fused. Thus, a cross-modal feature aggregation module based on the attention mechanism is adopted to replace the traditional method of directly extracting features or extracting features separately and then superimposing them, which can better capture the shape and structure information of the object, assist the semantic segmentation network to learn more discriminative representations, and improve the accuracy of semantic segmentation.

[0237] In a possible implementation, the first attention weight map corresponding to the orthophoto image and the second attention weight map corresponding to the elevation image are obtained based on the first feature and the second feature, including: performing attention learning based on the first feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image; performing attention learning based on the second feature through a second attention module to obtain a second transformation feature corresponding to the elevation image; splicing the first transformation feature and the second transformation feature to obtain a first splicing feature; performing attention learning based on the first splicing feature through a third attention module to obtain a first attention weight map corresponding to the orthophoto image; performing attention learning based on the first splicing feature through a fourth attention module to obtain a second attention weight map corresponding to the elevation image.

[0238] For example, in Figure 4 In the example, the first attention module may be the ConvA module above the concatenation operation module (C) on the left, and the second attention module may be the ConvA module below the concatenation operation module (C) on the left. The ConvA module may include a 3×3 convolution layer, a batch normalization (BN) layer, and a ReLU activation layer.

[0239] In this implementation, the first attention module can perform attention learning based on the first feature to obtain the first change feature corresponding to the orthophoto image ( Figure 4 The output of the ConvA module above the splicing operation module (C) on the left side of the middle); the second transformation feature corresponding to the elevation image can be obtained by performing attention learning based on the second feature through the second attention module ( Figure 4 The output of the ConvA module below the concatenation operation module (C) on the left in the figure).

[0240] In this implementation, the first transformation feature and the second transformation feature may be spliced ​​along the channel dimension to obtain a first spliced ​​feature. Figure 4 The splicing operation module (C) on the left side of the figure splices the first transformation feature and the second transformation feature to obtain a first splicing feature. In one example, the first splicing feature can be obtained by using F concat express.

[0241] exist Figure 4 In the example, the third attention module can be the upper ConvA+ConvD+Sigmoid module, and the fourth attention module can be the lower ConvA+ConvD+Sigmoid module. ConvD can be a 1×1 convolution layer. The third attention module can be used to perform attention learning based on the first splicing feature to obtain the first attention weight map corresponding to the orthophoto image ( Figure 4 The output of the ConvA+ConvD+Sigmoid module above); the fourth attention module can be used to perform attention learning based on the first splicing feature to obtain the second attention weight map corresponding to the elevation image ( Figure 4 The output of the ConvA+ConvD+Sigmoid module below).

[0242] Taking the orthophoto branch as an example, the processing method of the elevation image branch is similar. In the orthophoto branch, the first splicing feature F is obtained after splicing. concat After that, we can first use the ConvA module containing a 3×3 convolutional layer to concatenate the first feature F concat Processing is performed to fuse the features of the two modalities, and then the output channel is set to 1 through a 1×1 convolutional layer (ConvD), and finally normalized by the Sigmoid function to obtain the first attention weight map.

[0243] For pixels at different positions on the image, their requirements for the two modal features are also different. In this implementation, the information transfer rate of the first feature and the second feature can be regulated based on the first attention weight map and the second attention weight map, thereby improving the accuracy of semantic segmentation of the three-dimensional model.

[0244] In a possible implementation, performing attention learning based on the first feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image includes: performing dimensionality reduction on the first feature to obtain a first reduced dimensionality feature corresponding to the orthophoto image; performing attention learning based on the first reduced dimensionality feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image.

[0245] refer to Figure 4 , the first feature F can be rgb ∈R C×H×W Perform dimensionality reduction to obtain the first dimensionality reduction feature F′ corresponding to the orthophoto image rgb Among them, the ConvB module may include a 1×1 convolution layer, a batch normalization (BN) layer, and a ReLU activation layer.

[0246] In one example, the first transformation feature can be convA(F′ rgb ) indicates that the second transformation feature can be represented by convA(F ele ) indicates that the first splicing feature F concat =concat(convA(F′ rgb ),convA(F ele )).

[0247] In one example, the first dimension reduction feature ( Figure 4 The output of the ConvB module in ( Figure 4 The output of the ConvA+ConvD+Sigmoid module above) is multiplied to obtain the first feature to be fused corresponding to the orthophoto image ( Figure 4 The output of the multiplication module above); the second feature (F ele ) and the second attention weight map ( Figure 4 The output of the ConvA+ConvD+Sigmoid module below) is multiplied to obtain the second feature to be fused corresponding to the elevation image ( Figure 4 The output of the multiplication module below).

[0248] In this implementation, by reducing the dimension of the first feature, a first dimensionality reduction feature corresponding to the orthophoto image is obtained, and by performing attention learning based on the first dimensionality reduction feature through a first attention module, a first transformation feature corresponding to the orthophoto image is obtained, thereby reducing the amount of calculation and improving the speed of semantic segmentation of the three-dimensional model.

[0249] In a possible implementation, obtaining the first feature to be fused corresponding to the orthophoto image based on the first feature and the first attention weight map includes: obtaining the first feature to be fused corresponding to the orthophoto image based on the first feature, the first attention weight map and a learnable first scale coefficient; obtaining the second feature to be fused corresponding to the elevation image based on the second feature and the second attention weight map includes: obtaining the second feature to be fused corresponding to the elevation image based on the second feature, the second attention weight map and a learnable second scale coefficient.

[0250] In this implementation, the first scale factor and the second scale factor may be updated during the training process of the semantic segmentation network. In one example, the first scale factor may be represented by α, and the second scale factor may be represented by β.

[0251] In one example, the first feature to be fused may be α×W rgb ×F′ rgb , the second feature to be fused can be β×W ele ×F ele .

[0252] In this implementation, by introducing a learnable first scale coefficient and a second scale coefficient, scaling of the first feature and the second feature can be achieved.

[0253] In a possible implementation, obtaining a fused feature based on the first feature to be fused and the second feature to be fused includes: splicing the first feature to be fused and the second feature to be fused to obtain a second spliced ​​feature; and obtaining a fused feature based on the second spliced ​​feature.

[0254] For example, you can use Figure 4 The splicing operation module (C) on the right performs the first feature to be fused ( Figure 4 The output of the multiplication module above) and the second feature to be fused ( Figure 4 The output of the multiplication module below) is spliced ​​to obtain the second splicing feature ( Figure 4 The output of the splicing operation module (C) on the right).

[0255] In one example, the second concatenation feature may be concat(α×W rgb ×F′ rgb ,β×W ele ×F ele ).

[0256] As an example of this implementation, the second concatenated feature can be upsampled to obtain a fused feature. For example, the fused feature F fuse =UP(concat(α*W rgb *F′ rgb ,β*W ele *F ele )). Wherein, UP indicates upsampling the second concatenated feature to the same size as the first feature.

[0257] As another example of this implementation, the second splicing feature may be used as a fusion feature.

[0258] By adopting this implementation method, the semantic segmentation network can be assisted to learn more discriminative representations, thereby improving the accuracy of semantic segmentation of three-dimensional models.

[0259] In a possible implementation, obtaining the semantic segmentation result of the orthoimage based on the fusion feature includes: obtaining the probability that a pixel in the orthoimage belongs to each of the N preset semantic categories based on the fusion feature, wherein N is an integer greater than 1; determining N category features corresponding to the N preset semantic categories based on pixels in the orthoimage with a maximum probability greater than a confidence threshold and the fusion feature; wherein the maximum probability corresponding to any pixel represents the maximum probability among the probabilities that the pixel belongs to each of the N preset semantic categories; obtaining the semantic segmentation result of the orthoimage based on the fusion feature and the N category features.

[0260] As an example of this implementation, in the case of multi-level feature extraction of orthophoto images and elevation images, the last level of fusion features (e.g. Figure 2 The last level of concatenation features (e.g., the output of the rightmost CM-FAM in Figure 2 The output of the rightmost C in the orthophoto image is obtained to obtain the probability that the pixel in the orthophoto image belongs to each of the N preset semantic categories. In one example, the last level of splicing features can be input into a context guidance module based on category features (CF-CGM), and the probability that the pixel in the orthophoto image belongs to each of the N preset semantic categories is obtained through the context guidance module.

[0261] Figure 5 A schematic diagram of a category-level context guidance module in a semantic segmentation network in a semantic segmentation method for a three-dimensional model provided in an embodiment of the present disclosure is shown. The category-level context guidance module may also be referred to as a context guidance module based on category features, etc., which is not limited here. In scene semantic segmentation tasks, objects of the same category usually have higher feature similarity, while the similarity between different categories is usually lower. Based on this observation, an embodiment of the present disclosure proposes a category-level context guidance module. Figure 5 As shown, the last level of splicing features (in Figure 5 The pixel features are input into the category-level context guidance module, and the probability of the pixel in the orthophoto image belonging to each of the N preset semantic categories is obtained by the category-level context guidance module (in Figure 5 In one example, the category-level context guidance module may include a classification convolution module. The input of the classification convolution module may be the last level of splicing features, and the output may be the probability that a pixel in the orthoimage belongs to each of the N preset semantic categories. The classification convolution module may use a 3×3 convolution to map pixel features to preset semantic categories, and may use a softmax normalization function to obtain the probability that each pixel belongs to a different preset semantic category. For example, the probability that a pixel at position (r, c) belongs to each of the N preset semantic categories may be expressed as:

[0262] A confidence threshold can be set, and N category features corresponding to N preset semantic categories can be obtained based on the pixels in the orthoimage whose maximum probability is greater than the confidence threshold and the last level of stitching features. The N category features correspond one-to-one to the N preset semantic categories. If the maximum probability of any pixel belonging to each of the N preset semantic categories is greater than the confidence threshold, the pixel can be used to participate in the calculation of the category feature; if the maximum probability of any pixel belonging to each of the N preset semantic categories is less than or equal to the preset height threshold, the pixel can be filtered out, that is, the pixel is not considered in the calculation of the category feature.

[0263] In one example, for any preset semantic category, the features of the target pixel in the last level of splicing features can be averaged to obtain the category features corresponding to the preset semantic category. The target pixel can represent a pixel whose maximum probability is greater than a confidence threshold and whose maximum probability corresponds to the preset semantic category. The maximum probability corresponding to any pixel corresponds to a pixel of the preset semantic category, which can indicate that among the probabilities of the pixel belonging to each of the N preset semantic categories, the probability of the pixel belonging to the preset semantic category is the highest.

[0264] In one example, the confidence threshold can be δ c express.

[0265] In one example, F cate ∈R C×N It can represent N category features, where C represents the fusion feature F fuse Same dimensions. cate In , each column is a feature vector of a preset semantic category.

[0266] In this implementation, based on the fusion feature, the probability that the pixel in the orthophoto image belongs to each of the N preset semantic categories is obtained, wherein N is an integer greater than 1, and based on the pixels in the orthophoto image whose maximum probability is greater than the confidence threshold and the fusion feature, N category features corresponding to the N preset semantic categories are determined, wherein the maximum probability corresponding to any pixel represents the maximum probability of the pixel belonging to each of the N preset semantic categories, and based on the fusion feature and the N category features, the semantic segmentation result of the orthophoto image is obtained, thereby improving the accuracy of semantic segmentation of the three-dimensional model.

[0267] In a possible implementation, the semantic segmentation result of the orthophoto image is obtained based on the fusion feature and the N category features, including: calculating the similarity based on the fusion feature and the N category features to obtain a similarity matrix corresponding to the orthophoto image; obtaining an enhanced feature corresponding to the orthophoto image based on the similarity matrix and the N category features; and obtaining an enhanced fusion feature corresponding to the orthophoto image based on the fusion feature and the enhanced feature.

[0268] refer to Figure 5 , the last level splicing feature can be multiplied with the N category features to obtain a similarity matrix. The multiplication of the last level splicing feature with the N category features may include: calculating the dot product of each pixel feature in the last level splicing feature and the N category features. Thus, the category features can be aggregated into the local features of the pixel according to the similarity between the N category features and each pixel feature.

[0269] In one example, the size of the similarity matrix S is H×W×N. Represents the local feature f of the pixel at (r,c) (r,c) The category feature f corresponding to the nth preset semantic category n The similarities between. It can be calculated according to formula 3:

[0270]

[0271] The similarity matrix may be multiplied with the N category features to obtain enhanced features corresponding to the orthophoto image. The last level of splicing features and the enhanced features may be concatenated to obtain enhanced fusion features corresponding to the orthophoto image.

[0272] By adopting the above implementation method, the accuracy of semantic segmentation of three-dimensional models can be improved.

[0273] Combination Figure 2 and Figure 5 , the enhanced fusion features can be input into the decoder for semantic segmentation to obtain the semantic segmentation result of the orthophoto. In one example, the structure of the decoder can be the same as the last classification layer of DeepLab V3+, including three convolution operations to further refine the fusion features, and upsampling using bilinear interpolation to adjust the output to the size of the original input image.

[0274] In the embodiment of the present disclosure, after obtaining the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition can be determined according to the semantic segmentation result of the orthoimage.

[0275] The target region may include at least one pixel. In most cases, the target region may include multiple pixels. In some application scenarios, the target region may also be referred to as a superpixel, a connected domain, or a connected region, etc., which is not limited here. The number of the target region may be at least one.

[0276] In the embodiment of the present disclosure, the target area may be an irregular area, that is, the target area may be of any shape, and is not limited to a rectangle. Of course, in some application scenarios, the target area may also be a regular area, such as a rectangle.

[0277] For a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition, manually annotated annotation data may be obtained, and based on the annotation data, the semantic segmentation result of the orthoimage may be updated.

[0278] In a possible implementation, steps S13 to S15 may be repeated for a preset number of times, or may be repeated until there is no target area in the orthoimage whose semantic segmentation quality satisfies a preset low-quality condition, or may be repeated until the updated semantic segmentation result of the orthoimage satisfies a preset high-quality condition. For example, the preset high-quality condition may be that the entropy of each pixel in the updated semantic segmentation result of the orthoimage is less than a first preset entropy value; for another example, the preset high-quality condition may be that the average entropy of each pixel in the updated semantic segmentation result of the orthoimage is less than a second preset entropy value; and so on.

[0279] In another possible implementation, after obtaining the annotation data of the target area, the annotation data of the target area can be used to replace the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image to obtain an updated semantic segmentation result of the orthophoto image.

[0280] In a possible implementation, the performing semantic segmentation on the orthophoto image to obtain the semantic segmentation result of the orthophoto image includes: performing semantic segmentation on the orthophoto image through a semantic segmentation network to obtain the semantic segmentation result of the orthophoto image; updating the semantic segmentation result of the orthophoto image based on the annotation data of the target area includes: updating the parameters of the semantic segmentation network based on the annotation data of the target area and the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image; performing semantic segmentation on the orthophoto image through the semantic segmentation network after updating the parameters to obtain the updated semantic segmentation result of the orthophoto image.

[0281] In this implementation, the labeling data of any target region can be used as the labeling data of each pixel in the target region, that is, the labeling data of each pixel in the same target region can be the same.

[0282] The semantic segmentation result of the target area may be obtained from the semantic segmentation result of the orthoimage, wherein the semantic segmentation result of the target area may include the probability that each pixel in the target area belongs to each of the N preset semantic categories.

[0283] The value of the loss function corresponding to the semantic segmentation network can be determined based on the labeling data of each pixel in the target area and the probability that each pixel in the target area belongs to each of the N preset semantic categories, and the parameters of the semantic segmentation network can be updated based on the value of the loss function.

[0284] After updating the parameters of the semantic segmentation network, the orthophoto image may be input into the semantic segmentation network with updated parameters to obtain an updated semantic segmentation result of the orthophoto image.

[0285] In this implementation, the parameters of the semantic segmentation network are updated based on the annotation data of the target area and the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image, thereby improving the accuracy of the semantic segmentation network in performing two-dimensional semantic segmentation. The orthophoto image is semantically segmented by the semantic segmentation network after the parameter update to obtain an updated semantic segmentation result of the orthophoto image, thereby improving the accuracy of the semantic segmentation result of the orthophoto image.

[0286] In a possible implementation, the determining, based on the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition, includes: for each pixel in the orthoimage, obtaining, from the semantic segmentation result of the orthoimage, a probability that the pixel belongs to each of N preset semantic categories, where N is an integer greater than 1; determining the entropy of the pixel based on the probability that the pixel belongs to each of the N preset semantic categories; and determining, based on the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition.

[0287] In this implementation, the entropy of any pixel in the orthoimage can represent the uncertainty of the semantic segmentation result of the pixel. The larger the entropy of the pixel, the higher the uncertainty of the semantic segmentation result of the pixel, and the lower the semantic segmentation quality of the pixel; the smaller the entropy of the pixel, the smaller the uncertainty of the semantic segmentation result of the pixel, and the higher the semantic segmentation quality of the pixel.

[0288] In one example, the entropy u of a pixel at (r,c) is (r,c) It can be determined according to formula 4:

[0289]

[0290] in, Represents the probability that the pixel (r, c) belongs to the nth preset semantic category.

[0291] The entropy of a pixel may range from [0, 1].

[0292] In this implementation, for each pixel in the orthoimage, the probability that the pixel belongs to each of N preset semantic categories is obtained from the semantic segmentation result of the orthoimage, and the entropy of the pixel is determined according to the probability that the pixel belongs to each of the N preset semantic categories, and the target area in the orthoimage whose semantic segmentation quality meets the preset low quality condition is determined according to the entropy of the pixel in the orthoimage. Thus, in the process of determining the target area in the orthoimage whose semantic segmentation quality meets the preset low quality condition, the probability that the pixel in the orthoimage belongs to each preset semantic category is taken into account, so that the semantic segmentation quality of the pixel can be measured more stably and robustly, thereby determining a more accurate target area.

[0293] In one example, the connected domain to which pixels having an entropy greater than or equal to a first preset threshold belong may be determined as a target region whose semantic segmentation quality satisfies a preset low-quality condition.

[0294] In a possible implementation, determining a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition based on the entropy of pixels in the orthoimage includes: generating a cross entropy map corresponding to the orthoimage based on the entropy of each pixel in the orthoimage; binarizing the cross entropy map to obtain a binarized cross entropy map corresponding to the orthoimage; determining a connected domain in the binarized cross entropy map; determining a semantic segmentation quality metric value of the connected domain; and determining a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition based on the semantic segmentation quality metric value of the connected domain.

[0295] In this implementation, a cross entropy map corresponding to the orthoimage can be obtained according to the entropy of each pixel in the orthoimage, wherein the size of the cross entropy map is the same as the size of the orthoimage, and the pixel value of any pixel in the cross entropy map can be positively correlated with the entropy of the corresponding pixel in the orthoimage.

[0296] Figure 6A schematic diagram showing the intermediate results of the active learning method in the semantic segmentation method of the three-dimensional model provided by the embodiment of the present disclosure. Figure 6 As shown, in one example, the jet color mapping algorithm can be used to map the orthophoto image (in Figure 6 The entropy of each pixel in the image (represented as “RGB image”) is mapped to obtain a cross entropy graph.

[0297] In another example, the cross entropy map may be a grayscale map. The entropy of the pixels in the orthoimage may be mapped to [0, 255] to obtain the cross entropy map. In this example, the pixel value of any pixel in the cross entropy map may be the entropy of the corresponding pixel in the orthoimage × 255.

[0298] like Figure 6 As shown, the cross entropy map can be binarized to obtain the binarized cross entropy map corresponding to the orthophoto image. After obtaining the binarized cross entropy map, the connected domain in the binarized cross entropy map can be determined according to the binarized cross entropy map. In one example, each connected domain can be assigned an independent label or index. Figure 6 In (d), different colors represent different connected domains, and each connected domain can be actively learned and selected as an independent query unit. The sizes and shapes of different connected domains can be different.

[0299] In this implementation, adjacent pixels with relatively poor semantic segmentation quality can be combined into irregular connected domains according to the semantic segmentation results of the orthophoto image, and then connected domains that meet preset low-quality conditions are selected from the generated connected domains for annotation, thereby reducing the burden of annotation work.

[0300] As an example of this implementation method, for any connected domain, the larger the semantic segmentation quality metric value of the connected domain, the higher the uncertainty of the semantic segmentation result of the connected domain, and the lower the semantic segmentation quality of the connected domain; the smaller the semantic segmentation quality metric value of the connected domain element, the smaller the uncertainty of the semantic segmentation result of the connected domain, and the higher the semantic segmentation quality of the connected domain.

[0301] For example, the preset low quality condition may include: the semantic segmentation quality metric value is greater than or equal to a second preset threshold. In this example, for any connected domain, if the semantic segmentation quality metric value of the connected domain is greater than or equal to the second preset threshold, the connected domain may be determined as the target area.

[0302] For another example, the preset low quality condition may include: the semantic segmentation quality metric value belongs to one of the largest M semantic segmentation quality metric values. In this example, for any connected domain, if the semantic segmentation quality metric value of the connected domain belongs to one of the largest M semantic segmentation quality metric values ​​among the semantic segmentation quality metric values ​​of all connected domains, the connected domain can be determined as the target area. That is, the connected domains can be sorted according to the order of the semantic segmentation quality metric values ​​from large to small, and the top M connected domains can be determined as target areas respectively.

[0303] In a possible implementation, binarizing the cross entropy map to obtain a binarized cross entropy map corresponding to the orthoimage includes: determining a binarization threshold based on the entropy of pixels in the orthoimage; and binarizing the cross entropy map based on the binarization threshold to obtain a binarized cross entropy map corresponding to the orthoimage.

[0304] As an example of this implementation method, determining the binarization threshold based on the entropy of the pixels in the orthoimage includes: sorting the pixels in the orthoimage in order from large to small according to the entropy of the pixels in the orthoimage, and determining the entropy of the qth pixel in the sorted pixels as the binarization threshold; wherein q is a value calculated based on the total number of pixels of the orthoimage and a preset proportional coefficient.

[0305] In this example, the pixels in the orthoimage can be sorted in descending order of entropy, and the entropy of the qth pixel in the sorted pixels can be determined as the binarization threshold, where q is an integer greater than 1. The product of the total number of pixels in the orthoimage and the preset proportional coefficient can be rounded to obtain q. For example, the preset proportional coefficient is 10%, which is not limited here. In this example, the ratio of q to Q is approximately equal to the preset proportional coefficient, and Q represents the total number of pixels in the orthoimage.

[0306] As another example of this implementation method, the pixels in the orthoimage can be sorted in descending order of the entropy of the pixels in the orthoimage, and the entropy of the qth pixel in the sorted pixels is determined as the binarization threshold; wherein q is a first preset value.

[0307] As another example of this implementation, the average value of the entropy of each pixel in the orthoimage may be determined as the binarization threshold.

[0308] As an example of this implementation method, for any pixel in the cross entropy map, if the grayscale value of the pixel is greater than the binarization threshold, the pixel value of the pixel can be converted to a preset maximum value (e.g., 255) through binarization; if the grayscale value of the pixel is less than or equal to the binarization threshold, the pixel value of the pixel can be converted to a preset minimum value (e.g., 0) through binarization.

[0309] In this implementation, an adaptive binarization threshold selection method is proposed. That is, the binarization threshold can be automatically adjusted according to the current semantic segmentation quality of the orthoimage. The binarization threshold is dynamically adjusted for different iteration rounds. According to this implementation, the annotation workload of the annotator in each iteration can be reduced, and the segmentation accuracy of the semantic segmentation network can be gradually improved.

[0310] In another possible implementation, the binarization threshold may be a fixed value.

[0311] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes an uncertainty metric value of the connected domain; and determining the semantic segmentation quality metric value of the connected domain includes: determining the uncertainty metric value of the connected domain according to the entropy of pixels in the connected domain.

[0312] In this implementation, for any connected domain, the average value of the entropy of each pixel in the connected domain may be determined as the uncertainty measure value of the connected domain.

[0313] In one example, the uncertainty measure E of the i-th connected domain can be determined using Equation 5: i :

[0314]

[0315] As an example of this implementation, for any connected domain in the orthoimage, the uncertainty measure value of the connected domain may be determined as the semantic segmentation quality measure value of the connected domain.

[0316] As another example of this implementation, for any connected domain in the orthoimage, a semantic segmentation quality metric value of the connected domain may be determined according to an uncertainty metric value of the connected domain and other metric values ​​of the connected domain.

[0317] In this implementation, the larger the uncertainty measure value of any connected domain, the higher the uncertainty of the semantic segmentation result of the connected domain, and the lower the semantic segmentation quality of the connected domain; the smaller the uncertainty measure value of the connected domain element, the smaller the uncertainty of the semantic segmentation result of the connected domain, and the higher the semantic segmentation quality of the connected domain.

[0318] In this implementation, the uncertainty measure value of the connected domain is determined based on the entropy of the pixels in the connected domain, and the semantic segmentation quality measure value of the connected domain is determined based on the uncertainty measure value of the connected domain. The semantic segmentation quality measure value thus determined can accurately measure the semantic segmentation quality of the connected domain.

[0319] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes a diversity metric value of the connected domain; determining the semantic segmentation quality metric value of the connected domain includes: determining the probability that the connected domain belongs to each of the N preset semantic categories based on the probability that pixels in the connected domain belong to each of the N preset semantic categories; determining the diversity metric value of the connected domain based on the probability that the connected domain belongs to each of the N preset semantic categories.

[0320] As an example of this implementation, for any connected domain, the probability that the connected domain belongs to any preset semantic category may be an average value of the probabilities that the pixels in the connected domain belong to the preset semantic category.

[0321] As an example of this implementation method, for any pixel in any connected domain, the preset semantic category corresponding to the maximum probability among the probabilities that the pixel belongs to each preset semantic category in the N preset semantic categories can be determined as the predicted semantic category corresponding to the pixel; the ratio of the number of pixels in the connected domain whose predicted semantic category is the nth preset semantic category to the total number of pixels in the connected domain can be calculated to obtain the probability that the connected domain belongs to the nth preset semantic category.

[0322] In one example, the diversity measure D of the i-th connected domain can be determined using Formula 6: i :

[0323]

[0324] Where N represents the number of preset semantic categories. It represents the probability that the i-th connected domain belongs to the n-th preset semantic category.

[0325] In this implementation, the larger the diversity metric value of any connected domain, the higher the uncertainty of the semantic segmentation result of the connected domain, and the lower the semantic segmentation quality of the connected domain; the smaller the diversity metric value of the connected domain elements, the smaller the uncertainty of the semantic segmentation result of the connected domain, and the higher the semantic segmentation quality of the connected domain.

[0326] As an example of this implementation, for any connected domain in the orthoimage, the diversity metric value of the connected domain may be determined as the semantic segmentation quality metric value of the connected domain.

[0327] As another example of this implementation, for any connected domain in the orthoimage, a semantic segmentation quality metric value of the connected domain may be determined according to a diversity metric value of the connected domain and other metric values ​​of the connected domain.

[0328] In this implementation, the probability that the connected domain belongs to each of the N preset semantic categories is determined according to the probability that the pixels in the connected domain belong to each of the N preset semantic categories, the diversity measure value of the connected domain is determined according to the probability that the connected domain belongs to each of the N preset semantic categories, and the semantic segmentation quality measure value of the connected domain is determined according to the diversity measure value of the connected domain. The semantic segmentation quality measure value thus determined can accurately measure the semantic segmentation quality of the connected domain.

[0329] In a possible implementation, the semantic segmentation quality metric value of any connected domain in the orthoimage can be determined based on the uncertainty metric value of the connected domain and the diversity metric value of the connected domain. i =(1-γ)E i +γD i , determine the semantic segmentation quality metric M of the i-th connected domain i , where γ can represent the balance coefficient, which is used to weigh the uncertainty measure E i and the diversity measure D i . For example, γ = 0.5.

[0330] In an embodiment of the present disclosure, when updating the semantic segmentation result of the orthophoto image, the semantic segmentation result of the three-dimensional model can be determined based on the updated semantic segmentation result of the orthophoto image and the correspondence between the pixels in the orthophoto image and the three-dimensional primitives in the three-dimensional model.

[0331] There is a big difference between label transfer based on orthophotos and label transfer based on multi-view images. When generating orthophotos, we have determined the index position of each 3D primitive in the 3D model on the image plane. Therefore, the semantic label of the 3D primitive can be determined by the segmentation result of the corresponding position in the 2D image. Taking the face f in the 3D texture mesh model as an example, let its visible area on the image plane be R f , R f The predicted probability of a pixel (r,c) in is op (r,c) , then the semantic label l of the patch ff It can be determined using Formula 7:

[0332]

[0333] In a possible implementation, the determining the semantic segmentation result of the 3D model based on the updated semantic segmentation result of the orthoimage and the correspondence between pixels in the orthoimage and 3D primitives in the 3D model includes: for each pixel in the orthoimage, determining a 3D primitive set corresponding to the pixel in the 3D model; traversing the 3D primitives in the 3D primitive set in descending order of height values, and in response to a height difference between a target 3D primitive and a previous 3D primitive of the target 3D primitive being greater than a preset height threshold, determining the semantic label of the pixel in the updated semantic segmentation result of the orthoimage as the semantic label of the previous 3D primitive and the 3D primitive before the previous 3D primitive, and determining the semantic label of the target 3D primitive and the 3D primitive after the target 3D primitive based on geometric consistency.

[0334] In urban aerial photography scenes, for points with the same x and y coordinates, the points with lower altitudes have a high probability of being consistent with the points with the maximum altitude in semantic labels. Therefore, using orthophotos for 3D semantic segmentation of urban scenes is a feasible solution.

[0335] Since the pixel value of any pixel in the orthophoto image can be determined based on the pixel value of the three-dimensional primitive with the largest height value in the three-dimensional primitive set corresponding to the pixel in the three-dimensional model, in this implementation method, for any pixel in the orthophoto image, the semantic label of the pixel can be assigned to the three-dimensional primitive with the largest height value in the three-dimensional primitive set corresponding to the pixel.

[0336] We found that for points on the same visible ray, their semantic labels tend to be consistent with the highest point, but if their heights differ too much, the semantic labels often change. Therefore, in this implementation, the elevation information can be used again to assist and set a filtering criterion. Figure 7 FIG. 1 is a schematic diagram showing a method for determining a semantic label of a three-dimensional primitive in a semantic segmentation method of a three-dimensional model provided by an embodiment of the present disclosure. Figure 7 As shown, d1,…,d n They represent the actual height values ​​of the n intersection points of a ray in space and the three-dimensional model. Arrange them in descending order. If the height value d of the mth point m and the height value d of the m+1th point m+1If the difference exceeds the preset height threshold δ, the label is no longer transferred downward, but the geometric consistency of the 3D model is used to assign semantic labels to the 3D points / triangles that have not yet been assigned semantic labels.

[0337] In one example, for a texture mesh model, each face is adjacent to three surrounding faces, so a graph model can be used for optimization. The vertices of the graph are triangular faces and the negative probabilities of their corresponding semantic labels. Each vertex has three edges, and the weight of the edge is determined by the difference in the normals of two adjacent triangular faces.

[0338] In one example, for a point cloud model, the points are discrete and the number is usually very large, so building a graph model will consume a lot of memory and the calculation speed will be greatly affected. Therefore, for the point cloud model, the KNN (K-Nearest Neighbor) algorithm can be used for processing.

[0339] In this implementation, by determining, for any pixel in the orthophoto image, a set of three-dimensional primitives corresponding to the pixel in the three-dimensional model, traversing the three-dimensional primitives in the three-dimensional primitive set in order of height values ​​from large to small, and in response to a height difference between any target three-dimensional primitive and the previous three-dimensional primitive of the target three-dimensional primitive being greater than a preset height threshold, the semantic label of the pixel in the semantic segmentation result of the orthophoto image is determined as the semantic label of the previous three-dimensional primitive and the three-dimensional primitive before the previous three-dimensional primitive, and the semantic label of the target three-dimensional primitive and the three-dimensional primitive after the target three-dimensional primitive is determined based on geometric consistency, thereby enabling the semantic segmentation result of the orthophoto image to be efficiently mapped back to the three-dimensional model, and the geometric consistency constraint can complete the semantic label of the three-dimensional primitive and improve the semantic segmentation quality of local details.

[0340] The semantic segmentation method of the three-dimensional model provided in the embodiment of the present disclosure can be applied to technical fields such as computer vision, high-precision three-dimensional maps, and three-dimensional scene understanding, and is not limited here. The semantic segmentation method of the three-dimensional model provided in the embodiment of the present disclosure can be applied to three-dimensional semantic segmentation scenarios such as geographic terrain analysis, urban planning, and large-scale complex scenes, and is not limited here. For example, the semantic segmentation method of the three-dimensional model provided in the embodiment of the present disclosure can be used to quickly perform semantic segmentation on large-scale urban three-dimensional scenes. For another example, the semantic segmentation method of the three-dimensional model provided in the embodiment of the present disclosure can assist in geographic terrain analysis and urban planning.

[0341] The following is an explanation of the semantic segmentation method of the three-dimensional model provided by the embodiment of the present disclosure through a specific application scenario. In this application scenario, the semantic segmentation method of the three-dimensional model includes four main steps: orthogonal projection, semantic segmentation of the two-dimensional image through a semantic segmentation network, active learning selection, and 2D-3D semantic fusion. The four steps are explained below respectively.

[0342] Step 1: Orthographic projection.

[0343] In this step, firstly, a three-dimensional model to be segmented may be obtained, wherein the three-dimensional model to be segmented may be a textured mesh model or a point cloud model with color.

[0344] Orthogonal projection can be used to project the three-dimensional model to be segmented onto a two-dimensional plane to obtain an orthophoto image and an elevation image corresponding to the three-dimensional model.

[0345] Step 2: Perform semantic segmentation on the two-dimensional image through the semantic segmentation network.

[0346] In this step, the orthophoto image and the elevation image can be input into the semantic segmentation network, and the semantic segmentation network can be used to perform pixel-by-pixel semantic prediction.

[0347] refer to Figure 2 , the semantic segmentation network can include a feature extraction backbone network, a cross-modal feature aggregation module (CM-FAM), a concatenation operation module (C), a category-level contextual guidance module (CF-CGM) and a decoder.

[0348] The feature extraction backbone network may include an orthophoto branch and an elevation image branch. The orthophoto branch may include five first feature extraction modules, and the elevation image branch may include five second feature extraction modules. The five first feature extraction modules include Figure 2 Layer_1~4 above can also include settings Figure 2 Layer_0 of Layer_1 above (in Figure 2 The 5 second feature extraction modules include Figure 2 Layer_1 to Layer_4 below can also include Figure 2 Layer_0 of Layer_1 below (in Figure 2 not shown). Figure 2 The output of Layer_1 above can be recorded as the first level first feature, Figure 2 The output of Layer_2 above can be recorded as the first feature of the second level. Figure 2 The output of Layer_3 above can be recorded as the first feature of the third level. Figure 2 The output of Layer_4 above can be recorded as the first feature of the fourth level. Figure 2The output of Layer_1 below can be recorded as the first-level second feature. Figure 2 The output of Layer_2 below can be recorded as the second level second feature, Figure 2 The output of Layer_3 below can be recorded as the second feature of the third level. Figure 2 The output of Layer_4 below can be recorded as the second feature of the fourth level.

[0349] The first level first feature and the first level second feature can be fused to obtain the first level fusion feature (i.e. Figure 2 The output of the first CM-FAM in the first CM-FAM); the second-level first feature and the second-level second feature can be fused to obtain the second-level fusion feature (i.e. Figure 2 The output of the second CM-FAM in the third level); the third level first feature and the third level second feature can be fused to obtain the third level fusion feature (i.e. Figure 2 The output of the third CM-FAM in the fourth level); the fourth level first feature and the fourth level second feature can be fused to obtain the fourth level fusion feature (i.e. Figure 2 output of the fourth CM-FAM in ).

[0350] The first splicing operation module ( Figure 2 The first C in the first level fusion feature (i.e. Figure 2 The output of the first CM-FAM in the Figure 2 The output of the second CM-FAM in the Figure 2 The output of the first C in ). The second splicing operation module ( Figure 2 The second C in the second level concatenation feature and the third level fusion feature (i.e. Figure 2 The output of the third CM-FAM in the above example is concatenated to obtain the third-level concatenated features (i.e. Figure 2 The output of the second C in the example). The third splicing operation module ( Figure 2 The third C in the third level concatenation feature and the fourth level fusion feature (i.e. Figure 2 The output of the fourth CM-FAM in is concatenated to obtain the fourth-level concatenated features (i.e. Figure 2 The output of the third C in ).

[0351] The last level of splicing features (i.e. Figure 2 The output of the third C in Figure 5 The pixel features are input into the category-level context guidance module (CF-CGM), and the probability of the pixel in the orthophoto image belonging to each of the N preset semantic categories is obtained by the category-level context guidance module (CF-CGM). Figure 5 (represented as "coarse segmentation result" in ). For any preset semantic category, the features of the target pixels in the last level of splicing features can be averaged to obtain the category features corresponding to the preset semantic category. Among them, the target pixel can represent a pixel with a maximum probability greater than a confidence threshold and a maximum probability corresponding to the preset semantic category. The maximum probability corresponding to any pixel corresponds to a pixel of the preset semantic category, which can indicate that among the probabilities of the pixel belonging to each of the N preset semantic categories, the probability of belonging to the preset semantic category is the largest. The last level of splicing features can be multiplied by the N category features to obtain a similarity matrix. The similarity matrix can be multiplied by the N category features to obtain the enhanced features corresponding to the orthophoto image. The last level of splicing features and the enhanced features can be spliced ​​to obtain the enhanced fusion features corresponding to the orthophoto image.

[0352] The enhanced fusion features can be input into a decoder, and a semantic segmentation result of the orthophoto image can be obtained through the decoder, wherein the semantic segmentation result of the orthophoto image can include the probability that each pixel in the orthophoto image belongs to each preset semantic category in the N preset semantic categories.

[0353] Step 3: Actively learn and choose.

[0354] In this step, the above formula 4 can be used to determine the entropy of each pixel in the orthophoto image according to the probability that each pixel in the orthophoto image belongs to each of the N preset semantic categories. The jet color mapping algorithm can be used to determine the entropy of each pixel in the orthophoto image according to the probability that each pixel in the orthophoto image belongs to each of the N preset semantic categories. Figure 6 The entropy of each pixel in the image is represented as "RGB image" and the cross entropy graph is obtained (refer to Figure 6 (b)). The pixels in the orthoimage can be sorted in descending order of entropy, and the entropy of the qth pixel in the sorted pixels can be determined as a binarization threshold, where q is an integer greater than 1. For example, the product of the number of pixels in the orthoimage and the second preset value can be rounded to obtain q. For example, the second preset value is 10%. The cross entropy map can be binarized based on the binarization threshold to obtain a binarized cross entropy map corresponding to the orthoimage (refer to Figure 6 (c)). The connected domain can be determined based on the binary cross entropy graph (refer to Figure 6 (d)). For any connected domain, the uncertainty measure value E of the connected domain can be determined according to the above formula 5: i , the diversity measure value D of the connected domain can be determined according to the above formula 6: i . According to M i =(1-γ)E i +γD i , determine the semantic segmentation quality metric M of the i-th connected domaini , for example, γ = 0.5. According to the semantic segmentation quality metric value of the connected domain, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition can be determined.

[0355] The annotation data of the target area can be obtained. Among them, the annotation data of any target area can be used as the annotation data of each pixel in the target area. That is, the annotation data of each pixel in the same target area can be the same. The probability that each pixel in the target area belongs to each of the N preset semantic categories can be obtained from the semantic segmentation result of the orthophoto image. According to the annotation data of each pixel in the target area and the probability that each pixel in the target area belongs to each of the N preset semantic categories, the value of the loss function corresponding to the semantic segmentation network can be determined, and the parameters of the semantic segmentation network can be updated according to the value of the loss function. After updating the parameters of the semantic segmentation network, the orthophoto image can be input into the semantic segmentation network after the updated parameters to obtain the updated semantic segmentation result of the orthophoto image.

[0356] Step 3 can be repeated multiple times.

[0357] Step 4: 2D-3D semantic fusion.

[0358] In this step, the semantic segmentation result of the two-dimensional image can be reprojected onto the three-dimensional model to perform 2D-3D semantic fusion to obtain the semantic segmentation result of the three-dimensional model. Specifically, for any pixel in the orthophoto image, the set of three-dimensional primitives corresponding to the pixel in the three-dimensional model can be determined. The three-dimensional primitives in the three-dimensional primitive set can be traversed in order of height values ​​from large to small, and in response to the height difference between any target three-dimensional primitive and the previous three-dimensional primitive of the target three-dimensional primitive being greater than a preset height threshold, the semantic label of the pixel in the updated semantic segmentation result of the orthophoto image is determined as the semantic label of the previous three-dimensional primitive and the three-dimensional primitive before the previous three-dimensional primitive, and the semantic label of the target three-dimensional primitive and the three-dimensional primitive after the target three-dimensional primitive is determined based on geometric consistency.

[0359] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0360] In addition, the present disclosure also provides a semantic segmentation device, an electronic device, a computer-readable storage medium, and a computer program product for a three-dimensional model, all of which can be used to implement any three-dimensional model semantic segmentation method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records in the method part and will not be repeated here.

[0361] Figure 8 FIG. 1 is a block diagram of a semantic segmentation device for a three-dimensional model provided by an embodiment of the present disclosure. Figure 8 As shown, the semantic segmentation device of the three-dimensional model includes:

[0362] A first projection module 81 is used to project the three-dimensional model to be segmented onto a two-dimensional plane using orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model;

[0363] A semantic segmentation module 82, configured to perform semantic segmentation on the orthoimage to obtain a semantic segmentation result of the orthoimage;

[0364] A first determination module 83 is used to determine, based on the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition;

[0365] An acquisition module 84 is used to obtain the labeling data of the target area;

[0366] An updating module 85, configured to update the semantic segmentation result of the orthoimage based on the annotation data of the target area;

[0367] The second determination module 86 is used to determine the semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthoimage and the correspondence between the pixels in the orthoimage and the three-dimensional primitives in the three-dimensional model.

[0368] In one possible implementation,

[0369] The semantic segmentation module 82 is used to: perform semantic segmentation on the orthoimage through a semantic segmentation network to obtain a semantic segmentation result of the orthoimage;

[0370] The updating module 85 is used to: update the parameters of the semantic segmentation network based on the annotation data of the target area and the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image; perform semantic segmentation on the orthophoto image through the semantic segmentation network with updated parameters to obtain an updated semantic segmentation result of the orthophoto image.

[0371] In a possible implementation, the first determining module 83 is used to:

[0372] For each pixel in the orthoimage,

[0373] Obtaining, from the semantic segmentation result of the orthoimage, a probability that the pixel belongs to each of N preset semantic categories, where N is an integer greater than 1;

[0374] Determining the entropy of the pixel according to the probability that the pixel belongs to each of the N preset semantic categories;

[0375] According to the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

[0376] In a possible implementation, the first determining module 83 is used to:

[0377] Based on the entropy of each pixel in the orthoimage, generating a cross entropy map corresponding to the orthoimage;

[0378] Binarizing the cross entropy map to obtain a binary cross entropy map corresponding to the orthoimage;

[0379] Determining a connected domain in the binary cross entropy graph;

[0380] Determining a semantic segmentation quality metric value of the connected domain;

[0381] According to the semantic segmentation quality metric value of the connected domain, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

[0382] In a possible implementation, the first determining module 83 is used to:

[0383] Determining a binarization threshold based on the entropy of pixels in the orthoimage;

[0384] Based on the binarization threshold, the cross entropy map is binarized to obtain a binarized cross entropy map corresponding to the orthoimage.

[0385] In a possible implementation, the first determining module 83 is used to:

[0386] The pixels in the orthoimage are sorted in descending order of the entropy of the pixels in the orthoimage, and the entropy of the qth pixel in the sorted pixels is determined as the binarization threshold; wherein q is a value calculated based on the total number of pixels of the orthoimage and a preset proportional coefficient.

[0387] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes an uncertainty metric value of the connected domain;

[0388] The first determining module 83 is used for:

[0389] An uncertainty measure value of the connected domain is determined according to entropy of pixels in the connected domain.

[0390] In a possible implementation, the semantic segmentation quality metric value of each connected domain includes a diversity metric value of the connected domain;

[0391] The first determining module 83 is used for:

[0392] Determining the probability that the connected domain belongs to each of the N preset semantic categories according to the probability that the pixels in the connected domain belong to each of the N preset semantic categories;

[0393] The diversity metric value of the connected domain is determined according to the probability that the connected domain belongs to each of the N preset semantic categories.

[0394] In one possible implementation,

[0395] The device further comprises: a second projection module, configured to project the three-dimensional model onto a two-dimensional plane using orthogonal projection to obtain an elevation image corresponding to the three-dimensional model;

[0396] The semantic segmentation module 82 is used to perform semantic segmentation on the orthoimage based on the elevation image to obtain a semantic segmentation result of the orthoimage.

[0397] In a possible implementation, the semantic segmentation module 82 is used to:

[0398] Performing feature extraction on the orthophoto image to obtain a first feature corresponding to the orthophoto image;

[0399] Performing feature extraction on the elevation image to obtain a second feature corresponding to the elevation image;

[0400] fusing the first feature and the second feature to obtain a fused feature;

[0401] Based on the fusion features, a semantic segmentation result of the orthophoto is obtained.

[0402] In a possible implementation, the semantic segmentation module 82 is used to:

[0403] Based on the first feature and the second feature, obtaining a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image;

[0404] Based on the first feature and the first attention weight map, obtaining a first feature to be fused corresponding to the orthoimage;

[0405] Based on the second feature and the second attention weight map, obtaining a second feature to be fused corresponding to the elevation image;

[0406] A fusion feature is obtained based on the first feature to be fused and the second feature to be fused.

[0407] In a possible implementation, the semantic segmentation module 82 is used to:

[0408] Performing attention learning based on the first feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image;

[0409] Performing attention learning based on the second feature through a second attention module to obtain a second transformation feature corresponding to the elevation image;

[0410] Splicing the first transformed feature and the second transformed feature to obtain a first spliced ​​feature;

[0411] Performing attention learning based on the first splicing feature through a third attention module to obtain a first attention weight map corresponding to the orthoimage;

[0412] Attention learning is performed based on the first splicing features through a fourth attention module to obtain a second attention weight map corresponding to the elevation image.

[0413] In a possible implementation, the semantic segmentation module 82 is used to:

[0414] Based on the first feature, the first attention weight map and a learnable first scale coefficient, obtaining a first feature to be fused corresponding to the orthoimage;

[0415] Based on the second feature, the second attention weight map and the learnable second scale coefficient, a second feature to be fused corresponding to the elevation image is obtained.

[0416] In a possible implementation, the semantic segmentation module 82 is used to:

[0417] Performing multi-level feature extraction on the orthoimage to obtain multi-level first features corresponding to the orthoimage;

[0418] Performing multi-level feature extraction on the elevation image to obtain multi-level second features corresponding to the elevation image;

[0419] fusing the multi-level first features and the multi-level second features respectively to obtain multi-level fused features;

[0420] Based on the multi-level fusion features, a semantic segmentation result of the orthophoto is obtained.

[0421] In a possible implementation, the semantic segmentation module 82 is used to:

[0422] The first-level fusion features and the second-level fusion features are spliced ​​to obtain the second-level splicing features;

[0423] In the case of including three or more levels of fusion features, the k-1th level splicing feature and the kth level fusion feature are spliced ​​to obtain the kth level splicing feature, where k is an integer greater than or equal to 3;

[0424] Based on the last level of splicing features, a semantic segmentation result of the orthophoto image is obtained.

[0425] In a possible implementation, the semantic segmentation module 82 is used to:

[0426] Based on the fusion feature, obtaining a probability that a pixel in the orthoimage belongs to each of the N preset semantic categories, where N is an integer greater than 1;

[0427] Based on the pixels in the orthoimage whose maximum probability is greater than the confidence threshold and the fusion features, determine the N category features corresponding to the N preset semantic categories; wherein the maximum probability corresponding to any pixel represents the maximum probability of the pixel belonging to each preset semantic category in the N preset semantic categories;

[0428] Based on the fusion features and the N category features, a semantic segmentation result of the orthoimage is obtained.

[0429] In a possible implementation, the semantic segmentation module 82 is used to:

[0430] Calculate similarity based on the fusion feature and the N category features to obtain a similarity matrix corresponding to the orthoimage;

[0431] Based on the similarity matrix and the N category features, obtaining enhanced features corresponding to the orthoimage;

[0432] Based on the fusion feature and the enhancement feature, an enhanced fusion feature corresponding to the orthoimage is obtained.

[0433] In a possible implementation, the second determining module 86 is used to:

[0434] For each pixel in the orthoimage, determining a set of three-dimensional primitives corresponding to the pixel in the three-dimensional model;

[0435] The three-dimensional primitives in the three-dimensional primitive set are traversed in order of height values ​​from large to small, and in response to the height difference between the traversed target three-dimensional primitive and the previous three-dimensional primitive of the target three-dimensional primitive being greater than a preset height threshold, the semantic label of the pixel in the updated semantic segmentation result of the orthophoto image is determined as the semantic label of the previous three-dimensional primitive and the three-dimensional primitive before the previous three-dimensional primitive, and the semantic label of the target three-dimensional primitive and the three-dimensional primitive after the target three-dimensional primitive is determined based on geometric consistency.

[0436] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

[0437] The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

[0438] The embodiment of the present disclosure further provides a computer program, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

[0439] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

[0440] An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; and a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

[0441] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0442] Fig. 9 1 is a block diagram of an electronic device 1900 provided in an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal. Fig. 9, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0443] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM ), a graphical user interface operating system launched by Apple (MacOS X TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0444] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0445] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0446] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0447] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0448] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0449] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0450] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0451] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0452] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0453] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0454] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0455] If the technical solution of the embodiments of the present disclosure involves personal information, the product using the technical solution of the embodiments of the present disclosure has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of the embodiments of the present disclosure involves sensitive personal information, the product using the technical solution of the embodiments of the present disclosure has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to collect his or her personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0456] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A semantic segmentation method for a three-dimensional model, characterized in that: include: Projecting the three-dimensional model to be segmented onto a two-dimensional plane using orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model; Performing semantic segmentation on the orthophoto image to obtain a semantic segmentation result of the orthophoto image; Based on the semantic segmentation result of the orthoimage, determining a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition; Obtaining the labeling data of the target area; Based on the annotation data of the target area, updating the semantic segmentation result of the orthoimage; Based on the updated semantic segmentation result of the orthoimage and the correspondence between pixels in the orthoimage and three-dimensional primitives in the three-dimensional model, a semantic segmentation result of the three-dimensional model is determined.

2. The method according to claim 1, characterized in that The performing semantic segmentation on the orthophoto image to obtain the semantic segmentation result of the orthophoto image comprises: performing semantic segmentation on the orthophoto image through a semantic segmentation network to obtain the semantic segmentation result of the orthophoto image; The updating of the semantic segmentation result of the orthophoto image based on the annotation data of the target area includes: updating the parameters of the semantic segmentation network based on the annotation data of the target area and the semantic segmentation result of the target area in the semantic segmentation result of the orthophoto image; performing semantic segmentation on the orthophoto image through the semantic segmentation network with updated parameters to obtain an updated semantic segmentation result of the orthophoto image.

3. The method according to claim 1 or 2, characterized in that: The determining, based on the semantic segmentation result of the orthoimage, a target area whose semantic segmentation quality in the orthoimage satisfies a preset low-quality condition comprises: For each pixel in the orthoimage, Obtaining, from the semantic segmentation result of the orthoimage, a probability that the pixel belongs to each of N preset semantic categories, where N is an integer greater than 1; Determining the entropy of the pixel according to the probability that the pixel belongs to each of the N preset semantic categories; According to the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

4. The method according to claim 3, characterized in that: The step of determining, according to the entropy of each pixel in the orthoimage, a target area in the orthoimage whose semantic segmentation quality satisfies a preset low-quality condition comprises: Based on the entropy of each pixel in the orthoimage, generating a cross entropy map corresponding to the orthoimage; Binarizing the cross entropy map to obtain a binary cross entropy map corresponding to the orthoimage; Determining a connected domain in the binary cross entropy graph; Determining a semantic segmentation quality metric value of the connected domain; According to the semantic segmentation quality metric value of the connected domain, a target area in the orthoimage whose semantic segmentation quality meets a preset low quality condition is determined.

5. The method according to claim 4, characterized in that Binarizing the cross entropy map to obtain a binary cross entropy map corresponding to the orthoimage includes: Determining a binarization threshold based on the entropy of pixels in the orthoimage; Based on the binarization threshold, the cross entropy map is binarized to obtain a binarized cross entropy map corresponding to the orthoimage.

6. The method according to claim 5, characterized in that The step of determining a binarization threshold based on the entropy of pixels in the orthoimage comprises: The pixels in the orthoimage are sorted in descending order of the entropy of the pixels in the orthoimage, and the entropy of the qth pixel in the sorted pixels is determined as the binarization threshold; wherein q is a value calculated based on the total number of pixels of the orthoimage and a preset proportional coefficient.

7. The method according to any one of claims 4 to 6, characterized in that The semantic segmentation quality metric value of each connected domain includes an uncertainty metric value of the connected domain; The determining of the semantic segmentation quality metric value of the connected domain includes: An uncertainty measure value of the connected domain is determined according to entropy of pixels in the connected domain.

8. The method according to any one of claims 4 to 7, characterized in that The semantic segmentation quality metric value of each connected domain includes a diversity metric value of the connected domain; The determining of the semantic segmentation quality metric value of the connected domain includes: Determining the probability that the connected domain belongs to each of the N preset semantic categories according to the probability that the pixels in the connected domain belong to each of the N preset semantic categories; The diversity metric value of the connected domain is determined according to the probability that the connected domain belongs to each of the N preset semantic categories.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: projecting the three-dimensional model onto a two-dimensional plane using orthogonal projection to obtain an elevation image corresponding to the three-dimensional model; The performing semantic segmentation on the orthophoto image to obtain the semantic segmentation result of the orthophoto image includes: performing semantic segmentation on the orthophoto image based on the elevation image to obtain the semantic segmentation result of the orthophoto image.

10. The method according to claim 9, characterized in that The performing semantic segmentation on the orthoimage based on the elevation image to obtain the semantic segmentation result of the orthoimage includes: Performing feature extraction on the orthophoto image to obtain a first feature corresponding to the orthophoto image; Performing feature extraction on the elevation image to obtain a second feature corresponding to the elevation image; fusing the first feature and the second feature to obtain a fused feature; Based on the fusion features, a semantic segmentation result of the orthophoto is obtained.

11. The method according to claim 10, characterized in that The fusing the first feature and the second feature to obtain a fused feature includes: Based on the first feature and the second feature, obtaining a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image; Based on the first feature and the first attention weight map, obtaining a first feature to be fused corresponding to the orthoimage; Based on the second feature and the second attention weight map, obtaining a second feature to be fused corresponding to the elevation image; A fusion feature is obtained based on the first feature to be fused and the second feature to be fused.

12. The method according to claim 11, characterized in that The obtaining, based on the first feature and the second feature, a first attention weight map corresponding to the orthophoto image and a second attention weight map corresponding to the elevation image, comprises: Performing attention learning based on the first feature through a first attention module to obtain a first transformation feature corresponding to the orthophoto image; Performing attention learning based on the second feature through a second attention module to obtain a second transformation feature corresponding to the elevation image; Splicing the first transformed feature and the second transformed feature to obtain a first spliced ​​feature; Performing attention learning based on the first splicing feature through a third attention module to obtain a first attention weight map corresponding to the orthoimage; Attention learning is performed based on the first splicing features through a fourth attention module to obtain a second attention weight map corresponding to the elevation image.

13. The method according to claim 11 or 12, characterized in that: The obtaining, based on the first feature and the first attention weight map, a first feature to be fused corresponding to the orthophoto image, comprises: obtaining, based on the first feature, the first attention weight map and a learnable first scale coefficient, a first feature to be fused corresponding to the orthophoto image; The obtaining of the second feature to be fused corresponding to the elevation image based on the second feature and the second attention weight map includes: obtaining the second feature to be fused corresponding to the elevation image based on the second feature, the second attention weight map and a learnable second scale coefficient.

14. The method according to any one of claims 10 to 13, characterized in that The step of extracting features from the orthoimage to obtain a first feature corresponding to the orthoimage includes: extracting multiple levels of features from the orthoimage to obtain multiple levels of first features corresponding to the orthoimage; The extracting features from the elevation image to obtain the second features corresponding to the elevation image comprises: extracting multiple levels of features from the elevation image to obtain multiple levels of second features corresponding to the elevation image; The fusing the first feature and the second feature to obtain a fused feature comprises: fusing the multi-level first feature and the multi-level second feature respectively to obtain a multi-level fused feature; The obtaining of the semantic segmentation result of the orthophoto image based on the fusion features includes: obtaining the semantic segmentation result of the orthophoto image based on the multi-level fusion features.

15. The method according to claim 14, characterized in that The obtaining of the semantic segmentation result of the orthophoto image based on the multi-level fusion features includes: The first-level fusion features and the second-level fusion features are spliced ​​to obtain the second-level splicing features; In the case of including three or more levels of fusion features, the k-1th level splicing feature and the kth level fusion feature are spliced ​​to obtain the kth level splicing feature, where k is an integer greater than or equal to 3; Based on the last level of splicing features, a semantic segmentation result of the orthophoto image is obtained.

16. The method according to any one of claims 10 to 15, characterized in that The obtaining of the semantic segmentation result of the orthophoto image based on the fusion feature includes: Based on the fusion feature, obtaining a probability that a pixel in the orthoimage belongs to each of the N preset semantic categories, where N is an integer greater than 1; Based on the pixels in the orthoimage whose maximum probability is greater than the confidence threshold and the fusion features, determine the N category features corresponding to the N preset semantic categories; wherein the maximum probability corresponding to any pixel represents the maximum probability of the pixel belonging to each preset semantic category in the N preset semantic categories; Based on the fusion features and the N category features, a semantic segmentation result of the orthoimage is obtained.

17. The method according to claim 16, characterized in that The obtaining of the semantic segmentation result of the orthoimage based on the fusion feature and the N category features includes: Calculate similarity based on the fusion feature and the N category features to obtain a similarity matrix corresponding to the orthoimage; Based on the similarity matrix and the N category features, obtaining enhanced features corresponding to the orthoimage; Based on the fusion feature and the enhancement feature, an enhanced fusion feature corresponding to the orthoimage is obtained.

18. The method according to any one of claims 1 to 17, characterized in that The determining the semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthoimage and the correspondence between pixels in the orthoimage and three-dimensional primitives in the three-dimensional model comprises: For each pixel in the orthoimage, determining a set of three-dimensional primitives corresponding to the pixel in the three-dimensional model; The three-dimensional primitives in the three-dimensional primitive set are traversed in order of height values ​​from large to small, and in response to the height difference between the traversed target three-dimensional primitive and the previous three-dimensional primitive of the target three-dimensional primitive being greater than a preset height threshold, the semantic label of the pixel in the updated semantic segmentation result of the orthophoto image is determined as the semantic label of the previous three-dimensional primitive and the three-dimensional primitive before the previous three-dimensional primitive, and the semantic label of the target three-dimensional primitive and the three-dimensional primitive after the target three-dimensional primitive is determined based on geometric consistency.

19. A semantic segmentation device for a three-dimensional model, characterized in that: include: A first projection module is used to project the three-dimensional model to be segmented onto a two-dimensional plane by orthogonal projection to obtain an orthophoto image corresponding to the three-dimensional model; A semantic segmentation module, used to perform semantic segmentation on the orthophoto image to obtain a semantic segmentation result of the orthophoto image; A first determination module is used to determine, based on the semantic segmentation result of the orthoimage, a target area in the orthoimage whose semantic segmentation quality meets a preset low-quality condition; An acquisition module, used for acquiring the labeling data of the target area; An updating module, used for updating the semantic segmentation result of the orthoimage based on the annotation data of the target area; The second determination module is used to determine the semantic segmentation result of the three-dimensional model based on the updated semantic segmentation result of the orthoimage and the correspondence between the pixels in the orthoimage and the three-dimensional primitives in the three-dimensional model.

20. An electronic device, characterized in that: include: one or more processors; a memory for storing executable instructions; The one or more processors are configured to call the executable instructions stored in the memory to execute the method according to any one of claims 1 to 18.

21. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 18 is implemented.

Images