A method for three-dimensional reconstruction of remote sensing images based on semantic information

Abstract

The application discloses a remote sensing image three-dimensional reconstruction method based on semantic information, and relates to the technical field of remote sensing image three-dimensional modeling. The method comprises the following steps: obtaining point cloud data based on remote sensing image processing of a target area; determining semantic category labels based on semantic segmentation of a semantic segmentation model; dividing voxel units based on the distribution of three-dimensional sparse point clouds, and obtaining initial anchor points; adjusting the distribution density of the initial anchor points in combination with the semantic category labels, and obtaining adjusted initial anchor points; iteratively training a three-dimensional Gaussian splashing model multiple times to update the anchor points, and obtaining scene anchor points; and rendering a three-dimensional reconstruction model of the target area based on the scene anchor points. The application effectively improves the anchor point densification quality and the geometric boundary definition of the model by introducing semantic information and combining 3DGS three-dimensional modeling technology, and can improve the modeling efficiency and enhance the structural rationality and semantic interpretation of the three-dimensional reconstruction result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of remote sensing image three-dimensional modeling, and in particular to a remote sensing image three-dimensional reconstruction method based on semantic information. BACKGROUND

[0002] With the improvement of remote sensing observation capability, high-resolution optical remote sensing satellites can provide a solid data foundation for city three-dimensional modeling, digital twin and other applications.

[0003] At present, in order to perform three-dimensional reconstruction based on remote sensing images, related technologies mostly adopt traditional methods based on Structure-from-Motion (SfM) and Multi-View Stereo (MVS). Although such methods are relatively mature in geometric recovery, they generally have a series of problems such as sparse anchor point distribution, insufficient texture fidelity, high computational complexity, and high model storage cost.

[0004] In recent years, three-dimensional Gaussian splatting technology (3D Gaussian Splatting, 3DGS) can be used for three-dimensional reconstruction of large-scale urban scenes. This method usually realizes hierarchical Gaussian ellipsoid expression through anchor point structure, so that 3DGS can gradually refine the reconstructed model at different resolutions. This multi-scale approach is particularly critical for processing large-scale urban scenes, enabling efficient, compact scene modeling and real-time rendering.

[0005] However, direct application of the 3DGS method in remote sensing image reconstruction still faces challenges. On the one hand, current 3DGS methods mainly rely on image geometry and texture signals in the anchor point initialization and optimization process. On the other hand, the geometric boundaries between different types of structural features in remote sensing scenes are prone to anchor point drift and fuzzy fusion. These two issues make it difficult for existing 3DGS methods to accurately distinguish between different types of structural features (such as buildings, roads, bridges, etc.) and the boundaries between different types of structural features during modeling. The resulting model is prone to mixed areas at the boundaries of adjacent features, which can severely affect the clarity, consistency, and interpretability of the modeling results, leading to defects such as feature class confusion and inaccurate boundary modeling.

[0006] Therefore, there is currently a lack of a three-dimensional reconstruction method that can accurately distinguish between feature types and boundaries based on remote sensing images. SUMMARY

[0007] The present application provides a remote sensing image three-dimensional reconstruction method and device based on semantic information, equipment and medium, to solve the defects of the above related technologies. The technical solution is as follows:

[0008] In a first aspect, the application provides a remote sensing image three-dimensional reconstruction method based on semantic information, characterized in that it comprises:

[0009] Obtaining a plurality of high-resolution remote sensing images of different perspectives covering a target area, and processing point cloud data of the target area based on each remote sensing image;

[0010] Performing semantic segmentation on each remote sensing image based on a semantic segmentation model to determine the semantic class label of each pixel point on each remote sensing image;

[0011] Dividing a plurality of voxel units based on the three-dimensional sparse point cloud distribution of the point cloud data, and constructing an initial anchor point at the position of each voxel unit in combination with the point cloud data, each remote sensing image, and the semantic class label;

[0012] Adjusting the initial distribution density of all initial anchor points in combination with the semantic class label to obtain adjusted initial anchor points;

[0013] Inputting the adjusted initial anchor points, each remote sensing image, and the semantic class label into a three-dimensional Gaussian splashing model, and iteratively training the three-dimensional Gaussian splashing model to update the anchor points to obtain scene anchor points;

[0014] Rendering a three-dimensional reconstruction model of the target area based on the scene anchor points.

[0015] In an optional solution of the first aspect, the obtaining of a plurality of high-resolution remote sensing images of different perspectives covering a target area and the processing of point cloud data of the target area based on each remote sensing image comprises:

[0016] Performing radiation correction processing on each remote sensing image;

[0017] Performing image registration based on the remote sensing image after radiation correction processing to convert all remote sensing images to a target coordinate system;

[0018] Extracting each three-dimensional sparse point cloud of the target area based on all remote sensing images converted to the target coordinate system to obtain the point cloud data.

[0019] In an optional solution of the first aspect, the dividing of a plurality of voxel units based on the point cloud distribution of the point cloud data comprises:

[0020] Uniformly dividing the three-dimensional sparse point cloud based on a preset voxel edge length to construct a plurality of voxel units;

[0021] Determining the number of point clouds and the voxel edge length in each voxel unit, and comparing the number of point clouds with a preset number threshold and comparing the voxel edge length with a minimum length threshold;

[0022] In a case where the number of point clouds is greater than the preset number threshold and the voxel edge length is greater than the minimum length threshold, the corresponding voxel unit is uniformly divided into a plurality of new voxel units of the same scale;

[0023] The number of point clouds in each voxel unit and the voxel edge length are determined based on the new voxel units, and the subsequent steps are performed;

[0024] In a case where the number of point clouds is less than or equal to the preset number threshold, or the voxel edge length is less than or equal to the minimum length threshold, the corresponding voxel unit is output.

[0025] In an optional implementation of the first aspect, the initial anchor point at the position of each voxel unit is constructed by combining the point cloud data, each of the remote sensing images, and the semantic category label, including:

[0026] The attribute parameters of the initial anchor point are determined by combining the point cloud data, each of the remote sensing images, and the semantic category label;

[0027] The initial anchor point at the position of each voxel unit is constructed based on the attribute parameters of the initial anchor point;

[0028] The determination process of the attribute parameters includes:

[0029] For each voxel unit, a geometric center point of the voxel unit is determined, and the geometric center point is taken as an anchor point position of the initial anchor point;

[0030] An anchor point scale of the initial anchor point is determined according to the voxel boundary and the voxel edge length of the voxel unit;

[0031] Based on the point cloud data, each point cloud in the voxel unit is determined, a Gaussian standard deviation of each point cloud relative to the anchor point position is determined, and an influence range of the initial anchor point is determined based on the Gaussian standard deviation;

[0032] A projection image of the voxel unit on each of the remote sensing images is obtained, an appearance attribute of the initial anchor point is determined according to a pixel value of each pixel point on the projection image, and a semantic category label set contained by the initial anchor point is determined according to a semantic category label of each pixel point on the projection image.

[0033] In an optional implementation of the first aspect, the initial distribution density of all initial anchor points is adjusted based on the semantic category label to obtain adjusted initial anchor points, including:

[0034] The distance between each initial anchor point and other adjacent initial anchor points is calculated;

[0035] determining that the corresponding initial anchor point belongs to an anchor point coverage blind area in a case where a distance between each of the initial anchor points and any other initial anchor point is greater than a preset distance threshold value;

[0036] projecting each of the initial anchor points onto each of the remote sensing images respectively to obtain an anchor point projection range;

[0037] calculating gradient values of each pixel point in the anchor point projection range, and determining that the corresponding initial anchor point belongs to a weak texture area in a case where an average value of gradient values of a plurality of pixel points is less than a preset gradient threshold value;

[0038] determining semantic class labels of each pixel point in the anchor point projection range, determining a proportion of a number of pixel points of each semantic class label to a total number of all pixel points, and determining that the corresponding initial anchor point belongs to a semantic stable area in a case where the proportion of any semantic class label is greater than a preset proportion threshold value;

[0039] reducing a number of anchors in all of the weak texture areas and all of the semantic stable areas to reduce anchor distribution density, and increasing a number of anchors in the anchor point coverage blind area to increase anchor distribution density;

[0040] obtaining the adjusted initial anchor points.

[0041] In an optional implementation of the first aspect, the multiple iterations of training the three-dimensional Gaussian splash model to update the anchors to obtain scene anchors include:

[0042] rendering, by the three-dimensional Gaussian splash model, based on the input initial anchors, and projecting the rendering results onto each of the remote sensing images respectively to obtain a synthesized image;

[0043] calculating a photometric error term according to pixel value differences at positions of each same-named pixel point on the synthesized image and a corresponding remote sensing image;

[0044] calculating a semantic consistency error term according to semantic class label differences at positions of each same-named pixel point on the synthesized image and a corresponding remote sensing image;

[0045] performing weighted fusion based on the photometric error term and the semantic consistency error term to obtain a fusion loss function;

[0046] updating attribute parameters of the initial anchors based on the fusion loss function to obtain updated initial anchors;

[0047] outputting the updated initial anchors as the scene anchors in a case where the fusion loss function is less than or equal to a preset loss threshold value;

[0048] Otherwise, input the updated initial anchor point into the three-dimensional Gaussian splash model for next round of iterative training, go to the step of rendering by the three-dimensional Gaussian splash model based on the input initial anchor point.

[0049] In an optional implementation of the first aspect, the updating of the attribute parameters of the initial anchor point based on the fusion loss function to obtain the updated initial anchor point includes:

[0050] In a case where the fusion loss function is greater than the preset loss threshold, a back propagation gradient is calculated according to a numerical value of the fusion loss function, and the attribute parameters of the initial anchor point are updated based on the back propagation gradient;

[0051] In the process of updating the attribute parameters of the initial anchor point, the anchor point scale of the initial anchor point is adjusted to adjust the distribution density of the initial anchor point according to the photometric error term and the semantic consistency error term in the fusion loss function, including:

[0052] In a case where the photometric error term is greater than a first photometric error threshold, the anchor point scale of the original initial anchor point is reduced to increase the anchor point density at the position of the original initial anchor point;

[0053] In a case where the photometric error term is less than a second photometric error threshold, the anchor point scale of the original initial anchor point is increased to reduce the anchor point density at the position of the original initial anchor point;

[0054] The adjustment gradient of the anchor point scale of the original initial anchor point is determined according to a mapping relationship between each semantic category label and the anchor point scale obtained in advance, and the anchor point scale of the initial anchor point is adjusted based on the adjustment gradient;

[0055] The updated initial anchor point is obtained.

[0056] In a second aspect, the application further provides a device for three-dimensional reconstruction of remote sensing images based on semantic information, including:

[0057] A data acquisition unit is configured to acquire a plurality of high-resolution remote sensing images of different perspectives covering a target region, and obtain point cloud data of the target region based on each remote sensing image;

[0058] A semantic segmentation unit is configured to perform semantic segmentation on each remote sensing image based on a semantic segmentation model, and determine a semantic category label of each pixel point on each remote sensing image;

[0059] An anchor point construction unit is configured to divide a plurality of voxel units based on a three-dimensional sparse point cloud distribution of the point cloud data, and construct an initial anchor point at a position of each voxel unit in combination with the point cloud data, each remote sensing image, and the semantic category label;

[0060] The anchor point construction unit is further configured to adjust initial distribution densities of all initial anchor points in combination with the semantic category label to obtain adjusted initial anchor points.

[0061] The anchor point construction unit is further configured to input the adjusted initial anchor points, each of the remote sensing images, and the semantic category label into a three-dimensional Gaussian splash model, and perform multiple iteration training on the three-dimensional Gaussian splash model to update anchor points, and obtain scene anchor points through the updating.

[0062] The model construction unit is configured to render a three-dimensional reconstruction model of the target area based on the scene anchor points.

[0063] In a third aspect, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method provided in the first aspect or any of the implementation manners of the first aspect when executing the program.

[0064] In a fourth aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program executable by a processor to implement the method provided in the first aspect or any of the implementation manners of the first aspect.

[0065] The technical scheme provided by some embodiments of the present application has at least the following beneficial effects:

[0066] The method provided by the present application performs pixel-level semantic classification on remote sensing images through a semantic segmentation model, associates semantic labels with point cloud data, introduces semantic constraints when constructing initial anchor points, and adjusts distribution densities in a targeted manner, thereby effectively improving the anchor point densification quality and the geometric boundary definition of the model, and overcoming the problem of detail loss caused by uniform sampling in traditional methods.

[0067] In addition, by introducing a semantic information guiding mechanism and fusing 3DGS three-dimensional modeling technology, the anchor point densification quality and the geometric boundary definition of the model are effectively improved, the structural rationality and semantic interpretation of the three-dimensional reconstruction result are significantly enhanced while ensuring the modeling efficiency, and the method can be widely applied to remote sensing intelligent perception scenes such as city modeling, remote sensing interpretation, digital twinning, and emergency assessment. BRIEF DESCRIPTION OF DRAWINGS

[0068] In order to more clearly illustrate the technical solutions in the present application or related art, the following will briefly introduce the drawings needed to be used in the embodiments or related art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without any creative effort.

[0069] Figure 1is a flowchart of a three-dimensional reconstruction method of a remote sensing image based on semantic information provided by an embodiment of the present application;

[0070] Figure 2 is a flowchart of iterative training of a three-dimensional Gaussian splatting model provided by an embodiment of the present application;

[0071] Figure 3 is a structural diagram of a device for three-dimensional reconstruction of a remote sensing image based on semantic information provided by an embodiment of the present application;

[0072] Figure 4 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0073] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0074] The terms “include” and “have” and any variations thereof in the specification and claims of the present application and the above-described drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but can optionally further include steps or modules not listed, or can optionally further include other steps or modules inherent to the process, method, product or device.

[0075] It should be noted that the terms “first” and “second” involved in the present application are merely to distinguish similar objects, and do not represent a specific order of the objects. Understandably, “first” and “second” can be interchanged in a specific order or sequence as allowed. It should be understood that the objects distinguished by “first” and “second” can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those described or illustrated herein.

[0076] It should be noted that in the related art, a three-dimensional Gaussian splatting model based on a three-dimensional Gaussian splatting (3D Gaussian Splatting, 3DGS) technique can be used to efficiently display and render a real scene. By placing a large number of Gaussian ellipsoids with position, scale, direction and color attributes in a three-dimensional space, the scene geometry and appearance information is modeled. Generally, the scene sparse point cloud can be input into the three-dimensional Gaussian splatting model, and the scene sparse point cloud is processed by the three-dimensional Gaussian splatting model. The flow can be summarized as follows:

[0077] First, the scene sparse point cloud is initialized as a set of Gaussian bodies, and then the Gaussian bodies are projected to each image plane through a differentiable rendering process and compared with the actual image at the pixel level. The geometry and color attributes of the Gaussian bodies are adjusted according to the comparison results, so as to gradually approach the appearance and structure of the real scene. Finally, rendering is performed based on the adjusted Gaussian bodies, and a three-dimensional reconstruction model of the target scene can be obtained.

[0078] It should be noted that the three-dimensional Gaussian splatting model above is an implementation manner in the three-dimensional Gaussian technology, and should not be understood as a limitation of the embodiments of the present application. The three-dimensional Gaussian splatting model in the related technology can be rendered in real time according to the input point cloud information. In addition, the three-dimensional Gaussian splatting model is different from the neural network model. The three-dimensional Gaussian splatting model does not have a modular structure such as a convolutional layer, and the object of training and adjustment is a Gaussian ellipsoid and parameters related to the Gaussian ellipsoid.

[0079] However, the related art directly optimizes a large number of Gaussian bodies as a whole, which has great challenges in efficiency and convergence.

[0080] The present application will be described in detail below with reference to specific embodiments.

[0081] Next, a three-dimensional reconstruction method of a remote sensing image based on semantic information provided by an embodiment of the present application will be described. Figure 1 The three-dimensional reconstruction method of a remote sensing image based on semantic information provided by an embodiment of the present application is shown in the flowchart. The method includes the following steps: Figure 1 , Figure 1 The three-dimensional reconstruction method of a remote sensing image based on semantic information provided by an embodiment of the present application is shown in the flowchart. The method includes the following steps:

[0082] S101, a plurality of high-resolution remote sensing images of different perspectives covering a target area are obtained, and point cloud data of the target area is processed based on each remote sensing image;

[0083] S102, each remote sensing image is segmented based on a semantic segmentation model, and a semantic class label of each pixel point on each remote sensing image is determined;

[0084] S103, a plurality of voxel units are divided based on the three-dimensional sparse point cloud distribution of the point cloud data, and an initial anchor point at the position of each voxel unit is constructed based on the point cloud data, each remote sensing image, and the semantic class label;

[0085] S104, the initial distribution density of all initial anchor points is adjusted in combination with the semantic class label, and an adjusted initial anchor point is obtained;

[0086] S105, input the adjusted initial anchor point, each of the remote sensing images and the semantic category label into a three-dimensional Gaussian splatting model, and perform multiple iteration training on the three-dimensional Gaussian splatting model to update the anchor point, and obtain a scene anchor point through the updating;

[0087] S106, render a three-dimensional reconstruction model of the target region based on the scene anchor point.

[0088] It should be noted that the embodiments of the present application adopt a 3DGS based on an anchor point structure. Firstly, point cloud data is obtained through step S101. Then, a group of anchor points is extracted and optimized from sparse point clouds through steps S102-S104, which are used to describe the main geometric skeleton or key region distribution of the scene. Subsequently, S104 is executed. Under the guidance of the anchor points, the Gaussian body in the surrounding area is locally managed and refined, so as to realize a more organized and efficient modeling process.

[0089] In this process, the anchor point not only undertakes the responsibility of global geometric guidance, but also plays a role in constraint and partition to a certain extent, which can effectively alleviate the optimization complexity problem caused by a large number of Gaussian parameters. Therefore, the initial distribution of the anchor point is crucial to the optimization process. Incorrect initial position and attribute will greatly reduce the convergence efficiency of the scene, or even cannot converge.

[0090] To realize the guarantee of scene convergence and modeling efficiency, the embodiments of the present application fuse semantic information under the above standard framework of anchor point based 3DGS. The semantic category label of each pixel point on each remote sensing image is extracted through S102. The semantic category weight is introduced on the basis of voxel sampling, the regional difference control of the anchor point scale, distribution density and the optimization parameter (such as anchor point update weight) of the subsequent iteration optimization process is realized, and the consistency of geometry and semantics is improved through such a way, so as to improve the modeling efficiency and accuracy.

[0091] In some embodiments, S101 further includes the following steps:

[0092] performing radiation correction processing on each of the remote sensing images;

[0093] performing image registration based on the remote sensing images after the radiation correction processing, and converting all remote sensing images to a target coordinate system;

[0094] extracting each three-dimensional sparse point cloud of the target region based on all remote sensing images converted to the target coordinate system, and obtaining the point cloud data.

[0095] Specifically, each remote sensing image in S101 can be collected by an optical remote sensing satellite, and at least needs to cover the target region. Each two remote sensing images can have different shooting angles, which is not limited by the present application.

[0096] Specifically, the process of converting all remote sensing images to the target coordinate system includes:

[0097] Based on the pre-processed images, image registration is performed to unify all images to the same three-dimensional coordinate system. Specifically, the Structure-from-Motion (SfM) method is used to extract the common feature point pairs between any two images, complete the camera internal and external parameter estimation and image registration, and thus unify all images to the same three-dimensional coordinate system (such as the geographic coordinate system WGS84). Then, the scene sparse point cloud of the target area can be constructed based on the registered images. The scene sparse point cloud can be understood as a sparse three-dimensional geometric representation with the distribution structure characteristics of ground objects, which can provide a basis for subsequent fine and high-fidelity modeling.

[0098] Specifically, in S102, the SAM semantic segmentation large model can be used to perform semantic segmentation on each high-resolution remote sensing image respectively, and a semantic label is assigned to each pixel in the remote sensing image, such as determining the ground object type of a certain pixel, such as bridge, building, open space, number, etc.

[0099] In some embodiments, in S103, the point cloud distribution of the point cloud data is divided into a plurality of voxel units, specifically including:

[0100] On the basis of the three-dimensional sparse point cloud distribution of the point cloud data, the anchor points are divided, specifically based on the preset voxel edge length, the three-dimensional sparse point cloud is uniformly divided, and a plurality of voxel units are constructed;

[0101] The number of point clouds in each voxel unit and the voxel edge length are determined, and the number of point clouds and the preset number threshold are compared, and the voxel edge length and the minimum length threshold are compared;

[0102] In the case where the number of point clouds is greater than the preset number threshold and the voxel edge length is greater than the minimum length threshold, the voxel unit is further divided, and the corresponding voxel unit is uniformly divided into a plurality of new voxel units with the same scale;

[0103] Based on the new voxel unit, the number of point clouds in each voxel unit and the voxel edge length are determined, and the subsequent steps are performed;

[0104] In the case where the number of point clouds is less than or equal to the preset number threshold, or the voxel edge length is less than or equal to the minimum length threshold, the corresponding voxel unit is output.

[0105] In some embodiments, in S103, the initial anchor point at the position of each voxel unit is constructed by combining the point cloud data, each remote sensing image, and the semantic category label, specifically including:

[0106] determine attribute parameters of the initial anchor points in combination with the point cloud data, each of the remote sensing images, and the semantic category labels;

[0107] construct the initial anchor points at the positions of each of the voxel units based on the attribute parameters of the initial anchor points;

[0108] The determination process of the attribute parameters includes:

[0109] For each of the voxel units, determine a geometric center point of the voxel unit, and take the geometric center point as an anchor point position of the initial anchor point;

[0110] determine an anchor point scale of the initial anchor point according to the voxel boundary and the voxel side length of the voxel unit;

[0111] determine each point cloud in the voxel unit based on the point cloud data, determine a Gaussian standard deviation of each of the point clouds relative to the anchor point position, and determine an influence range of the initial anchor point based on the Gaussian standard deviation, i.e., a Gaussian ellipsoid within the initial anchor point can be constructed;

[0112] obtain a projection image of the voxel unit on each of the remote sensing images, determine an appearance attribute of the initial anchor point according to a pixel value of each pixel point on the projection image, and determine a set of semantic category labels contained by the initial anchor point according to a semantic category label of each pixel point on the projection image.

[0113] In some embodiments, in S104, the initial distribution density of all the initial anchor points is adjusted in combination with the semantic category labels to obtain adjusted initial anchor points, and specifically includes:

[0114] Calculate the distance between each of the initial anchor points and other adjacent initial anchor points;

[0115] In a case where the distance between each of the initial anchor points and any other initial anchor point is greater than a preset distance threshold, it is determined that the corresponding initial anchor point belongs to an anchor point coverage blind area.

[0116] Specifically, it can be determined whether the distance between anchor points is too large and whether the boundaries between different types of semantics are covered by anchor points. If there is a case where the distance is too large or the semantic boundaries are not covered by anchor points, it is determined that there is an anchor point missing in the corresponding area.

[0117] Further, the anchor point coverage blind area identified can be subjected to a local interpolation and category-guided diffusion strategy, the number of anchor points in the anchor point coverage blind area is increased to improve the anchor point distribution density, potential missing anchor points are supplemented, and the initial distribution is ensured to be continuous and the semantic boundaries are ensured to be complete.

[0118] Further, each of the initial anchor points can be projected onto each of the remote sensing images respectively to obtain an anchor point projection range, i.e., a projection image.

[0119] The gradient value of each pixel point in the anchor point projection range is calculated, and in a case where the average value of the gradient values of a plurality of pixel points is less than a preset gradient threshold, it is determined that the corresponding initial anchor point belongs to a weak texture region.

[0120] Specifically, a weak texture region is usually characterized by weak local image gradient, few texture details, and gentle visual changes. The gradient amplitude graph of each projection image is calculated by using a Sobel operator or a local contrast index (such as variance or local entropy), and the average gradient amplitude G in the anchor point projection range is calculated. If G < G th (preset gradient threshold), the region is determined to be a weak texture region.

[0121] The semantic class label of each pixel point in the anchor point projection range is determined, and the proportion of the number of pixel points of each semantic class label in the total number of all pixel points is determined. In a case where the proportion of any semantic class label is greater than a preset proportion threshold, it is determined that the corresponding initial anchor point belongs to a semantic stable region.

[0122] Specifically, a semantic stable region usually refers to a region in which the semantic class exhibits high continuity, stable boundaries, and slow changes in space. By counting the dominant semantic class and its proportion P in the influence range of each anchor point, if P > P th (preset proportion threshold) and the class label remains stable in the images of multiple perspectives, it is determined to be a semantic stable region.

[0123] For each type of region, the density is adjusted respectively, specifically including:

[0124] The number of anchor points in all the weak texture regions and all the semantic stable regions is reduced to reduce the anchor point distribution density.

[0125] Thus, the adjusted initial anchor points are obtained.

[0126] In some embodiments, in the process of adjusting the initial distribution density of the initial anchor points, the anchor point scale and weight of the corresponding region are adjusted according to the ground object type corresponding to the semantic class. If the corresponding first type of semantic class label corresponds to a structure complex ground object, such as a visually prominent road, building, etc., the anchor point scale of the corresponding region is reduced to increase the anchor point density of the corresponding region.

[0127] If the corresponding second type of semantic class label corresponds to a structure relatively simple ground object, such as grassland, water body, sky, etc., the opposite operation is taken, i.e., the anchor point scale of the corresponding region is increased to reduce the anchor point density of the corresponding region.

[0128] Furthermore, embodiments of this application can also specify a third type of semantic category label, which can increase the weight of the initial anchor point corresponding to the third type in subsequent iterative training steps according to the actual scenario needs, for example, specifying bridge features as the third type.

[0129] For the third type, such as important categories (unclear), higher weights are assigned to the corresponding anchors, which means more frequent iterations.

[0130] In some embodiments, in S105, such as Figure 2 As shown, Figure 2 This is a flowchart of iterative training of a 3D Gaussian splash model. The process of iteratively training the 3D Gaussian splash model multiple times to update the anchor points and obtain updated scene anchor points specifically includes:

[0131] S201, the three-dimensional Gaussian splash model is used to render based on the input initial anchor point, and the rendering results are projected onto each of the remote sensing images to obtain a composite image.

[0132] S202, Calculate the loss term, including:

[0133] The photometric error term is calculated based on the difference in pixel values ​​at the locations of corresponding pixels in the synthetic image and the corresponding remote sensing image.

[0134] The semantic consistency error term is calculated based on the difference in semantic category labels at the locations of corresponding pixels in the synthetic image and the corresponding remote sensing image.

[0135] The fusion loss function is obtained by weighted fusion of the photometric error term and the semantic consistency error term.

[0136] S203, update the attribute parameters of the initial anchor point based on the fusion loss function to obtain the updated initial anchor point.

[0137] Next, after S203, the values ​​of the fusion loss function and the preset loss threshold are compared.

[0138] If the fusion loss function is less than or equal to a preset loss threshold, execute S204, including:

[0139] S204, output the updated initial anchor point as the scene anchor point.

[0140] Otherwise, perform the next iteration of training based on the updated initial anchor point, and execute S205, including:

[0141] S205, input the updated initial anchor points into the three-dimensional Gaussian splash model for the next round of iterative training, and proceed to step S201.

[0142] In some embodiments, in S105, the attribute parameters of the initial anchor points are updated based on the fusion loss function to obtain updated initial anchor points, specifically comprising:

[0143] In the case where the fusion loss function is greater than the preset loss threshold, a back propagation gradient is calculated according to the numerical value of the fusion loss function, and the attribute parameters of the initial anchor points are updated based on the back propagation gradient;

[0144] In the process of updating the attribute parameters of the initial anchor points, the anchor point scale of the initial anchor points is adjusted according to the photometric error term and the semantic consistency error term in the fusion loss function to adjust the distribution density of the initial anchor points, comprising:

[0145] In the case where the photometric error term is greater than the first photometric error threshold, the anchor point scale of the original initial anchor points is reduced to increase the anchor point density at the position of the original initial anchor points;

[0146] In the case where the photometric error term is less than the second photometric error threshold, the anchor point scale of the original initial anchor points is increased to reduce the anchor point density at the position of the original initial anchor points;

[0147] The adjustment gradient of the anchor point scale of the original initial anchor points is determined according to the mapping relationship between each type of semantic category label and the anchor point scale constructed in advance, and the anchor point scale of the initial anchor points is adjusted based on the adjustment gradient;

[0148] The updated initial anchor points are obtained.

[0149] In some embodiments, the mapping relationship between the semantic category label and the anchor point scale can also be established as a priori constraint of the anchor point attribute parameters, specifically, in the process of updating the attribute parameters of the initial anchor points in S105, comprising:

[0150] For complex structure and visually significant categories (such as buildings, roads, etc.), plane structure regularization is adopted to enhance the plane fitting ability of the anchor points on the building surface area.

[0151] For natural category ground objects (such as vegetation), the scale change is enhanced, that is, the adjustment gradient of the anchor point scale of the anchor point corresponding to the vegetation type in each iteration process is increased, so as to adapt to the diversity and spatial hierarchy of vegetation details.

[0152] For water bodies and other categories: according to the characteristics of the corresponding category ground objects, the corresponding shape or texture prior is designed, and the Gaussian parameters (position, scale, orientation, color, etc.) of the anchor points are combined with these prior constraints in the optimization process to realize the expression of the geometry and semantic structure more consistent with the real scene.

[0153] The anchor point distribution obtained by the above-mentioned manner is more reasonable, the initialization and densification strategy under the semantic guidance can effectively improve the anchor point quality and distribution uniformity of the structure edge and weak texture area, the obtained model geometric boundary is clearer, the semantic consistency constraint can enhance the intra-class continuity and inter-class boundary preservation in modeling, and improve the model spatial clarity; the model semantic interpretation is improved, the anchor point constructed with semantic attributes can support subsequent intelligent remote sensing applications such as semantic visualization and semantic simplified modeling; and the method has good expansibility and can be combined with any mainstream semantic segmentation model and applied to various satellite platforms and multi-scale remote sensing scenes.

[0154] The following is a device embodiment of the present application, which can be used to execute the method embodiments of the present application. For details not disclosed in the device embodiments of the present application, please refer to the method embodiments of the present application.

[0155] Next, please refer to Figure 3 A structural schematic diagram of a remote sensing image three-dimensional reconstruction device based on semantic information is provided for an exemplary embodiment of the present application. The device can be realized by software, hardware or a combination of the two to become all or part of the terminal, and can also be integrated as an independent module on the server. The remote sensing image three-dimensional reconstruction device based on semantic information in the embodiment of the present application can be applied to a terminal or cloud, and the device comprises a data acquisition unit, a semantic segmentation unit, an anchor point construction unit and a model construction unit, wherein:

[0156] The data acquisition unit is configured to acquire a plurality of high-resolution remote sensing images of different perspectives covering a target area, and obtain point cloud data of the target area based on each remote sensing image;

[0157] The semantic segmentation unit is configured to perform semantic segmentation on each of the remote sensing images based on a semantic segmentation model, and determine a semantic class label of each pixel point on each of the remote sensing images;

[0158] The anchor point construction unit is configured to divide a plurality of voxel units based on a three-dimensional sparse point cloud distribution of the point cloud data, and construct an initial anchor point at a position of each of the voxel units in combination with the point cloud data, each of the remote sensing images and the semantic class label;

[0159] The anchor point construction unit is further configured to adjust an initial distribution density of all the initial anchor points in combination with the semantic class label, and obtain an adjusted initial anchor point;

[0160] The anchor point construction unit is further configured to input the adjusted initial anchor point, each of the remote sensing images and the semantic class label into a three-dimensional Gaussian splatting model, and perform multiple iteration training on the three-dimensional Gaussian splatting model to update the anchor point, and obtain a scene anchor point through the update;

[0161] A model construction unit is configured to obtain a three-dimensional reconstruction model of the target region based on the scene anchor point rendering.

[0162] It should be noted that the device provided in the above embodiments is only used as an example to divide the above functions, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above functions. In addition, the device provided in the above embodiments and the remote sensing image three-dimensional reconstruction method based on semantic information belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0163] The embodiments of the present application further provide an electronic device, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the program to realize the steps of the method of any one of the above embodiments.

[0164] Please refer to Figure 4 The structural block diagram of an electronic device provided in the embodiments of the present application is shown in the figure.

[0165] As Figure 4 shown, the electronic device 400 includes a processor 401 and a memory 402.

[0166] In the embodiments of the present application, the processor 401 is the control center of the computer system, which can be the processor of a physical machine or the processor of a virtual machine. The processor 401 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 401 can be implemented in at least one of the hardware forms of a DSP (Digital Signal Processing), a FPGA (Field-Programmable Gate Array) and a PLA (Programmable Logic Array).

[0167] The processor 401 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state.

[0168] The memory 402 can include one or more computer-readable storage media. The computer-readable storage media can be non-transitory. The memory 402 can also include high-speed random access memory and can include non-volatile memory, such as one or more magnetic disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments of the present application, the non-transitory computer-readable storage medium in the memory 402 is used to store at least one instruction for being executed by the processor 401 to implement the method in the embodiments of the present application.

[0169] In some embodiments, the electronic device 400 further includes a peripheral device interface 403 and at least one peripheral device 404. The processor 401, the memory 402, and the peripheral device interface 403 can be connected through a bus or a signal line. Each peripheral device 404 can be connected to the peripheral device interface 403 through a bus, a signal line, or a circuit board. Specifically, the peripheral device interface 403 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 401 and the memory 402.

[0170] In some embodiments of the present application, the processor 401, the memory 402, and the peripheral device interface 403 are integrated on the same chip or circuit board; in some other embodiments of the present application, any one or two of the processor 401, the memory 402, and the peripheral device interface 403 can be implemented on a separate chip or circuit board. The embodiments of the present application do not make specific limitations in this regard.

[0171] The structural block diagram of the electronic device shown in the embodiments of the present application does not constitute a limitation on the electronic device 400, and the electronic device 400 can include more or fewer components than shown, or combine certain components, or use a different arrangement of components.

[0172] The embodiments of the present application also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of any of the preceding embodiments. The computer-readable storage medium can include, but is not limited to, any type of disk including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro-drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic or optical card, a nanosystem (including a molecular memory IC), or any type of media or device suitable for storing instructions and / or data.

[0173] Those skilled in the art can clearly understand the implementation of the various embodiments by means of software and the necessary general hardware platform from the above description of the embodiments, and of course, the embodiments can also be implemented by hardware. Based on such understanding, the above technical solutions or the part that contributes to the related art can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0174] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for three-dimensional reconstruction of remote sensing images based on semantic information, characterized in that, The method comprises the following steps: acquiring a plurality of high-resolution remote sensing images of different perspectives covering a target area, and processing point cloud data of the target area based on each remote sensing image; performing semantic segmentation on each remote sensing image based on a semantic segmentation model to determine a semantic class label of each pixel point on each remote sensing image; dividing a plurality of voxel units based on a three-dimensional sparse point cloud distribution of the point cloud data, and constructing an initial anchor point at a position of each voxel unit in combination with the point cloud data, each remote sensing image, and the semantic class label; adjusting an initial distribution density of all initial anchor points in combination with the semantic class label to obtain adjusted initial anchor points; inputting the adjusted initial anchor points, each remote sensing image, and the semantic class label into a three-dimensional Gaussian splashing model, and performing multiple iterations on the three-dimensional Gaussian splashing model to update the anchor points and obtain scene anchor points; rendering a three-dimensional reconstruction model of the target area based on the scene anchor points; the step of adjusting the initial distribution density of all initial anchor points in combination with the semantic class label to obtain adjusted initial anchor points comprises the following steps: calculating a distance between each initial anchor point and other adjacent initial anchor points; determining that a corresponding initial anchor point belongs to an anchor point coverage blind area when the distance between the initial anchor point and any adjacent initial anchor point is greater than a preset distance threshold; projecting each initial anchor point onto each remote sensing image to obtain an anchor point projection range; calculating a gradient value of each pixel point in the anchor point projection range, and determining that a corresponding initial anchor point belongs to a weak texture area when an average value of gradient values of a plurality of pixel points is less than a preset gradient threshold; determining a semantic class label of each pixel point in the anchor point projection range, determining a proportion of a number of pixel points of each semantic class label to a total number of all pixel points, and determining that a corresponding initial anchor point belongs to a semantic stable area when the proportion of any semantic class label is greater than a preset proportion threshold; reducing the number of anchor points in all weak texture areas and all semantic stable areas to reduce the anchor point distribution density, and increasing the number of anchor points in the anchor point coverage blind area to increase the anchor point distribution density; obtaining the adjusted initial anchor points. 2.The method of claim 1, wherein, The step of acquiring a plurality of high-resolution remote sensing images of different perspectives covering a target area and processing point cloud data of the target area based on each remote sensing image comprises the following steps: performing radiation correction processing on each remote sensing image; performing image registration based on the remote sensing image after the radiation correction processing, and converting all remote sensing images to a target coordinate system; extracting each three-dimensional sparse point cloud of the target area based on all remote sensing images converted to the target coordinate system to obtain the point cloud data. 3.The method of claim 1, wherein, The step of dividing a plurality of voxel units based on a point cloud distribution of the point cloud data comprises the following steps: uniformly dividing the three-dimensional sparse point cloud based on a preset voxel edge length to construct a plurality of voxel units; determining a number of point clouds in each voxel unit and a voxel edge length, and comparing the number of point clouds with a preset number threshold and comparing the voxel edge length with a minimum length threshold; In a case where the number of point clouds is greater than a preset number threshold and a voxel edge length is greater than a minimum length threshold, the corresponding voxel unit is uniformly divided into a plurality of new voxel units of the same scale; The number of point clouds in each voxel unit and the voxel edge length are determined based on the new voxel units, and the subsequent steps are performed; In a case where the number of point clouds is less than or equal to the preset number threshold, or the voxel edge length is less than or equal to the minimum length threshold, the corresponding voxel unit is output.

4. The method of claim 1, wherein, The initial anchor point at the position of each voxel unit is constructed based on the point cloud data, each remote sensing image, and the semantic category label, and includes: The attribute parameters of the initial anchor point are determined based on the point cloud data, each remote sensing image, and the semantic category label; The initial anchor point at the position of each voxel unit is constructed based on the attribute parameters of the initial anchor point. The determination process of the attribute parameters includes: For each voxel unit, the geometric center point of the voxel unit is determined, and the geometric center point is taken as the anchor point position of the initial anchor point; The anchor point scale of the initial anchor point is determined based on the voxel boundary and the voxel edge length of the voxel unit; Based on the point cloud data, each point cloud in the voxel unit is determined, the Gaussian standard deviation of each point cloud relative to the anchor point position is determined, and the influence range of the initial anchor point is determined based on the Gaussian standard deviation; The projection image of the voxel unit on each remote sensing image is obtained, the appearance attribute of the initial anchor point is determined based on the pixel value of each pixel point on the projection image, and the semantic category label set contained by the initial anchor point is determined based on the semantic category label of each pixel point on the projection image.

5. The method of claim 4, wherein the semantic information is obtained from a semantic database. The three-dimensional Gaussian splash model is iteratively trained multiple times to update the anchor point, and a scene anchor point is obtained after the update, and includes: The three-dimensional Gaussian splash model is used to render the input initial anchor point, and the rendering result is projected onto each remote sensing image to obtain a synthesized image; A photometric error term is calculated based on the pixel value difference of each same-named pixel point at the position on the synthesized image and the corresponding remote sensing image; A semantic consistency error term is calculated based on the semantic category label difference of each same-named pixel point at the position on the synthesized image and the corresponding remote sensing image; The photometric error term and the semantic consistency error term are weighted and fused to obtain a fusion loss function; The attribute parameters of the initial anchor point are updated based on the fusion loss function, and an updated initial anchor point is obtained; In a case where the fusion loss function is less than or equal to a preset loss threshold, the updated initial anchor point is output as the scene anchor point; Otherwise, the updated initial anchor point is input into the three-dimensional Gaussian splash model for the next round of iterative training, and the step of rendering the input initial anchor point based on the three-dimensional Gaussian splash model is performed.

6. The method of claim 5, wherein the semantic information is obtained from a semantic database. The attribute parameters of the initial anchor point are updated based on the fusion loss function, and an updated initial anchor point is obtained, and includes: In a case where the fusion loss function is greater than the preset loss threshold, a backpropagation gradient is calculated based on the numerical value of the fusion loss function, and the attribute parameters of the initial anchor point are updated based on the backpropagation gradient; In the process of updating the attribute parameter of the initial anchor point, the anchor point scale of the initial anchor point is adjusted according to the photometric error term and the semantic consistency error term in the fusion loss function to adjust the distribution density of the initial anchor point, comprising: In the case that the photometric error term is greater than a first photometric error threshold, the anchor point scale of the original initial anchor point is reduced to increase the anchor point density at the position of the original initial anchor point; In the case that the photometric error term is less than a second photometric error threshold, the anchor point scale of the original initial anchor point is increased to reduce the anchor point density at the position of the original initial anchor point; The adjustment gradient of the anchor point scale of the original initial anchor point is determined according to the mapping relationship between the semantic category label of each type and the anchor point scale, and the anchor point scale of the initial anchor point is adjusted based on the adjustment gradient; An updated initial anchor point is obtained.

7. A device for a method of three-dimensional reconstruction of remote sensing images based on semantic information according to any one of claims 1 to 6, characterized in that, The device comprises: A data acquisition unit configured to acquire a plurality of high-resolution remote sensing images of different perspectives covering a target area, and obtain point cloud data of the target area based on each remote sensing image; A semantic segmentation unit configured to perform semantic segmentation on each remote sensing image based on a semantic segmentation model, and determine a semantic category label of each pixel point on each remote sensing image; An anchor point construction unit configured to divide a plurality of voxel units based on a three-dimensional sparse point cloud distribution of the point cloud data, and construct an initial anchor point at the position of each voxel unit in combination with the point cloud data, each remote sensing image, and the semantic category label; The anchor point construction unit is further configured to adjust the initial distribution density of all initial anchor points in combination with the semantic category label, and obtain adjusted initial anchor points; The anchor point construction unit is further configured to input the adjusted initial anchor points, each remote sensing image, and the semantic category label into a three-dimensional Gaussian splatting model, and perform multiple iterations of training on the three-dimensional Gaussian splatting model to update anchor points, and obtain scene anchor points through the update; A model construction unit configured to render a three-dimensional reconstruction model of the target area based on the scene anchor points.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the method of any one of claims 1 to 6 when executed by the processor.

Images