Semantic structure driven satellite image ray adaptive sampling three-dimensional reconstruction method

Abstract

The application provides a semantic structure driven satellite image ray adaptive sampling three-dimensional reconstruction method, comprising: acquiring sparse satellite multi-view images and imaging parameters of a target area, and determining pixel rays and their sampling depth intervals based on the imaging parameters; acquiring a semantic category map aligned with the multi-view image pixels, and constructing semantic structure information; generating a pixel ray adaptive sampling plan based on the semantic structure information; sampling the pixel rays along the lines according to the sampling plan, using a trained network combined with a differentiable volume rendering to perform three-dimensional reconstruction, and obtaining a depth map and / or a digital surface model of the target area; the application can focus the sampling budget on geometric mutations and semantic boundary areas, reduce redundant sampling in low structure areas, and improve the boundary definition, completeness and stability of three-dimensional reconstruction under sparse satellite multi-view conditions.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image processing and 3D reconstruction, specifically involving a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method. Background Technology

[0002] Stereo vision and multi-view stereo (MVS) techniques typically rely on dense and well-distributed multi-view images to form stable geometric constraints. However, in satellite imaging missions, due to factors such as orbit and mission planning, occlusion, a limited and unevenly distributed number of available viewpoints, and significant differences in imaging geometry, input images often exhibit sparse viewpoint characteristics. Under these conditions, matching constraints in weakly textured or repetitive textured regions significantly degrade, easily leading to problems such as depth holes, geometric noise, artifacts, and boundary overflow, resulting in blurred contours, unstable edges, and severe local distortion in high-precision digital surface models (DSMs).

[0003] Differentiable rendering of implicit fields (such as neural radiation field methods) achieves globally consistent modeling capabilities by continuously modeling the density and appearance of the scene and using differentiable rendering for end-to-end optimization. However, under sparse satellite multi-view input conditions, if a uniform sampling strategy for each ray or a structure-independent fixed sampling strategy is still used, at least the following shortcomings exist:

[0004] (1) The sampling budget is consumed in large quantities in areas that contribute little to the digital surface model, while the sampling of key geometric discontinuities such as building outlines and shorelines is insufficient, resulting in insufficient boundary representation ability.

[0005] (2) The depth discontinuity in the boundary region is weakened by low-density sampling and volume rendering smoothness, which easily leads to cross-class diffusion and depth overflow;

[0006] (3) Effective supervision signals are scarce in weak texture regions. Fixed sampling makes it difficult to focus sampling points on the regions where surfaces are more likely to appear, thus affecting convergence stability and the integrity of the digital surface model. Summary of the Invention

[0007] To overcome the problems of insufficient edge sharpness, integrity, and stability of digital surface models under sparse satellite multi-view input conditions, if uniform sampling of each ray or fixed sampling strategies independent of structure are still used, this invention provides a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method. For the generation of digital surface models of sparse satellite multi-view images, a semantic structure prior is constructed, and the number of ray sampling points, sampling distribution, sampling interval, and resampling / rejection rules are adaptively set accordingly. This makes the sampling process match the semantic category, scene structure, boundary position, and internal consistency of the region. Training and inference are carried out within the implicit field framework of differentiable volume rendering, thereby improving the boundary quality, geometric stability, and reliability of weak texture region reconstruction of digital surface models under sparse input conditions.

[0008] According to one aspect of the present invention, a semantic structure-driven satellite image ray adaptive sampling three-dimensional reconstruction method is provided, comprising:

[0009] Acquire sparse satellite multi-view images and imaging parameters of the target area, and determine the pixel rays corresponding to the satellite image pixels and the sampling depth range of the pixel rays based on the imaging parameters;

[0010] Obtain a semantic category map aligned with the pixels of the multi-view image, and construct semantic structure information based on the semantic category map. The semantic structure information includes at least semantic category information, semantic boundary probability map, and semantic region structure information.

[0011] Based on the semantic structure information and the sampling depth range of the pixel ray, an adaptive sampling plan for the pixel ray is generated, and sampling points along the pixel ray are sampled according to the adaptive sampling plan for the pixel ray.

[0012] The sampling points are input into the trained differentiable volume rendering implicit field network, which outputs the scene representation of the sampling points on all pixel rays and performs three-dimensional reconstruction through differentiable volume rendering. The result of the three-dimensional reconstruction includes at least the depth map and / or digital surface model of the target area. The differentiable volume rendering implicit field network is trained based on the sampling points obtained by sampling and their corresponding multi-view image observation information.

[0013] As a further technical solution, the semantic region structure information includes semantic connectivity domains, semantic region area, semantic region internal consistency index, and / or semantic region adjacency relationships.

[0014] As a further technical solution, the pixel ray adaptive sampling plan is generated based on semantic category information, semantic boundary probability map and semantic region structure information. The pixel ray adaptive sampling plan includes: adaptive allocation of the number of ray sampling points, adaptive setting of sampling depth interval and sampling distribution, encrypted sampling rules for semantic boundary regions, downsampling rules for low-structure regions and / or cross-category diffusion suppression rules.

[0015] As a further technical solution, the adaptive allocation of the number of ray sampling points adopts a categorized sampling budget mechanism, which determines the number of sampling points or the range of sampling points for pixel rays based on semantic category information, semantic boundary probability map and semantic region structure information.

[0016] When a pixel belongs to a building, road, bridge, shoreline, land-water boundary area, or other high-structure category or high-structure area with high abrupt changes, contour abrupt changes, or geometric discontinuities, increase the number of sampling points for the corresponding pixel ray.

[0017] When a pixel belongs to a body of water, homogeneous ground, bare ground, shaded area, invalid pixel area, or other low-texture, low-structure category or low-structure area, and the pixel is not located in a semantic boundary area, reduce the number of sampling points of the corresponding pixel ray.

[0018] When a pixel is located in a high-probability region of the semantic boundary probability map or in a connected region of the semantic boundary, the number of sampling points of the corresponding pixel ray is increased.

[0019] When a pixel is located within the same semantically connected domain and far from the semantic boundary region, reduce the number of sampling points of the corresponding pixel ray, or allow pixel rays within the same semantically connected domain to share sampling parameters.

[0020] As a further technical solution, the adaptive setting of the sampling depth range and sampling distribution includes two-stage sampling:

[0021] The first stage of coarse sampling is performed within the preset sampling depth range to obtain surface candidate regions;

[0022] The second stage constructs a focused sampling distribution based on the surface candidate region and performs fine sampling; the fine sampling adopts a hybrid distribution sampling of a weighted combination of uniform sampling distribution and focused sampling distribution; wherein, when a pixel belongs to a high structure category or is located in a semantic boundary region, the weight of the focused sampling distribution is increased and / or the width of the focused sampling window is reduced to increase the sampling density near the surface candidate region; when a pixel belongs to a low structure category and is far from the semantic boundary region, the weight of the focused sampling distribution is reduced and / or the number of fine sampling points is reduced.

[0023] As a further technical solution, the encryption sampling rule for the semantic boundary region includes: when the semantic boundary probability value of a pixel in the semantic boundary probability map exceeds a preset semantic boundary probability threshold, or when it is in a semantic boundary connected region, a boundary expansion sampling window is set on the near end and far end of the surface candidate region corresponding to the pixel ray; the boundary expansion sampling window is adjacent to or partially overlaps with the surface candidate region, and extends at least partially to the depth range outside the surface candidate region; sampling is performed within the boundary expansion sampling window at a higher sampling point density than that of non-boundary regions.

[0024] As a further technical solution, the cross-category diffusion suppression rule includes at least one of the following: rejecting sampling for sampling points located in the sampling depth range with high risk of cross-category aliasing, and transferring the sampling budget to the candidate regions on both sides of the semantic boundary;

[0025] Alternatively, when determining that the coarse sampling results characterize a cross-category aliasing trend, the candidate surface depths on both sides of the semantic boundary are determined respectively, and a focused sampling window is constructed with the candidate surface depths on both sides as the center. The depth interval between the two focused sampling windows is set as a low sampling probability interval or a rejection sampling interval. Based on the two focused sampling windows, the fine sampling distribution is reconstructed and resampling is performed so that the sampling points are concentrated near the candidate surfaces on both sides of the semantic boundary.

[0026] As a further technical solution, the steps for 3D reconstruction using microvoid rendering include:

[0027] For any pixel ray, obtain all sampling points on the pixel ray, and use the scene representation output by the trained differentiable rendering implicit field network to calculate the pixel-level color prediction value of the pixel ray in the form of differentiable rendering, and / or calculate the depth prediction value of the pixel ray using the expectation form of the differentiable rendering weights with respect to the depth parameters, and finally obtain the pixel-level color prediction value and / or depth prediction value of all pixel rays.

[0028] Based on the depth prediction values ​​of all pixel rays, a depth map of the target area is generated, and / or based on the depth prediction values ​​of all pixel rays and the corresponding imaging parameters, the depth prediction values ​​are converted into three-dimensional points or elevation values ​​in a unified coordinate system, and a digital surface model of the target area is constructed based on the three-dimensional points or elevation values.

[0029] According to another aspect of this specification, a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction system is provided, characterized in that it includes:

[0030] The input module is used to acquire sparse satellite multi-view images and imaging parameters of the target area, and determine the pixel rays corresponding to the satellite image pixels and the sampling depth range of the pixel rays based on the imaging parameters.

[0031] The semantic structure acquisition module is used to acquire a semantic category map aligned with the pixels of the multi-view image, and to construct semantic structure information based on the semantic category map. The semantic structure information includes at least semantic category information, semantic boundary probability map, and semantic region structure information.

[0032] The pixel ray sampling module is used to generate an adaptive sampling plan for the pixel ray based on the semantic structure information and the sampling depth range of the pixel ray, and to sample the pixel ray along the line according to the adaptive sampling plan; wherein, the adaptive sampling plan for the pixel ray is used to increase the number of sampling points and / or sampling density for high structure category or semantic boundary regions, and to reduce the number of sampling points and / or sampling density for low structure category or internal regions of the same type of connected domain far from the semantic boundary.

[0033] The 3D reconstruction module is used to input the sampling points into the trained differentiable volume rendering implicit field network, output the scene representation of the sampling points on all pixel rays, and perform 3D reconstruction through differentiable volume rendering. The result of the 3D reconstruction includes at least the depth map and / or digital surface model of the target area. The differentiable volume rendering implicit field network is trained based on the sampling points obtained by sampling and their corresponding multi-view image observation information.

[0034] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0035] (1) By using semantic structure-driven ray adaptive sampling, the sampling budget is focused on key areas of DSM such as building outlines and shorelines, thereby improving boundary clarity and geometric stability;

[0036] (2) By using categorized sampling budget and boundary densification sampling strategy, the effective sampling coverage of high-structure areas such as buildings, roads, bridges, and shorelines is improved, while the redundant sampling of low-structure areas such as water bodies and homogeneous ground is reduced, thereby improving the integrity and sampling efficiency of DSM.

[0037] (3) By using resampling / rejection rules with cross-class diffusion suppression, the boundary depth overflow and cross-class aliasing artifacts are reduced, thereby improving the accuracy of DSM contours;

[0038] (4) Under sparse view conditions, the quality of DSM can be improved without increasing the number of input viewpoints, and it has good engineering applicability.

[0039] This invention, without relying on increasing the number of viewpoints, can explicitly utilize semantic structure information to adaptively control the ray sampling process, focusing the sampling budget on key regions of the DSM and enhancing the geometric expression of boundaries and weak texture regions, thereby improving the edge clarity, integrity and stability of the DSM. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method provided in an embodiment of the present invention;

[0042] Figure 2 This is a schematic diagram of a semantic structure-driven satellite image ray adaptive sampling three-dimensional reconstruction system provided in an embodiment of the present invention. Detailed Implementation

[0043] It should be noted that:

[0044] The terms “comprising” and “having”, and any variations thereof, in the specification, claims, and accompanying drawings of this invention are intended to cover a non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0045] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices. The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be decomposed, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form new technical solutions. Such combinations are not bound by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0047] like Figure 1 As shown, this invention provides a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method, comprising the following steps:

[0048] Step 1: Obtain sparse satellite multi-view images and imaging parameters of the target area, and determine the pixel rays corresponding to the satellite image pixels and the sampling depth range of the pixel rays based on the imaging parameters;

[0049] Step 2: Obtain a semantic category map aligned with the pixels of the multi-view image, and construct semantic structure information based on the semantic category map;

[0050] Step 3: Based on semantic structure information, generate an adaptive ray sampling plan for the pixel ray, and sample the pixel ray along the line according to the adaptive ray sampling plan;

[0051] Step 4: Input the sampling points into the trained differentiable volume rendering implicit field network, output the scene representation of the sampling points on all pixel rays, and perform three-dimensional reconstruction through differentiable volume rendering. The result of the three-dimensional reconstruction includes at least the depth map and / or digital surface model of the target area. The differentiable volume rendering implicit field network is trained based on the sampling points obtained by sampling and their corresponding multi-view image observation information.

[0052] Preferably, step 1 further includes performing at least one consistency preprocessing on the multi-view images, including but not limited to geometric consistency processing and radiometric consistency processing, and on this basis, performing invalid pixel removal, region cropping and / or block processing and other preprocessing to obtain multi-view image data for training and inference.

[0053] As one specific implementation method, step 1 includes:

[0054] Acquire sparse multi-view satellite image sets of the same target area and its imaging parameters, among which, This represents the satellite image from the k-th perspective. The number of viewpoints is finite. The imaging parameters include at least one of the following: RPC parameters, rigorous imaging model parameters, georeferenced parameters, projected coordinate system parameters, pixel resolution, and / or image exterior orientation parameters. A geometric mapping relationship between satellite image pixel coordinates and a unified 3D coordinate system is established based on these imaging parameters, and used to determine the pixel ray corresponding to each pixel and the sampling depth range of that pixel ray. Geometric uniformity is performed on the satellite images from each viewpoint, ensuring that the satellite images from each viewpoint have a corresponding relationship under a unified coordinate reference. Uniformity preprocessing includes, but is not limited to: radiometric uniformity, shadow or saturation pixel identification, invalid pixel removal, region cropping, tiled segmentation, and overlapping region setting, to obtain image blocks and their corresponding imaging parameters for training and inference.

[0055] In step 2, a semantic category map is generated that is aligned with the pixels of the multi-view image. The semantic categories include at least ground, buildings, and water bodies.

[0056] Optionally, the semantic category map can be generated or obtained by one of the following methods: (1) manually annotating semantic labels; (2) obtaining the semantic category map by reasoning satellite images using a semantic segmentation model; (3) generating the semantic category map by converting external geographic information data.

[0057] In step 2, the semantic structure information includes at least semantic category information S, semantic boundary probability map B, and semantic region structure information R.

[0058] The semantic category information S is used to identify the category to which a pixel belongs, including but not limited to buildings, roads, bridges, shorelines, water bodies, ground, vegetation, shadow areas, invalid pixel areas, etc.

[0059] The semantic boundary probability map B is used to characterize the probability or intensity of a category change or geometrical abrupt change at a pixel.

[0060] Semantic region structural information R is used to characterize the extent and internal consistency of similar regions, including semantic connectivity, semantic region area, semantic region adjacency relationships, and / or category internal consistency indicators.

[0061] The semantic category information S, semantic boundary probability map B, and semantic region structure information R are used together to drive the adaptive adjustment of the number of subsequent pixel ray sampling points, sampling distribution, and sampling density.

[0062] Specifically, firstly, the semantic boundary probability map B is used to characterize the probability or intensity of a category change at a pixel, and is obtained by any of the following methods: (1) by performing gradient operation, morphological gradient operation or edge operator operation on the semantic category map; (2) by the output of the boundary prediction branch of the semantic segmentation model.

[0063] Secondly, semantic region structure information R is used to characterize the range and internal consistency of similar regions. It can be obtained through connected component analysis or instance segmentation, including at least one of connected components, instance regions, or category internal consistency indicators, to guide the category internal sampling consistency strategy.

[0064] In addition, the core purpose of semantic category information S, semantic boundary probability map B, and semantic region structure information R is to drive the generation of the ray adaptive sampling plan in the subsequent step 3, and it is not required to be used as network input features.

[0065] In step 3, adaptive ray sampling includes at least: adaptive allocation of the number of ray sampling points; adaptive setting of sampling depth interval and sampling distribution; encrypted sampling rules for semantic boundary regions; downsampling rules for low-structure regions; resampling rules and / or rejection rules for cross-class diffusion suppression.

[0066] Specifically, firstly, the adaptive allocation of the number of pixel ray sampling points adopts a categorical sampling budget mechanism.

[0067] Based on semantic category information S, semantic boundary probability map B, and semantic region structure information R, different pixel rays are divided into high-structure sampling type, ordinary structure sampling type, and low-structure sampling type.

[0068] For high-structure categories or high-structure areas with highly abrupt changes, abrupt changes in outline, or geometric discontinuities, such as buildings, roads, bridges, shorelines, and water-land boundary areas, a higher number of sampling points or a higher range of sampling points should be set.

[0069] For low-texture, low-structure categories or low-structure regions such as water bodies, homogeneous ground, bare ground, shaded areas, and invalid pixel areas, and when these regions are not located in semantic boundary areas, set a lower number of sampling points or a lower range of sampling points.

[0070] For pixels whose semantic boundary probability value exceeds a preset threshold, or pixels located within the semantic boundary connected domain, increase the number of sampling points for their corresponding pixel rays.

[0071] For pixels located within the same semantically connected domain and far from the semantic boundary, reduce the number of sampling points of their corresponding pixel rays, or allow pixel rays within the same semantically connected domain to share sampling parameters to reduce redundant sampling.

[0072] Secondly, the adaptive setting of the sampling depth range and sampling distribution includes two-stage sampling:

[0073] A preset sampling depth range is used. In the first stage, coarse sampling is performed within the preset depth range to obtain surface candidate areas.

[0074] The second stage involves constructing a focused sampling distribution based on the surface candidate region and performing fine sampling; the fine sampling adopts a hybrid distribution sampling that is a weighted combination of uniform sampling distribution and focused sampling distribution.

[0075] Optionally, the focused sampling distribution can be any one or more of a truncated Gaussian distribution, a windowed dense sampling distribution, or a segmented dense sampling distribution.

[0076] In this mixed distribution, the weight of the focused sampling is adaptively adjusted based on the semantic category information S and the semantic boundary probability map B. When a pixel belongs to a high-structure category such as a building, road, or shoreline, or is located in a high-probability semantic boundary region, the weight of the focused sampling distribution is increased, making the sampling points more concentrated in the surface candidate area and its neighboring area. When a pixel belongs to a low-structure category such as a water body or homogeneous ground and is far from the semantic boundary, the weight of the focused sampling distribution is decreased, reducing the number of fine sampling points or increasing the proportion of low-density uniform sampling.

[0077] Third, the semantic boundary region encryption sampling rules include: when the semantic boundary probability value of a pixel in the semantic boundary probability map exceeds a preset semantic boundary probability threshold, or when it is in a semantic boundary connected region, a boundary extension sampling window is set on the near end and far end of the surface candidate region of the pixel corresponding to the pixel ray; the boundary extension sampling window is adjacent to or partially overlaps with the surface candidate region, and extends at least partially to the depth range outside the surface candidate region; sampling is performed within the boundary extension sampling window at a higher sampling point density than that in non-boundary regions.

[0078] Optionally, the boundary expansion sampling window on both sides can be selected as a symmetrical window or an asymmetrical window.

[0079] Fourth, the resampling rules and / or rejection rules for cross-class diffusion suppression shall include at least one of the following:

[0080] Sampling points located in the sampling depth range with high risk of cross-category aliasing are rejected for sampling, and the sampling budget is transferred to the candidate regions on both sides of the semantic boundary;

[0081] Alternatively, when determining that the coarse sampling results characterize a cross-category aliasing trend, the candidate surface depths on both sides of the semantic boundary are determined respectively, and a focused sampling window is constructed with the candidate surface depths on both sides as the center. The depth interval between the two focused sampling windows is set as a low sampling probability interval or a rejection sampling interval. Based on the two focused sampling windows, the fine sampling distribution is reconstructed and resampling is performed so that the sampling points are concentrated near the candidate surfaces on both sides of the semantic boundary.

[0082] Wherein, when a pixel is located within a semantic boundary connected domain, or when at least two different semantic categories exist within a preset neighborhood of the pixel, or when the semantic boundary probability value of the pixel exceeds a preset threshold, at least a portion of the sampling depth range of the pixel ray corresponding to the pixel is determined to be a sampling depth range with a high risk of cross-category aliasing.

[0083] When the rendering weight distribution in the coarse sampling result forms a continuous transition, multiple local peaks, or a high variance depth distribution between the candidate regions on both sides of the semantic boundary, it is determined that the coarse sampling result represents a cross-class aliasing trend.

[0084] Preferably, step 3 further includes constructing auxiliary priors and observability metrics to assist in enhancing the sampling plan. These auxiliary priors and observability metrics include at least one of the following: texture intensity / gradient metrics, weak texture region identifiers, coarse depth priors and their confidence levels, and viewpoint coverage metrics. The ray-adaptive sampling plan in step 3 is modified or jointly decided based on these auxiliary priors and observability metrics. When the above auxiliary priors and observability metrics are missing, the ray-adaptive sampling plan can still be generated solely based on semantic structure information.

[0085] Preferably, step 3 further includes: sharing sampling parameters within the same semantic region structure or imposing consistency constraints on the sampling parameters to enhance geometric consistency within similar regions.

[0086] As a specific implementation method, step 3, the generation and execution of the pixel ray adaptive sampling plan based on semantic structure information, includes:

[0087] 3-1. Light parameterization based on imaging parameters.

[0088] For any given satellite image pixel, its corresponding pixel ray in a unified three-dimensional coordinate system is determined based on the imaging parameters of the viewpoint to which that pixel belongs. The pixel ray... The mathematical representation of is:

[0089]

[0090] In the formula, The starting point of the ray determined by the imaging parameters, The direction of the light rays is determined by the imaging parameters. The depth parameter is along the direction of the light ray. The sampling depth range is determined by imaging parameters, the elevation range of the target area, and the external DEM or scene bounding box.

[0091] 3-2. (Optional) Construction of auxiliary priors and observability indicators.

[0092] To improve the robustness of the sampling strategy in sparse viewpoints and weak texture scenarios, auxiliary information can be further constructed, including:

[0093] Texture intensity map Used to identify weakly textured regions, it can be obtained by calculating local gradient magnitude, local variance, or frequency domain energy;

[0094] Deep coarse prior With confidence level It can be generated from multi-view stereo, monocular depth, or external DEM;

[0095] Geometric coverage Used to describe observable information such as viewpoint overlap and differences between baseline and incident angle.

[0096] When the above auxiliary information is missing, steps 3.3 to 3.5 can still generate an adaptive ray sampling plan based solely on semantic structure information.

[0097] 3-3. Adaptive allocation of the number of sampling points.

[0098] For each pixel ray According to semantic category Boundary probability And optional texture intensity Depth confidence Wait, determine the number of sampling points In one embodiment, based on semantic category With boundary probability Allocate sampling budget:

[0099] When a pixel belongs to a high-structure category or high-structure region such as a building, road, bridge, shoreline, or water-land boundary area, or is located in a high-probability semantic boundary region, the number of sampling points is increased.

[0100] When a pixel belongs to a low-texture, low-structure category or low-structure region such as water, homogeneous ground, bare ground, shadow area, or invalid pixel area, and is not located in a semantic boundary region, reduce the number of sampling points to save budget.

[0101] When pixels are located within the same semantically connected domain and far from the semantic boundary, reduce the number of sampling points or share sampling parameters; when pixels belong to regions with weak texture but high structural risk, appropriately increase the number of sampling points and combine candidate region focusing sampling.

[0102]

[0103] in, It is a monotonic mapping function, which can be a linear mapping, piecewise mapping, or compression function; These are the upper and lower limits for the number of sampling points.

[0104] 3-4. Adaptive setting of sampling interval and sampling distribution.

[0105] To balance efficiency and surface representation ability, a two-stage sampling method is preferred:

[0106] First stage coarse sampling: Based on uniform distribution... Upsampling From these points, we can obtain the initial rendering weights or coarse depth estimates.

[0107] Second-stage fine sampling: based on the surface candidate depth obtained from coarse sampling. Or based on deep coarse priors Construct a focused sampling distribution near the candidate region. One point.

[0108] The fine sampling distribution can be set as a mixed distribution:

[0109]

[0110] in To ensure uniform distribution, To surround The focused distribution To focus on sampling weights. It adaptively adjusts based on semantic category information S and semantic boundary probability B; when a pixel belongs to a high-structure category such as a building, road, or shoreline, or is located in a high-probability semantic boundary region, it improves... This makes sampling points more concentrated in the surface candidate area and its vicinity; when pixels belong to low-structure categories such as water bodies and homogeneous ground and are far from semantic boundaries, it reduces... Reduce the number of fine sampling points or increase the proportion of low-density uniform sampling.

[0111] 3-5. Encrypted sampling of semantic boundary regions and cross-category diffusion suppression.

[0112] When the boundary probability B at a pixel is greater than a preset threshold or belongs to a semantic boundary connected region, a boundary enhancement and cross-class suppression strategy is triggered, including at least one or more of the following:

[0113] (1) Boundary encryption sampling: When the boundary probability B at a pixel is greater than a preset threshold or belongs to a semantic boundary connected region, in the candidate depth Alternatively, boundary extension sampling windows may be set on both the proximal and distal sides of the surface candidate area. These boundary extension sampling windows are adjacent to or partially overlap with the surface candidate area, and extend at least partially into the depth range outside the surface candidate area. The sampling point density is increased within the boundary extension sampling windows to enhance the representation of depth discontinuities such as building edges, road edges, and shorelines.

[0114] (2) Cross-class aliasing risk determination: When a pixel is located in the semantic boundary connected domain, or there are at least two different semantic categories in the preset neighborhood of the pixel, or the semantic boundary probability value of the pixel exceeds the preset threshold, at least part of the sampling depth interval of the pixel ray corresponding to the pixel is determined to be the sampling depth interval with high cross-class aliasing risk.

[0115] (3) Reject sampling: Discard sampling points located in the sampling depth range with high risk of cross-category aliasing, and transfer the corresponding sampling budget to the candidate regions on both sides of the semantic boundary;

[0116] (4) Resampling: When the rendering weight distribution in the coarse sampling result forms a continuous transition, multiple local peaks or high variance depth distribution between the candidate regions on both sides of the semantic boundary, it is determined that the coarse sampling result represents a cross-class aliasing trend; when it is determined that the coarse sampling result represents a cross-class aliasing trend, the candidate surface depths on both sides of the semantic boundary are determined respectively, and a focused sampling window is constructed with the candidate surface depths on both sides as the center. The depth interval between the focused sampling windows on both sides is set as a low sampling probability interval or a rejection sampling interval. Based on the focused sampling windows on both sides, the fine sampling distribution is reconstructed and resampling is performed so that the sampling points are concentrated near the candidate surfaces on both sides of the semantic boundary.

[0117] (5) Category Consistency Constraint Sampling: Maintain consistency of sampling strategies or share sampling parameters within the same semantically connected domain to enhance depth consistency within the region.

[0118] By using the above rules, the impact of cross-category aliasing on rendering and geometric representation is reduced at the sampling level, thereby improving the stability of DSM boundary contours.

[0119] In step 4, the implicit field network for microvoid rendering is a neural radiation field network, and the scene representation includes at least volume density representation and / or volume radiation representation.

[0120] Preferably, the scene representation may also include semantic distribution indicators of the sampling points.

[0121] Specifically, the pixel rays obtained in step 3 The sampling points on the sampling point are mathematically represented as:

[0122]

[0123] In the formula, Indicates the i-th sampling point along the pixel ray The depth parameter; This represents the spatial coordinates of the i-th sampling point.

[0124] sampling points Input a differentiable volume rendering implicit field network, output the volume density of the sampled points. With body radiation ,in This indicates that the sampling point is on the pixel ray. The color emitted from the direction of view.

[0125] Specifically, in step 4, the training objective function and training optimization process of the differentiable volume rendering implicit field network include:

[0126] During training, the training data for the differentiable volume rendering implicit field network includes sparse satellite multi-view images of the target region, imaging parameters corresponding to the multi-view images, pixel rays determined by the imaging parameters, sampling points obtained according to the pixel ray adaptive sampling plan, multi-view image observation information corresponding to the sampling points, and semantic category maps and semantic structure information aligned with the multi-view image pixels. The multi-view image observation information includes the pixel color value corresponding to the pixel ray where the sampling point is located, view information, and / or optional depth coarse prior information. The pixel color value is used to construct the color consistency loss, the semantic category map and semantic structure information are used to construct the semantic consistency loss, boundary geometric constraint loss and / or sampling weight field, and the depth coarse prior information is used to construct the depth consistency loss.

[0127] Training objective function At least include color consistency loss In some preferred embodiments, a depth consistency loss may be further introduced. Semantic consistency loss With boundary geometric constraint loss Mathematically, it is represented as:

[0128]

[0129] in, , and These represent the depth consistency loss respectively. Semantic consistency loss With boundary geometric constraint loss The weighting coefficients can be preset fixed values ​​or adaptively set by a weight field generated from semantic categories and boundary probabilities, in order to work with the sampling plan in step 3 to improve the robustness of the sparse perspective.

[0130] The training and optimization process uses gradient descent and an optimizer to iteratively update the network parameters until convergence.

[0131] Furthermore, step 4, the step of performing 3D reconstruction through microvoid rendering, includes:

[0132] For any pixel ray, obtain all sampling points on the pixel ray, and use the scene representation output by the trained differentiable rendering implicit field network to calculate the pixel-level color prediction value of the pixel ray in the form of differentiable rendering, and / or calculate the depth prediction value of the pixel ray using the expectation form of rendering weights with respect to depth parameters, and finally obtain the pixel-level color prediction value and / or depth prediction value of all pixel rays.

[0133] A depth map of the target region is constructed based on the depth prediction values ​​of all pixel rays; and / or, based on the depth prediction values ​​of all pixel rays and the corresponding imaging parameters, the depth prediction values ​​are converted into three-dimensional points or elevation values ​​in a unified coordinate system, and a digital surface model of the target region is constructed based on the three-dimensional points or elevation values; wherein, the pixel-level color prediction values ​​are used for color consistency training, new perspective synthesis and / or reconstruction quality evaluation, and are not used as the direct basis for constructing the digital surface model.

[0134] Specifically, firstly, pixel-level color prediction is calculated using differentiable volume rendering, mathematically represented as:

[0135]

[0136] In the formula, Represents pixel rays Pixel-level color prediction values; Indicates sampling point The transmission probability; Indicates sampling point The opacity represents the probability that the ray is not absorbed by any preceding point before it propagates from the pixel ray origin to the i-th sampling point; Indicates sampling Volume radiation at a point;

[0137] As a supplement to the above formula:

[0138]

[0139]

[0140] In the formula, Indicates sampling point Volume density; This represents the distance between the i-th and (i+1)-th sampling points along the pixel ray direction; This represents the sampling points preceding the i-th sampling point along the pixel ray direction.

[0141] Secondly, depth prediction is calculated using the expected value of the rendering weights, mathematically expressed as:

[0142]

[0143] In the formula, Represents pixel rays The depth prediction value.

[0144] Preferably, step 4 may also output new perspective synthetic images and / or semantic prediction results and boundary quality related information (optional), such as boundary confidence maps, sampling budget maps or reconstruction uncertainty maps, as needed, for post-processing and automatic quality inspection.

[0145] The implementation of the various embodiments of this invention is based on programmed processing through a system with processor functionality. Therefore, in practical engineering, the technical solutions and functions of the various embodiments of this invention are encapsulated into various modules. Based on this reality, and building upon the above embodiments, the embodiments of this invention provide a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction system. This system is used to execute a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method from the above method embodiments.

[0146] See Figure 2 The system includes:

[0147] The input module is used to acquire sparse satellite multi-view images and imaging parameters of the target area, and determine the pixel rays corresponding to the satellite image pixels and the sampling depth range of the pixel rays based on the imaging parameters.

[0148] The semantic structure acquisition module is used to acquire a semantic category map aligned with the pixels of the multi-view image, and to construct semantic structure information based on the semantic category map. The semantic structure information includes at least semantic category information, semantic boundary probability map, and semantic region structure information.

[0149] The pixel ray sampling module is used to generate an adaptive sampling plan for the pixel ray based on the semantic structure information and the sampling depth range of the pixel ray, and to sample the pixel ray along the line according to the adaptive sampling plan; wherein, the adaptive sampling plan for the pixel ray is used to increase the number of sampling points and / or sampling density for high structure category or semantic boundary regions, and to reduce the number of sampling points and / or sampling density for low structure category or internal regions of the same type of connected domain far from the semantic boundary.

[0150] The 3D reconstruction module is used to input the sampling points into the trained differentiable volume rendering implicit field network, output the scene representation of the sampling points on all pixel rays, and perform 3D reconstruction through differentiable volume rendering. The result of the 3D reconstruction includes at least the depth map and / or digital surface model of the target area. The differentiable volume rendering implicit field network is trained based on the sampling points obtained by sampling and their corresponding multi-view image observation information.

[0151] It should be noted that the system embodiments provided by the present invention are used not only to implement the methods in the above method embodiments, but also to implement the methods in other method embodiments provided by the present invention. The only difference is that corresponding functional modules are set. The principle is basically the same as that of the above system embodiments provided by the present invention. As long as those skilled in the art can improve the modules in the above system embodiments by referring to the specific technical solutions in other method embodiments and combining technical features to obtain corresponding technical means and technical solutions composed of these technical means, on the basis of the above system embodiments, and on the premise of ensuring the practicality of the technical solutions, they can obtain corresponding system-like embodiments for implementing the methods in other method-like embodiments.

[0152] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, located in one place, or distributed across multiple network units. The purpose of this embodiment is achieved by selecting some or all of the modules according to actual needs. Those skilled in the art will understand and implement this without any inventive effort.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] In summary, the present invention relates to a semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method, comprising the following steps: acquiring sparse satellite multi-view images of the same target region and corresponding imaging parameters, and determining pixel rays and their sampling depth ranges based on the imaging parameters; generating or acquiring a semantic category map aligned with image pixels, and constructing semantic structure information including semantic category information S, semantic boundary probability map B, and semantic region structure information R; generating an adaptive ray sampling plan for pixel rays based on the semantic structure information, wherein the sampling plan includes adaptive allocation of sampling points, adaptive setting of sampling depth range and sampling distribution, encrypted sampling of semantic boundary regions, downsampling of low-structure regions, and resampling / rejection rules for cross-category diffusion suppression; sampling rays according to the sampling plan and inputting them into a differentiable rendering implicit field network for rendering and training to obtain pixel-level color prediction and depth prediction; outputting 3D reconstruction results based on the trained model, wherein the results include at least a depth map and / or a digital surface model (DSM). Using the method of this invention, the sampling budget can be focused on high-structure categories and semantic boundary areas such as buildings, roads, bridges, and shorelines, reducing redundant sampling of low-structure areas such as water bodies and homogeneous ground, reducing boundary aliasing and depth overflow, and improving the boundary clarity, integrity and stability of 3D reconstruction under sparse view conditions.

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention.

Claims

1. A semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method, characterized in that, include: Acquire sparse satellite multi-view images and imaging parameters of the target area, and determine the pixel rays corresponding to the satellite image pixels and the sampling depth range of the pixel rays based on the imaging parameters; Obtain a semantic category map aligned with the pixels of the multi-view image, and construct semantic structure information based on the semantic category map; Based on the semantic structure information and the sampling depth range of the pixel ray, an adaptive sampling plan for the pixel ray is generated, and sampling points along the pixel ray are sampled according to the adaptive sampling plan for the pixel ray. The sampling points are input into the trained differentiable volume rendering implicit field network, which outputs the scene representation of the sampling points on all pixel rays and performs three-dimensional reconstruction through differentiable volume rendering. The result of the three-dimensional reconstruction includes at least the depth map and / or digital surface model of the target area. The differentiable volume rendering implicit field network is trained based on the sampling points obtained by sampling and their corresponding multi-view image observation information.

2. The semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 1, characterized in that, The semantic structure information includes at least semantic category information, semantic boundary probability graph, and semantic region structure information; the semantic region structure information includes semantic connected domains, semantic region area, semantic region internal consistency index, and / or semantic region adjacency relationship.

3. The semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 2, characterized in that, The pixel ray adaptive sampling plan is generated based on semantic category information, semantic boundary probability map and semantic region structure information. The pixel ray adaptive sampling plan includes: adaptive allocation of the number of ray sampling points, adaptive setting of sampling depth interval and sampling distribution, encrypted sampling rules for semantic boundary regions, downsampling rules for low-structure regions and / or cross-category diffusion suppression rules.

4. The semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 3, characterized in that, The adaptive allocation of the number of ray sampling points adopts a categorical sampling budget mechanism, which determines the number of sampling points or the range of sampling points for pixel rays based on semantic category information, semantic boundary probability map and semantic region structure information. When a pixel belongs to a building, road, bridge, shoreline, land-water boundary area, or other high-structure category or high-structure area with high abrupt changes, contour abrupt changes, or geometric discontinuities, increase the number of sampling points for the corresponding pixel ray. When a pixel belongs to a body of water, homogeneous ground, bare ground, shaded area, invalid pixel area, or other low-texture, low-structure category or low-structure area, and the pixel is not located in a semantic boundary area, reduce the number of sampling points of the corresponding pixel ray. When a pixel is located in a high-probability region of the semantic boundary probability map or in a connected region of the semantic boundary, the number of sampling points of the corresponding pixel ray is increased. When a pixel is located within the same semantically connected domain and far from the semantic boundary region, reduce the number of sampling points of the corresponding pixel ray, or allow pixel rays within the same semantically connected domain to share sampling parameters.

5. The semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 3, characterized in that, The adaptive setting of the sampling depth range and sampling distribution includes two-stage sampling: The first stage of coarse sampling is performed within the preset sampling depth range to obtain surface candidate regions; The second stage involves constructing a focused sampling distribution based on the surface candidate regions and performing fine sampling; the fine sampling adopts a hybrid distribution sampling method that is a weighted combination of uniform sampling distribution and focused sampling distribution; Specifically, when a pixel belongs to a high-structure category or is located in a semantic boundary region, the weight of the focused sampling distribution is increased and / or the width of the focused sampling window is reduced to increase the sampling density near the surface candidate region; when a pixel belongs to a low-structure category and is far from the semantic boundary region, the weight of the focused sampling distribution is reduced and / or the number of fine sampling points is reduced.

6. A semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 3 or 4, characterized in that, The encryption sampling rules for the semantic boundary region include: when the semantic boundary probability value of a pixel in the semantic boundary probability map exceeds a preset semantic boundary probability threshold, or when it is in a semantic boundary connected region, a boundary expansion sampling window is set on the near end and far end of the surface candidate region corresponding to the pixel ray; the boundary expansion sampling window is adjacent to or partially overlaps with the surface candidate region, and extends at least partially to the depth range outside the surface candidate region; sampling is performed within the boundary expansion sampling window at a higher sampling point density than that in non-boundary regions.

7. A semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 3 or 4, characterized in that, The cross-category diffusion inhibition rule includes at least one of the following: Sampling points located in the sampling depth range with high risk of cross-category aliasing are rejected for sampling, and the sampling budget is transferred to the candidate regions on both sides of the semantic boundary; Alternatively, when determining that the coarse sampling results characterize a cross-category aliasing trend, the candidate surface depths on both sides of the semantic boundary are determined respectively, and a focused sampling window is constructed with the candidate surface depths on both sides as the center. The depth interval between the two focused sampling windows is set as a low sampling probability interval or a rejection sampling interval. Based on the two focused sampling windows, the fine sampling distribution is reconstructed and resampling is performed so that the sampling points are concentrated near the candidate surfaces on both sides of the semantic boundary.

8. The semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 1, characterized in that, The differentiable volume rendering implicit field network is a neural radiation field network, and the scene representation includes at least volume density representation and / or volume radiation representation.

9. The semantic structure-driven satellite image ray adaptive sampling 3D reconstruction method as described in claim 1, characterized in that, The steps involved in 3D reconstruction using microvoid rendering include: For any pixel ray, obtain all sampling points on the pixel ray, and use the scene representation output by the trained differentiable rendering implicit field network to calculate the pixel-level color prediction value of the pixel ray in the form of differentiable rendering, and / or calculate the depth prediction value of the pixel ray using the expectation form of the differentiable rendering weights with respect to the depth parameters, and finally obtain the pixel-level color prediction value and / or depth prediction value of all pixel rays. Based on the depth prediction values ​​of all pixel rays, a depth map of the target area is generated, and / or based on the depth prediction values ​​of all pixel rays and the corresponding imaging parameters, the depth prediction values ​​are converted into three-dimensional points or elevation values ​​in a unified coordinate system, and a digital surface model of the target area is constructed based on the three-dimensional points or elevation values.

10. A semantically structure-driven satellite image ray adaptive sampling 3D reconstruction system, characterized in that, include: The input module is used to acquire sparse satellite multi-view images and imaging parameters of the target area, and determine the pixel rays corresponding to the satellite image pixels and the sampling depth range of the pixel rays based on the imaging parameters. The semantic structure acquisition module is used to acquire a semantic category map aligned with the pixels of the multi-view image, and to construct semantic structure information based on the semantic category map. The semantic structure information includes at least semantic category information, semantic boundary probability map, and semantic region structure information. The pixel ray sampling module is used to generate an adaptive sampling plan for the pixel ray based on the semantic structure information and the sampling depth range of the pixel ray, and to sample the pixel ray along the line according to the adaptive sampling plan; wherein, the adaptive sampling plan for the pixel ray is used to increase the number of sampling points and / or sampling density for high structure category or semantic boundary regions, and to reduce the number of sampling points and / or sampling density for low structure category or internal regions of the same type of connected domain far from the semantic boundary. The 3D reconstruction module is used to input the sampling points into the trained differentiable volume rendering implicit field network, output the scene representation of the sampling points on all pixel rays, and perform 3D reconstruction through differentiable volume rendering. The result of the 3D reconstruction includes at least the depth map and / or digital surface model of the target area. The differentiable volume rendering implicit field network is trained based on the sampling points obtained by sampling and their corresponding multi-view image observation information.

Images