Feedforward gaussian splatting method, device and equipment for multi-view scene reconstruction based on feature matching and adaptive densification, and storage medium

Abstract

The application discloses a feedforward Gaussian splatting method based on feature matching and adaptive densification of multi-view scene reconstruction, a device, equipment and a storage medium, and relates to the technical field of computer vision and three-dimensional reconstruction. The method comprises the following steps: acquiring multi-view image set information and camera parameter information; determining corresponding resolution feature map information based on the image set information and the camera parameter information; performing global and local matching on image points at different resolutions based on the resolution feature map information, and determining matching prior information and matching confidence map information; inputting a predefined Gaussian splatting model and a Gaussian densification model based on the resolution feature map information, the matching prior information and the matching confidence map information, and predicting corresponding Gaussian attributes and Gaussian densification attributes, and determining Gaussian attribute information and Gaussian densification attribute information; and reconstructing to-be-densified anchor points in a scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information, and determining a scene space reconstruction result.

Description

Technical Field

[0001] This application relates to the fields of computer vision and 3D reconstruction technology, and in particular to a feedforward Gaussian splashing method, apparatus, device and storage medium for multi-view scene reconstruction based on feature matching and adaptive densification. Background Technology

[0002] With the rapid development of immersive display technologies such as virtual reality, mixed reality, and extended reality, the demand for technologies that can quickly and with high quality reconstruct 3D scenes from sparse multi-view images is increasing. Therefore, it is necessary to have technologies that can be widely applied to different scenarios and support real-time rendering and interaction while maintaining high visual fidelity to meet current needs.

[0003] Currently, existing approaches achieve photorealistic rendering through implicit volumetric representations or by explicitly representing scenes using Gaussian ellipsoid primitives, supporting real-time rendering. To overcome the efficiency bottleneck caused by scene-by-scene optimization, feedforward reconstruction can be used. For example, methods combining multi-view stereo vision (MVS) to construct cost volumes and employing a pure Transformer architecture for multi-view feature fusion, using the encoder and decoder structure to achieve single-pass inference from sparse input to a 3D Gaussian representation. However, existing approaches are slow in rendering and time-consuming in training. While Gaussian splashing methods render quickly, they still rely on high-quality point cloud initialization and scene-by-scene optimization. Furthermore, when the input viewpoint is limited, overfitting is prone to occur due to ambiguity in the solution space, resulting in inaccurate depth estimation, geometric distortion, and excessive smoothing of details, especially in sparse texture regions and complex occluded scenes, with insufficient high-frequency detail reconstruction capabilities. Therefore, how to perform multi-view scene reconstruction more efficiently and accurately has become an urgent problem to be solved.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a feedforward Gaussian splashing method, apparatus, device, and storage medium for multi-view scene reconstruction based on feature matching and adaptive densification, aiming to solve the technical problem of how to perform multi-view scene reconstruction more efficiently and accurately.

[0006] To achieve the above objectives, this application proposes a feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification. The method includes:

[0007] Acquire multi-view image set information and camera parameter information; Based on the image set information and the camera parameter information, the corresponding resolution feature map information is determined; Based on the resolution feature map information, global and local matching of image points at different resolutions is performed to determine the matching prior information and matching confidence map information; Based on the resolution feature map information, the matching prior information, and the matching confidence map information, the predefined Gaussian splash model and Gaussian densification model are input to predict the corresponding Gaussian attributes and Gaussian densification attributes, thereby determining the Gaussian attribute information and Gaussian densification attribute information. Based on the Gaussian compaction attribute information, the anchor points to be compacted in the scene space corresponding to the Gaussian attribute information are reconstructed, and the scene space reconstruction result is determined.

[0008] In one embodiment, the step of determining the corresponding resolution feature map information based on the image set information and the camera parameter information includes: Obtain the image resolution scale; The image set information and camera parameter information are extracted according to the image resolution scale using a feature pyramid network to obtain resolution feature map information.

[0009] In one embodiment, the step of performing global and local matching of image points at different resolutions based on the resolution feature map information to determine the matching prior information and the matching confidence map information includes: Using the Gaussian kernel function and the resolution feature map information, global matching is performed on image points at different resolutions to determine the global coarse matching result and the global coarse matching confidence. The global coarse matching result and the prior residual and confidence corresponding to the global coarse matching confidence are locally matched by bilinear interpolation to obtain matching prior information and matching confidence map information.

[0010] In one embodiment, the step of using the Gaussian kernel function and the resolution feature map information to perform global matching of image points at different resolutions, and determining the global coarse matching result and the global coarse matching confidence, includes: Based on the resolution feature map information, the corresponding coarse feature information is determined; The posterior mean and covariance of image points at different resolutions are calculated using the Gaussian kernel function and the resolution feature map information to determine the posterior mean and covariance information. Based on the posterior mean information, the covariance information, and the coarse feature information, the predefined global decoder model is input to perform global matching of the corresponding matching coordinates and confidence scores to obtain the global coarse matching result and the global coarse matching confidence score.

[0011] In one embodiment, the step of locally matching the global coarse matching result and the prior residual and confidence level corresponding to the global coarse matching confidence level using bilinear interpolation to obtain matching prior information and matching confidence map information includes: Obtain fine feature information of the source image and distortion feature information of the target image; The global coarse matching result and the global coarse matching confidence are upsampled to the corresponding target initial resolution by bilinear interpolation to determine the optimized value of the target initial resolution. Based on the optimized initial resolution value of the target, the fine feature information of the source image, and the distortion feature information of the target image, the corresponding stitching feature information is determined; Based on the splicing feature information and the target initial resolution optimization value, the predefined convolutional neural optimization model performs local matching on the corresponding prior residuals and confidence scores to obtain matching prior information and matching confidence map information.

[0012] In one embodiment, the step of predicting the corresponding Gaussian attributes and Gaussian compaction attributes based on the resolution feature map information, the matching prior information, and the matching confidence map information by inputting predefined Gaussian splashing models and Gaussian compaction models, and determining the Gaussian attribute information and Gaussian compaction attribute information includes: Based on the resolution feature map information and the matching prior information, the predefined target decoder model is input to predict the corresponding Gaussian attributes to obtain Gaussian attribute information, which includes Gaussian position, opacity, scaling, rotation and color. Based on the matching prior information and the matching confidence map information, the predefined Gaussian compaction model is input to predict the scene confidence distribution in the corresponding Gaussian compaction attribute, thereby obtaining the Gaussian compaction attribute information.

[0013] In one embodiment, the step of predicting the corresponding Gaussian attribute information by inputting the predefined target decoder model based on the resolution feature map information and the matching prior information to obtain Gaussian attribute information includes: Obtain camera spindle information, sampling depth value, and cost volume residual; Based on the camera principal axis information, the sampling depth value, and the resolution feature map information, the corresponding multi-view cost volume information is constructed using a differentiable homography transformation. Based on the resolution feature map information, the multi-view cost volume information, the cost volume residual, and the matching prior information, the predefined target decoder model is input to predict the corresponding Gaussian attributes, thereby obtaining Gaussian attribute information.

[0014] Furthermore, to achieve the above objectives, this application also proposes a feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification, wherein the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification includes: The acquisition module is used to acquire multi-view image set information and camera parameter information; The processing module is used to determine the corresponding resolution feature map information based on the image set information and the camera parameter information; The processing module is also used to perform global and local matching of image points at different resolutions based on the resolution feature map information, and to determine the matching prior information and the matching confidence map information; The execution module is used to input a predefined Gaussian splash model and Gaussian densification model based on the resolution feature map information, the matching prior information and the matching confidence map information, predict the corresponding Gaussian attributes and Gaussian densification attributes, and determine the Gaussian attribute information and Gaussian densification attribute information. The execution module is also used to reconstruct the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information, and determine the scene space reconstruction result.

[0015] Furthermore, to achieve the above objectives, this application also proposes a feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement the steps of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification as described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: This embodiment proposes a feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification. The method acquires multi-view image set information and camera parameter information; determines corresponding resolution feature map information based on the image set information and camera parameter information; performs global and local matching on image points at different resolutions based on the resolution feature map information to determine matching prior information and matching confidence map information; inputs the resolution feature map information, the matching prior information, and the matching confidence map information into a predefined Gaussian splashing model and a Gaussian densification model to predict the corresponding Gaussian attributes and Gaussian densification attributes to determine Gaussian attribute information and Gaussian densification attribute information; and reconstructs the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information to determine the scene space reconstruction result. This application acquires multi-view image set information and camera parameter information, performs global and local matching of image points at different resolutions based on the extracted multi-scale resolution feature maps, and uses Gaussian splashing model and Gaussian compaction model to predict Gaussian properties and Gaussian compaction properties, thereby dynamically identifying anchor points to be compacted in the corresponding scene space for reconstruction. This significantly enhances multi-view geometric consistency, reconstruction detail richness and overall rendering quality, alleviates geometric distortion and floating artifacts caused by sparse viewpoints, enhances the detail representation of complex textures and edge areas, effectively suppresses over-smoothing, and achieves an adaptive balance between reconstruction accuracy and computational resources while maintaining single forward inference efficiency. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a simplified flowchart of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. Figure 2 This is a flowchart illustrating an embodiment of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. Figure 3 This is a schematic diagram of the global matcher for the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. Figure 4This is a schematic diagram of the matching refiner for the feedforward Gaussian splashing method based on feature matching and adaptive densification for multi-view scene reconstruction in this application. Figure 5 This is a flowchart illustrating Embodiment 2 of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. Figure 6 This is a schematic diagram of the module structure of the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification according to an embodiment of this application. Figure 7 This is a schematic diagram of the device structure of the hardware operating environment involved in the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in the embodiments of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0024] The main solution of this application embodiment is as follows: acquiring multi-view image set information and camera parameter information; determining corresponding resolution feature map information based on the image set information and camera parameter information; performing global and local matching of image points at different resolutions based on the resolution feature map information to determine matching prior information and matching confidence map information; inputting predefined Gaussian splashing model and Gaussian densification model based on the resolution feature map information, the matching prior information, and the matching confidence map information to predict the corresponding Gaussian attributes and Gaussian densification attributes to determine Gaussian attribute information and Gaussian densification attribute information; reconstructing the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information to determine the scene space reconstruction result.

[0025] In this embodiment, for ease of description, the following description will focus on a feedforward Gaussian splashing device for recognizing multi-view scene reconstruction based on feature matching and adaptive densification.

[0026] Existing technologies are slow to render and time-consuming to train. While the Gaussian splashing method renders quickly, it still relies on high-quality point cloud initialization and scene-by-scene optimization. At the same time, when the input viewpoint is limited, it is prone to overfitting due to ambiguity in the solution space, which manifests as inaccurate depth estimation, geometric distortion, and excessive smoothing of details. This is especially prominent in sparse texture areas and complex occluded scenes, and the ability to reconstruct high-frequency details is insufficient.

[0027] This application provides a solution, such as Figure 1 As shown, Figure 1 This is a simplified flowchart of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. Based on the image set information and camera parameter information, multi-scale resolution feature map information is extracted through a Feature Pyramid Network (FPN). Using this resolution feature map information, on the one hand, a multi-view cost volume is constructed through differentiable homography transformation to explicitly encode geometric consistency; on the other hand, a coarse-to-fine matching process is performed through a global matcher and a matching refiner to determine matching prior information and matching confidence map information. Based on this, the resolution feature map information, the constructed cost volume, and the matching... Prior information fusion is performed, inputting a predefined attribute decoder, namely the Gaussian splatter model, to predict the position, opacity, rotation, scaling, and color attributes of the Gaussians, thereby determining the Gaussian attribute information. To improve the reconstruction quality of high-frequency details, an adaptive densification mechanism is further used based on the matching confidence map information. This mechanism dynamically identifies weak reconstruction areas as anchor points to be densified using the Gaussian densification model, and generates supplementary Gaussian primitives using a lightweight multilayer perceptron to determine the Gaussian densification attribute information. Based on the Gaussian densification attribute information, the anchor points to be densified in the scene space are reconstructed to determine the scene space reconstruction result.

[0028] As can be seen from the above embodiments, this application acquires multi-view image set information and camera parameter information, performs global and local matching of image points at different resolutions based on the extracted multi-scale resolution feature maps, and uses Gaussian splashing model and Gaussian compaction model to predict Gaussian properties and Gaussian compaction properties, thereby dynamically identifying anchor points to be compacted in the corresponding scene space for reconstruction. This significantly enhances multi-view geometric consistency, reconstruction detail richness and overall rendering quality, alleviates geometric distortion and floating artifacts caused by sparse viewpoints, enhances the detail representation of complex textures and edge areas, effectively suppresses over-smoothing, and achieves an adaptive balance between reconstruction accuracy and computational resources while maintaining single forward inference efficiency.

[0029] Based on this, embodiments of this application provide a feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification, referring to... Figure 2 , Figure 2This is a flowchart illustrating the first embodiment of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application.

[0030] In this embodiment, the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification includes steps S10~S50: Step S10: Obtain multi-view image set information and camera parameter information; It should be noted that the image set information is a set of digital images taken from the same static scene at different spatial locations and from different perspectives. The image set information consists of at least two images, preferably multiple images, such as three, five or more RGB (red, green and blue) images from different perspectives, covering multiple sides of the scene to be reconstructed.

[0031] It is understood that the camera parameter information is a set of camera intrinsic and extrinsic parameters corresponding to each image in the image set information. The camera intrinsic parameters define the imaging geometric properties of the camera, which may include focal length, principal point coordinates, and possible lens distortion coefficients, while the camera extrinsic parameters define the pose of the camera in the world coordinate system, including a 3x3 rotation matrix and a 3x1 translation vector.

[0032] In a specific embodiment, as an optional implementation, the multi-view image set information and camera parameter information can be synchronously acquired through a professional image acquisition system. For example, multiple synchronously triggered camera arrays arranged around the object or scene to be reconstructed can be used for shooting, and the intrinsic and extrinsic parameters of each camera can be accurately calculated through pre-calibration. The calibration process can be completed offline before acquisition, and the calibration results are bound and stored with the acquired image data.

[0033] In a specific embodiment, as another optional implementation method, for dynamically acquired scenarios, such as using a handheld device for surround scanning, the camera parameter information can be calculated from the image set information through a real-time or post-processing SfM algorithm. For example, the SfM algorithm can automatically recover the camera's motion trajectory (extrinsic parameters) and the sparse 3D point cloud of the scene from an unordered image set, and can also estimate the camera's intrinsic parameters. Commonly used SfM tools include COLMAP, OpenMVG, etc.

[0034] In specific embodiments, the acquired image set should cover the key areas of the scene as much as possible, and there should be sufficient overlap between adjacent viewpoints to ensure the reliability of feature matching and the integrity of 3D reconstruction. The accuracy of camera parameters directly affects the accuracy of subsequent cost volume construction and 3D point back projection. Therefore, it is preferable to use results with higher accuracy.

[0035] Step S20: Determine the corresponding resolution feature map information based on the image set information and the camera parameter information; It should be noted that the resolution feature map information is a set of feature maps obtained by performing multi-scale feature extraction on the input image set using a deep convolutional neural network. Each feature map in this set corresponds to a specific spatial resolution, i.e., scale, which is used to capture different levels of semantic and geometric information of the image from coarse to fine, from global context to local details.

[0036] In a specific embodiment, as an optional implementation, the image resolution scale is obtained; the image set information and the camera parameter information are extracted according to the image resolution scale using a feature pyramid network to obtain resolution feature map information. That is, the image resolution scale is used to determine the spatial size of each level of feature map in the feature pyramid relative to the original input image. For example, the scale can be preset to {1 / 4, 1 / 8, 1 / 16, 1 / 32}, meaning that the extracted image set... and corresponding camera parameters For feature maps at 1 / 4, 1 / 8, 1 / 16, and 1 / 32 resolutions, the selection of the image resolution scale needs to strike a balance between feature semantic strength, spatial detail preservation, and computational cost. Subsequently, each image in the multi-view image set can be independently input into a feature pyramid network backbone with shared weights. This backbone network generates original feature maps of different scales at multiple stages, thereby upsampling the smallest feature map. This is then fused with the feature map from the previous stage of the backbone network (which has had its channel number adjusted via 1x1 convolution) through element-wise addition. This process proceeds from top to bottom, generating a set of fused feature maps with decreasing spatial size but progressively enhanced semantic information—the resolution feature map information. For example, the resolution feature map information could be… , This represents the feature map at the i-th scale.

[0037] In a specific embodiment, as another optional implementation, for application scenarios with limited computing resources or extremely high real-time requirements, a lightweight feature extraction network can be used to generate resolution feature maps. In this case, it is necessary to adjust the preset image resolution scale, for example, only retaining smaller or coarser scales such as {1 / 8, 1 / 16, 1 / 32}, in order to achieve a balance between speed and accuracy.

[0038] In specific embodiments, feature maps at coarser scales (e.g., 1 / 16, 1 / 32) have large receptive fields and contain rich global semantic information and scene layout, making them very suitable for global coarse matching to quickly establish large-scale correspondences between images. Feature maps at finer scales (e.g., 1 / 4, 1 / 8) retain more spatial details and edge information, making them suitable for local fine matching, which can correct errors in coarse matching and accurately locate detailed correspondence points.

[0039] In one feasible implementation, step S20 may include steps A11-A12: Step A11: Obtain the image resolution scale; It should be noted that the image resolution scale is a set of predefined scaling parameters for multi-scale feature extraction, such as {1 / 4, 1 / 8, 1 / 16, 1 / 32}, which are used as preset hyperparameter inputs to enable the network to generate a series of fused feature maps with decreasing spatial resolution and progressively enhanced semantic information from top to bottom.

[0040] Step A12: Extract the image set information and the camera parameter information according to the image resolution scale using a feature pyramid network to obtain resolution feature map information.

[0041] Understandably, the Feature Pyramid Network is a network structure that can efficiently construct multi-scale feature representations of images. Through top-down paths and lateral connections, it fuses the high semantic features of deep networks with the high-resolution features of shallow networks, thereby generating multi-level feature maps with strong semantic information and different spatial resolutions.

[0042] Step S30: Based on the resolution feature map information, perform global and local matching of image points at different resolutions to determine the matching prior information and the matching confidence map information; It should be noted that the matching prior information is a correspondence field obtained by performing dense feature matching on images from different perspectives. Essentially, it is a geometric constraint that indicates the most likely two-dimensional coordinates of a pixel in the source image under the reference perspective in the target image under other perspectives.

[0043] It is understood that the matching confidence map information is a metric map that characterizes the reliability of each matching point. The value of each position is between 0 and 1, quantifying the credibility of the matching result at that point. High confidence areas usually correspond to rich textures, no occlusion, and good matching consistency, while low confidence areas correspond to weak textures, repetitive textures, occlusion boundaries, or discontinuous parallax.

[0044] In a specific embodiment, as an optional implementation, the Gaussian kernel function and the resolution feature map information are used to perform global matching of image points at different resolutions to determine the global coarse matching result and the global coarse matching confidence level, i.e., as shown below. Figure 3 As shown, Figure 3 This diagram illustrates the global matcher for the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. The global matcher transforms global matching into a probabilistic regression problem involving embedded coordinates, implemented through Gaussian process modeling. An exponential cosine similarity kernel can be used to construct the Gaussian kernel function, which exhibits stronger robustness in dense matching. The Gaussian kernel function can be calculated as follows:

[0045] Based on resolution feature map information, coarse features of the target image corresponding to image points at different resolutions and their true embedded coordinates are obtained. As observed samples, the posterior mean and covariance of the coordinates corresponding to the coarse features of the source image are calculated, and expressed as:

[0046]

[0047] The posterior mean is the most likely matching coordinate. Represents the kernel matrix. This represents the standard deviation of the observation noise.

[0048] Finally, the posterior mean and coarse features of the source image are used by the decoder to output a global coarse matching result. Global coarse matching confidence .

[0049] The global coarse matching result and the prior residual and confidence level corresponding to the global coarse matching confidence level are locally matched using bilinear interpolation to obtain matching prior information and matching confidence map information, i.e. Figure 4 As shown, Figure 4 This diagram illustrates the matching refiner for the feedforward Gaussian splashing method based on feature matching and adaptive densification for multi-view scene reconstruction in this application. Since the feature matching resolution output by the global matcher is low and cannot meet the accuracy requirements of geometric estimation, the goal of the matching refiner is to refine the coarse matching to the original resolution through multi-scale optimization, while simultaneously correcting matching errors and improving the matching accuracy of local details. This refiner can improve the accuracy of the global coarse matching result output by the global matcher. Global coarse matching confidence The resolution of the current fine features is upsampled using bilinear interpolation as the initial value for optimization. Subsequently, the matching refiner refines the fine features of the source image. Distortion features of the target image The local correlation and warped embedding of the target image corresponding to the optimized initial resolution are input into the convolutional neural network-based optimization network, which outputs the residuals of matching and confidence, thus obtaining refined matching prior information of the original resolution. and matching confidence graph information .

[0050] In a specific embodiment, as another optional implementation, in scenarios with significant changes in illumination or differences in viewing angle, a visual module can be introduced in the feature extraction stage or within the matching network to enhance the discriminativeness of features using its self-attention mechanism. Alternatively, in the global matcher, in addition to using a Gaussian process, a learnable matching cost volume and an iterative optimization strategy can be combined. For applications with high real-time requirements, the number of stages of network matching from coarse to fine can be simplified. For example, only one global matching can be performed at the 1 / 8 scale and one local refinement can be performed at the 1 / 4 scale. After performing only global matching, upsampling can be performed directly, sacrificing some accuracy for speed. In this case, the accuracy of matching prior information will decrease accordingly, but the overall process will be accelerated.

[0051] Step S40: Based on the resolution feature map information, the matching prior information and the matching confidence map information, input the predefined Gaussian splash model and Gaussian compaction model to predict the corresponding Gaussian attributes and Gaussian compaction attributes, and determine the Gaussian attribute information and Gaussian compaction attribute information. It should be noted that the Gaussian attribute information is a set of parameters required to characterize each Gaussian primitive in the 3D Gaussian splash representation, including information such as Gaussian position, opacity, scaling, rotation, and color. The Gaussian densification attribute information is the confidence distribution information that enables the Gaussian primitive to be adaptively densified in the scene space, and is used to identify and reconstruct weak regions and generate new Gaussian primitives.

[0052] In a specific embodiment, as an optional implementation, camera principal axis information, sampling depth values, and cost volume residuals are obtained; based on the camera principal axis information, the sampling depth values, and the resolution feature map information, corresponding multi-view cost volume information is constructed using a differentiable homography transformation; based on the resolution feature map information, the multi-view cost volume information, the cost volume residuals, and the matching prior information, the corresponding Gaussian attributes are predicted using a predefined target decoder model to obtain Gaussian attribute information. That is, multi-view cost volume information can be constructed using image features extracted by the Feature Pyramid Network (FPN) based on a differentiable homography transformation, wherein the multi-view cost volume information is represented as:

[0053] in, Indicates camera spindle information, This represents the sampling depth value.

[0054] This transformation will change the view Feature mapping to view Constructing the view cone space Individual cost body At this point, the dimension of the cost body is Where H and W are the feature map space dimensions, D is the number of depth samples, and C is the number of feature channels.

[0055] resolution feature map information Multi-view cost body information Matching prior information The input decoder predicts Gaussian properties, and a 2D convolutional network outputs a cost volume residual to refine the cost volume. The depth map is then calculated based on the optimized cost volume. Gaussian position is obtained through back projection. The remaining properties of Gaussian primitives include opacity. Scaling Rotation and color It can be predicted using different decoder heads and different activation functions, as follows:

[0056]

[0057]

[0058]

[0059] in The prediction networks correspond to opacity, scaling, rotation, and color, respectively. The rotation parameter is represented by a quaternion and its effectiveness is ensured through normalization.

[0060] Based on the prior matching information and the confidence map information, the predefined Gaussian compaction model is input to predict the scene confidence distribution in the corresponding Gaussian compaction attributes, thereby obtaining Gaussian compaction attribute information. This allows for the statistical analysis of the confidence distribution across the entire scene. A lightweight confidence evaluation network is used to fuse and encode the input information, extracting confidence features of reconstruction reliability for each region of the scene. The mean and variance of the confidence of all matching in the current scene are calculated to obtain Gaussian compaction attribute information. Based on this, an adaptive threshold is dynamically generated to distinguish between high-confidence and low-confidence regions. At this point, the model outputs Gaussian compaction attribute information containing the adaptive threshold and the generated binary region segmentation mask. This mask clearly identifies the anchor point regions in the scene that need to be compacted by Gaussian compaction.

[0061] In a specific embodiment, as another optional implementation, if there are few input viewpoints or sparse scene textures, the confidence threshold can be appropriately reduced and the densification range expanded to enhance the robustness of reconstruction. Conversely, in scenes with sufficient viewpoints and rich textures, the threshold can be increased to reduce unnecessary computational overhead and achieve an adaptive balance between reconstruction accuracy and computational efficiency.

[0062] In one feasible implementation, step S40 may include steps B11-B12: Step B11: Based on the resolution feature map information and the matching prior information, input the predefined target decoder model to predict the corresponding Gaussian attributes and obtain Gaussian attribute information, which includes Gaussian position, opacity, scaling, rotation and color. It should be noted that the Gaussian attribute information is a set of differentiable parameters used to explicitly represent a 3D scene, defining the geometry, appearance, and interaction with light for each Gaussian ellipsoid primitive in the scene.

[0063] Understandably, Gaussian position determines the center point of the primitive in space, opacity controls its weight in the final rendering, scaling and rotation define the shape and orientation of the ellipsoid, and color encodes its view-dependent appearance information.

[0064] In one feasible implementation, step B11 may include steps C11-C13: Step C11: Obtain camera principal axis information, sampling depth value, and cost volume residual; It should be noted that the camera principal axis information refers to the ray direction from the camera's optical center to the center of the imaging plane, used to define the viewpoint and construct geometric transformations. The sampled depth values ​​are a set of distance values ​​discretely sampled along the camera principal axis in the depth direction, used to construct a discretized depth hypothesis plane. The cost volume residual is a feature tensor predicted by a neural network to refine the initial cost volume, aiming to correct for errors caused by feature matching or multiple views. Figure 1 The initial error caused by consistency calculation is reduced, thus obtaining a more accurate geometric code.

[0065] Step C12: Construct the corresponding multi-view cost volume information based on the camera principal axis information, the sampling depth value, and the resolution feature map information using a differentiable homography transformation. It should be noted that the multi-view cost volume information is a three-dimensional data structure that explicitly encodes the geometric consistency between multi-view images. For each candidate sampling depth value, the feature map of the reference view is projected onto the coordinate system of other source views through the plane homography matrix corresponding to that depth, and similarity calculation, such as variance or mean, is performed with the features of the source views. The similarity results under all depth assumptions are stacked to form a cost volume with dimensions [H, W, D, C], where H and W are the spatial dimensions of the feature map, D is the number of depth samples, and C is the number of feature or cost channels. The value at each position of this volume can characterize the probability that the assumption of depth d at pixel (x,y) is true, i.e., the degree of multi-view consistency.

[0066] Step C13: Based on the resolution feature map information, the multi-view cost volume information, the cost volume residual, and the matching prior information, input the predefined target decoder model to predict the corresponding Gaussian attributes and obtain Gaussian attribute information.

[0067] Understandably, the target decoder model is a deep neural network that can output Gaussian attribute parameters at each preset or predicted location, completing the mapping from two-dimensional image features to a three-dimensional explicit Gaussian representation.

[0068] Step B12: Based on the matching prior information and the matching confidence map information, input a predefined Gaussian compaction model to predict the scene confidence distribution in the corresponding Gaussian compaction attribute, and obtain Gaussian compaction attribute information.

[0069] It is understandable that the scene confidence distribution is a quantitative evaluation map of the reliability of the entire 3D scene reconstruction quality. It originates from the matching confidence map output in the feature matching stage and is used for adaptive resource allocation.

[0070] Step S50: Based on the Gaussian densification attribute information, reconstruct the anchor points to be densified in the scene space corresponding to the Gaussian attribute information, and determine the scene space reconstruction result.

[0071] It should be noted that the scene space reconstruction result is a complete and detailed explicit representation of the generated 3D scene, presented in the form of a set of 3D Gaussian primitives. Each Gaussian primitive is defined by its spatial position, opacity, anisotropic scaling, rotation, and color, etc. It contains the basic Gaussian attribute information predicted by the target decoder model, and incorporates newly generated Gaussian primitives supplemented in the weak reconstruction areas based on an adaptive densification mechanism, thereby achieving high-fidelity and detailed 3D reconstruction of the scene captured by the original multi-view images.

[0072] It is understandable that the anchor points to be densified are the basic Gaussian primitives corresponding to the 3D spatial regions with insufficient reconstruction quality or high uncertainty under the current basic Gaussian representation, identified based on Gaussian densification attribute information. These anchor points may be located in regions with complex textures (such as fabrics and hair), geometric edges, depth discontinuities, or regions with low feature matching confidence due to viewpoint occlusion or lighting changes. They are marked as the starting point for densification processing. That is, in the spatial neighborhood around them, additional Gaussian primitives need to be generated to enhance the geometric and appearance detail representation capabilities of the local region.

[0073] In a specific embodiment, low-confidence regions can be marked as anchor points to be densified based on Gaussian densification attribute information and the Gaussian attribute information, while high-confidence regions are considered as well-represented regions. During the densification execution phase, two parallel decoders work together. During densification, two parallel decoders process the data separately. The base decoder generates Gaussian attributes that do not need densification, and the densification decoder predicts M new Gaussian attributes from the anchor point features. The positions of the new Gaussian attributes are generated by adding offsets to the anchor point positions. The offset is adaptively adjusted based on the confidence value. The lower the confidence, the larger the range of possible offset values, so as to better represent the local geometric structure.

[0074] This embodiment proposes a feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification. The method acquires multi-view image set information and camera parameter information; determines corresponding resolution feature map information based on the image set information and camera parameter information; performs global and local matching on image points at different resolutions based on the resolution feature map information to determine matching prior information and matching confidence map information; inputs the resolution feature map information, the matching prior information, and the matching confidence map information into a predefined Gaussian splashing model and a Gaussian densification model to predict the corresponding Gaussian attributes and Gaussian densification attributes to determine Gaussian attribute information and Gaussian densification attribute information; and reconstructs the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information to determine the scene space reconstruction result. This invention addresses the technical challenge of more efficient and accurate multi-view scene reconstruction. Compared to existing technologies, this application acquires multi-view image set information and camera parameter information, performs global and local matching of image points at different resolutions based on the extracted multi-scale resolution feature maps, and uses Gaussian splashing and Gaussian compaction models to predict Gaussian and Gaussian compaction properties. This dynamically identifies anchor points to be compacted in the corresponding scene space for reconstruction, significantly enhancing multi-view geometric consistency, reconstruction detail richness, and overall rendering quality. It also alleviates geometric distortion and floating artifacts caused by sparse viewpoints, enhances the detail representation of complex textures and edge areas, effectively suppresses over-smoothing, and achieves an adaptive balance between reconstruction accuracy and computational resources while maintaining single-pass forward inference efficiency.

[0075] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the first embodiment can be referred to the above description, and will not be repeated hereafter.

[0076] In this embodiment, refer to Figure 5 , Figure 5 This is a flowchart illustrating Embodiment 2 of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in this application. Step S30 specifically includes steps S31 to S32: Step S31: Use the Gaussian kernel function and the resolution feature map information to perform global matching of image points at different resolutions, and determine the global coarse matching result and the global coarse matching confidence. It should be noted that the global coarse matching result is a preliminary pixel-level or feature point-level correspondence established between low-resolution feature maps from different viewpoints through a probabilistic regression model. For each position on the source viewpoint feature map, it provides the most likely coordinates corresponding to it on the target viewpoint feature map. The global coarse matching confidence is a quantitative assessment of the reliability of each matching result in the global coarse matching result. It is a confidence map with the same resolution as the feature map. The value of each position is between 0 and 1. The higher the value, the more reliable the match at that point. Conversely, the lower the value, the higher the uncertainty of the match or the possibility of error.

[0077] Understandably, the Gaussian kernel function is a mathematical function used to measure the similarity between two feature vectors. It can map features to a high-dimensional space and evaluate the matching probability by calculating their distance under a Gaussian distribution. It exponentially amplifies the difference in cosine similarity, making strong matching points more prominent. At the same time, it has good robustness to noise and local deformation, making it very suitable for dense matching tasks. In the global matcher, this kernel function is used to construct the covariance matrix of the Gaussian process, thereby performing probabilistic regression prediction of the matching position based on feature similarity.

[0078] In a specific embodiment, as an optional implementation, coarse feature information is determined based on the resolution feature map information; the posterior mean and covariance of image points at different resolutions are calculated using a Gaussian kernel function and the resolution feature map information to determine the posterior mean information and covariance information; based on the posterior mean information, the covariance information, and the coarse feature information, a predefined global decoder model is input to perform global matching of the corresponding matching coordinates and confidence scores to obtain the global coarse matching result and the global coarse matching confidence score. That is, based on the resolution feature map information, a feature map at a coarser scale is selected as input to extract coarse feature information from the source view image and coarse feature information from the target view image, respectively, and the coarse feature information between the source features and the target features is calculated using a predefined Gaussian kernel function. The correlation is used to model the matching problem as a Gaussian process regression problem. Specifically, the known real feature coordinates on the target view feature map can be used as observation samples. The posterior probability distribution of the coordinates corresponding to each position on the source view feature map is inferred through the Gaussian process. The mean of the posterior distribution is the most likely matching coordinate estimate. Its variance or some measure related to the mean is transformed by the decoder to generate the global coarse matching confidence. The posterior mean information and the coarse feature information of the source view are input into a pre-trained global decoder model. The decoder further integrates the feature context information and outputs the optimized global coarse matching result and the global coarse matching confidence. For example, when reconstructing an indoor scene, this step can quickly and roughly align large objects such as windows and furniture in images from different viewpoints.

[0079] In a specific embodiment, as another optional implementation, when there are changes in illumination and large differences in viewing angle, when constructing the Gaussian kernel function, not only cosine similarity between features can be considered, but also exponential weight decay of feature differences can be introduced, or learnable kernel function parameters can be used. In addition, the global decoder can adopt an attention-based architecture, which enables it to use long-range dependencies in the feature map to correct locally blurred matches.

[0080] In specific embodiments, for applications prioritizing extreme speed, a global match can be performed only once at the coarsest scale, simplifying the decoder structure. The posterior mean can be directly used as the coarse matching result, and a binarized confidence map can be generated using the reciprocal of the posterior variance or a fixed threshold. Although some accuracy is lost, the matching speed can be greatly improved.

[0081] In one feasible implementation, step S31 may include steps D11 to D13: Step D11: Determine the corresponding coarse feature information based on the resolution feature map information; It should be noted that the coarse feature information is a feature map at a coarser spatial scale selected from the resolution feature map information, such as a feature map at 1 / 16 or 1 / 32 resolution of the original image. Although the spatial details of these feature maps are simplified due to more downsampling and deep network processing, they contain stronger and more abstract semantic information and have a lighter computational burden.

[0082] Step D12: Calculate the posterior mean and covariance of image points at different resolutions using the Gaussian kernel function and the resolution feature map information to determine the posterior mean information and covariance information; It should be noted that the posterior mean information represents the most likely value of the matching coordinates under given observation features, i.e., the predicted coarse matching position, while the posterior covariance quantifies the corresponding confidence range, and its diagonal elements can be directly associated with the confidence level of the match.

[0083] Step D13: Based on the posterior mean information, the covariance information, and the coarse feature information, input the predefined global decoder model to perform global matching of the corresponding matching coordinates and confidence scores, and obtain the global coarse matching result and the global coarse matching confidence score.

[0084] Understandably, the global decoder model is a lightweight neural network module, such as consisting of several fully connected layers or convolutional layers. Its function is to receive and fuse posterior mean and covariance information as well as the original coarse feature information, and decode these high-dimensional features into a directly usable matching output. For example, the global decoder model uses the aforementioned posterior statistics as a strong geometric prior, combines the contextual semantics provided by the original features, and outputs the global coarse matching result and global coarse matching confidence through nonlinear mapping.

[0085] Step S32: The global coarse matching result and the prior residual and confidence corresponding to the global coarse matching confidence are locally matched by bilinear interpolation to obtain matching prior information and matching confidence map information.

[0086] Understandably, bilinear interpolation is a standard computational method for upsampling two-dimensional grid data (such as images and feature maps). When it is necessary to obtain the value of a non-integer coordinate location, the value of that location is estimated by finding the values ​​of the four nearest known integer coordinate points around that location and performing two linear interpolations based on their relative distances (first in the horizontal direction and then in the vertical direction, or vice versa). For example, bilinear interpolation is used to magnify the global coarse matching results (coordinate field) and global coarse matching confidence map at a lower resolution (such as 1 / 16 scale) to the same spatial size as the fine feature map at a higher resolution (such as 1 / 8 or 1 / 4 scale) that is currently being processed.

[0087] In a specific embodiment, as an optional implementation, fine feature information of the source image and distortion feature information of the target image are obtained; the global coarse matching result and the global coarse matching confidence are upsampled to the corresponding target initial resolution using bilinear interpolation to determine the target initial resolution optimization value; based on the target initial resolution optimization value, the fine feature information of the source image, and the distortion feature information of the target image, the corresponding stitching feature information is determined; based on the stitching feature information and the target initial resolution optimization value, a predefined convolutional neural optimization model is input to perform local matching of the corresponding prior residuals and confidence, obtaining matching prior information and matching confidence map information. That is, using the bilinear interpolation algorithm, the global coarse matching result and the global coarse matching confidence map can be upsampled to a resolution that matches the fine feature map at the current scale. The upsampled coarse matching result and confidence constitute the target initial resolution at the current resolution. The target initial resolution is optimized. Then, the fine feature information of the source image and the distortion feature information of the target image at the same scale are obtained. The fine feature information of the source image, the distortion feature information of the target image, and the local correlation features of the target image obtained after upsampling are embedded with the distortion position and spliced ​​to form rich spliced ​​feature information. This is then input into a predefined optimization model based on convolutional neural network, i.e., the matching refiner, to predict the residual of the matching coordinates (i.e., fine adjustment amount) and the residual of the confidence (i.e., confidence correction amount). The initial upsampling value is added to the residual predicted by the network to obtain more accurate matching prior information (fine matching coordinate field) and matching confidence map information (fine confidence map) at the current fine resolution. For example, the window outline matched at the 1 / 16 scale can be further aligned with the fine structure of the window frame and the reflective details on the glass at the 1 / 8 scale after this step of upsampling and optimization.

[0088] In a specific embodiment, as another optional implementation, in order to improve the fidelity of upsampling, a more advanced upsampling method can be used, such as learning-based upsampling to replace fixed bilinear interpolation, for example, transposed convolution or subpixel convolution, so that the upsampling process can adaptively learn the mapping relationship from low-resolution matching to high-resolution matching. In scenarios where computing resources are extremely limited or latency is extremely sensitive, local fine matching can also be simplified or even skipped, that is, only bilinear interpolation is used to directly upsample the matching result at the coarsest scale to the target resolution as a matching prior, which can obtain a usable matching baseline with extremely low computational cost.

[0089] In one feasible implementation, step S32 may include steps E11 to E14: Step E11: Obtain the fine feature information of the source image and the distortion feature information of the target image; It should be noted that the fine feature information of the source image is a feature map with high spatial resolution extracted from the feature pyramid network, such as 1 / 4 or 1 / 8 scale of the original image, which retains rich local texture and detail information. The distortion feature information of the target image is a feature representation obtained by spatially transforming the feature map of the target image through a differentiable sampling operation based on the currently estimated coarse matching. The purpose is to align the target features to the viewpoint of the source image.

[0090] Step E12: Upsample the global coarse matching result and the global coarse matching confidence to the corresponding target initial resolution using bilinear interpolation to determine the optimized value of the target initial resolution; It should be noted that the target initial resolution optimization value is the initial matching field and confidence map obtained by upsampling the global coarse matching result and global coarse matching confidence obtained by the global matcher at the coarse scale through bilinear interpolation to the same spatial resolution as the current fine feature map.

[0091] Step E13: Determine the corresponding stitching feature information based on the target initial resolution optimization value, the source image fine feature information, and the target image distortion feature information; It should be noted that the spliced ​​feature information is a fusion input feature constructed for the local matching refinement network. It is formed by splicing multi-source information through the channel dimension, which can comprehensively evaluate and correct the matching results.

[0092] Step E14: Based on the splicing feature information and the target initial resolution optimization value, input the predefined convolutional neural optimization model to perform local matching of the corresponding prior residuals and confidence scores to obtain matching prior information and matching confidence map information.

[0093] Understandably, the convolutional neural optimization model is a lightweight regression network built on a convolutional neural network. It receives spliced ​​feature information and the target initial resolution optimization value as a reference. Through multiple convolutions and nonlinear activations, it predicts the residual between the matching coordinates and the confidence score pixel by pixel, which is the increment that needs to be corrected. Then, the residual is added to the initial value to obtain the refined matching prior information and matching confidence map information.

[0094] This embodiment proposes a feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification. It utilizes a Gaussian kernel function and the resolution feature map information to globally match image points at different resolutions, determining a global coarse matching result and a global coarse matching confidence level. Then, it uses bilinear interpolation to locally match the prior residuals and confidence levels corresponding to the global coarse matching result and the global coarse matching confidence level, obtaining matching prior information and matching confidence map information. This solves the technical problem of how to perform multi-view scene reconstruction more efficiently and accurately. Compared with existing technologies, this application uses a Gaussian kernel function for global matching at low resolution to generate a coarse matching field and confidence map. It then upsamples this to high resolution using bilinear interpolation as initialization and constructs fused features. These features are input into a lightweight convolutional neural network for local optimization, predicting and correcting the matching and confidence residuals, and outputting the matching prior and confidence map. This significantly improves matching accuracy and efficiency, effectively overcomes the ambiguity caused by sparse texture and viewpoint changes, fully integrates geometric priors and high-resolution details, and enhances the viewpoint consistency and detail fidelity of the reconstruction results.

[0095] This application also provides a feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification. Please refer to [link / reference]. Figure 6 The feedforward Gaussian splashing device based on feature matching and adaptive densification for multi-view scene reconstruction includes: The acquisition module 10 is used to acquire multi-view image set information and camera parameter information; Processing module 20 is used to determine the corresponding resolution feature map information based on the image set information and the camera parameter information; The processing module 20 is also used to perform global and local matching of image points at different resolutions based on the resolution feature map information, and to determine the matching prior information and the matching confidence map information; Execution module 30 is used to input predefined Gaussian splashing model and Gaussian densification model based on the resolution feature map information, the matching prior information and the matching confidence map information to predict the corresponding Gaussian attributes and Gaussian densification attributes, and determine the Gaussian attribute information and Gaussian densification attribute information. The execution module 30 is further configured to reconstruct the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information, and determine the scene space reconstruction result.

[0096] The processing module 20 is also used to obtain the image resolution scale; The image set information and camera parameter information are extracted according to the image resolution scale using a feature pyramid network to obtain resolution feature map information.

[0097] The processing module 20 is further configured to use the Gaussian kernel function and the resolution feature map information to perform global matching of image points at different resolutions, and determine the global coarse matching result and the global coarse matching confidence. The global coarse matching result and the prior residual and confidence corresponding to the global coarse matching confidence are locally matched by bilinear interpolation to obtain matching prior information and matching confidence map information.

[0098] The processing module 20 is further configured to determine corresponding coarse feature information based on the resolution feature map information; The posterior mean and covariance of image points at different resolutions are calculated using the Gaussian kernel function and the resolution feature map information to determine the posterior mean and covariance information. Based on the posterior mean information, the covariance information, and the coarse feature information, the predefined global decoder model is input to perform global matching of the corresponding matching coordinates and confidence scores to obtain the global coarse matching result and the global coarse matching confidence score.

[0099] The processing module 20 is also used to acquire fine feature information of the source image and distortion feature information of the target image; The global coarse matching result and the global coarse matching confidence are upsampled to the corresponding target initial resolution by bilinear interpolation to determine the optimized value of the target initial resolution. Based on the optimized initial resolution value of the target, the fine feature information of the source image, and the distortion feature information of the target image, the corresponding stitching feature information is determined; Based on the splicing feature information and the target initial resolution optimization value, the predefined convolutional neural optimization model performs local matching on the corresponding prior residuals and confidence scores to obtain matching prior information and matching confidence map information.

[0100] The execution module 30 is further configured to input the predefined target decoder model based on the resolution feature map information and the matching prior information to predict the corresponding Gaussian attributes, thereby obtaining Gaussian attribute information, which includes Gaussian position, opacity, scaling, rotation and color. Based on the matching prior information and the matching confidence map information, the predefined Gaussian compaction model is input to predict the scene confidence distribution in the corresponding Gaussian compaction attribute, thereby obtaining the Gaussian compaction attribute information.

[0101] The execution module 30 is also used to acquire camera spindle information, sampling depth value and cost volume residual; Based on the camera principal axis information, the sampling depth value, and the resolution feature map information, the corresponding multi-view cost volume information is constructed using a differentiable homography transformation. Based on the resolution feature map information, the multi-view cost volume information, the cost volume residual, and the matching prior information, the predefined target decoder model is input to predict the corresponding Gaussian attributes, thereby obtaining Gaussian attribute information.

[0102] The feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification provided in this application, employing the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in the above embodiments, can solve the technical problem of how to perform multi-view scene reconstruction more efficiently and accurately. Compared with the prior art, the beneficial effects of the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification provided in this application are the same as the beneficial effects of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification provided in the above embodiments, and other technical features in the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0103] This application provides a feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification. The feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in the first embodiment described above.

[0104] The following is for reference. Figure 7 This document illustrates a structural schematic of a feedforward Gaussian splashing device suitable for implementing multi-view scene reconstruction based on feature matching and adaptive densification in the embodiments of this application. The feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification in the embodiments of this application can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7 The feedforward Gaussian splashing device shown is merely an example of multi-view scene reconstruction based on feature matching and adaptive densification, and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0105] like Figure 7 As shown, the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the feedforward Gaussian splashing device based on feature matching and adaptive densification for multi-view scene reconstruction to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a feedforward Gaussian splashing device with various systems for feature matching and adaptive densification for multi-view scene reconstruction, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems can be implemented alternatively.

[0106] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0107] The feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification provided in this application, employing the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in the above embodiments, can solve the technical problem of how to perform multi-view scene reconstruction more efficiently and accurately. Compared with the prior art, the beneficial effects of the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification provided in this application are the same as the beneficial effects of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification provided in the above embodiments, and other technical features in the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0108] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0109] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0110] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification in the above embodiments.

[0111] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0112] The aforementioned computer-readable storage medium may be included in a feedforward Gaussian splashing device based on feature matching and adaptive densification for multi-view scene reconstruction; or it may exist independently and not assembled into a feedforward Gaussian splashing device based on feature matching and adaptive densification for multi-view scene reconstruction.

[0113] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification, the feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification performs the following: acquires multi-view image set information and camera parameter information; determines corresponding resolution feature map information based on the image set information and camera parameter information; performs global and local matching of image points at different resolutions based on the resolution feature map information to determine matching prior information and matching confidence map information; inputs a predefined Gaussian splashing model and Gaussian densification model based on the resolution feature map information, the matching prior information, and the matching confidence map information to predict the corresponding Gaussian attributes and Gaussian densification attributes to determine Gaussian attribute information and Gaussian densification attribute information; and reconstructs the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information to determine the scene space reconstruction result.

[0114] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0115] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0116] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0117] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the aforementioned feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification. This method can solve the technical problem of how to perform multi-view scene reconstruction more efficiently and accurately. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification provided in the above embodiments, and will not be repeated here.

[0118] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification, characterized in that, The method includes: Acquire multi-view image set information and camera parameter information; Based on the image set information and the camera parameter information, the corresponding resolution feature map information is determined; Based on the resolution feature map information, global and local matching of image points at different resolutions is performed to determine the matching prior information and matching confidence map information; Based on the resolution feature map information, the matching prior information, and the matching confidence map information, the predefined Gaussian splash model and Gaussian densification model are input to predict the corresponding Gaussian attributes and Gaussian densification attributes, thereby determining the Gaussian attribute information and Gaussian densification attribute information. Based on the Gaussian compaction attribute information, the anchor points to be compacted in the scene space corresponding to the Gaussian attribute information are reconstructed, and the scene space reconstruction result is determined.

2. The method as described in claim 1, characterized in that, The step of determining the corresponding resolution feature map information based on the image set information and the camera parameter information includes: Obtain the image resolution scale; The image set information and camera parameter information are extracted according to the image resolution scale using a feature pyramid network to obtain resolution feature map information.

3. The method as described in claim 1, characterized in that, The step of performing global and local matching of image points at different resolutions based on the resolution feature map information, and determining the matching prior information and matching confidence map information, includes: Using the Gaussian kernel function and the resolution feature map information, global matching is performed on image points at different resolutions to determine the global coarse matching result and the global coarse matching confidence. The global coarse matching result and the prior residual and confidence corresponding to the global coarse matching confidence are locally matched by bilinear interpolation to obtain matching prior information and matching confidence map information.

4. The method as described in claim 3, characterized in that, The step of using the Gaussian kernel function and the resolution feature map information to perform global matching of image points at different resolutions, and determining the global coarse matching result and the global coarse matching confidence level, includes: Based on the resolution feature map information, the corresponding coarse feature information is determined; The posterior mean and covariance of image points at different resolutions are calculated using the Gaussian kernel function and the resolution feature map information to determine the posterior mean and covariance information. Based on the posterior mean information, the covariance information, and the coarse feature information, the predefined global decoder model is input to perform global matching of the corresponding matching coordinates and confidence scores to obtain the global coarse matching result and the global coarse matching confidence score.

5. The method as described in claim 3, characterized in that, The step of locally matching the global coarse matching result and the prior residual and confidence level corresponding to the global coarse matching confidence level using bilinear interpolation to obtain matching prior information and matching confidence map information includes: Obtain fine feature information of the source image and distortion feature information of the target image; The global coarse matching result and the global coarse matching confidence are upsampled to the corresponding target initial resolution by bilinear interpolation to determine the optimized value of the target initial resolution. Based on the optimized initial resolution value of the target, the fine feature information of the source image, and the distortion feature information of the target image, the corresponding stitching feature information is determined; Based on the splicing feature information and the target initial resolution optimization value, the predefined convolutional neural optimization model performs local matching on the corresponding prior residuals and confidence scores to obtain matching prior information and matching confidence map information.

6. The method as described in claim 1, characterized in that, The step of predicting the corresponding Gaussian attributes and Gaussian compaction attributes by inputting the predefined Gaussian splash model and Gaussian compaction model based on the resolution feature map information, the matching prior information, and the matching confidence map information, and determining the Gaussian attribute information and Gaussian compaction attribute information includes: Based on the resolution feature map information and the matching prior information, the predefined target decoder model is input to predict the corresponding Gaussian attributes to obtain Gaussian attribute information, which includes Gaussian position, opacity, scaling, rotation and color. Based on the matching prior information and the matching confidence map information, the predefined Gaussian compaction model is input to predict the scene confidence distribution in the corresponding Gaussian compaction attribute, thereby obtaining the Gaussian compaction attribute information.

7. The method as described in claim 6, characterized in that, The step of predicting the corresponding Gaussian attributes based on the resolution feature map information and the matching prior information input into a predefined target decoder model to obtain Gaussian attribute information includes: Obtain camera spindle information, sampling depth value, and cost volume residual; Based on the camera principal axis information, the sampling depth value, and the resolution feature map information, the corresponding multi-view cost volume information is constructed using a differentiable homography transformation. Based on the resolution feature map information, the multi-view cost volume information, the cost volume residual, and the matching prior information, the predefined target decoder model is input to predict the corresponding Gaussian attributes, thereby obtaining Gaussian attribute information.

8. A feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification, characterized in that, The device includes: The acquisition module is used to acquire multi-view image set information and camera parameter information; The processing module is used to determine the corresponding resolution feature map information based on the image set information and the camera parameter information; The processing module is also used to perform global and local matching of image points at different resolutions based on the resolution feature map information, and to determine the matching prior information and the matching confidence map information; The execution module is used to input a predefined Gaussian splash model and Gaussian densification model based on the resolution feature map information, the matching prior information and the matching confidence map information, predict the corresponding Gaussian attributes and Gaussian densification attributes, and determine the Gaussian attribute information and Gaussian densification attribute information. The execution module is also used to reconstruct the anchor points to be densified in the scene space corresponding to the Gaussian attribute information based on the Gaussian densification attribute information, and determine the scene space reconstruction result.

9. A feedforward Gaussian splashing device for multi-view scene reconstruction based on feature matching and adaptive densification, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the feedforward Gaussian splashing method for multi-view scene reconstruction based on feature matching and adaptive densification as described in any one of claims 1 to 7.

Images