Three-dimensional scene novel view synthesis method based on matched light

Abstract

The invention is suitable for the technical field of view synthesis, and provides a novel three-dimensional scene view synthesis method based on matched light rays, and the method comprises the steps: generating a plurality of 3D Gaussian ellipsoids according to a plurality of view angle images of a target scene; randomly screening out a plurality of target view angle images from the plurality of view angle images; screening out a plurality of target 3D Gaussian ellipsoids from the plurality of 3D Gaussian ellipsoids according to the plurality of target view angle images; based on the matching light, screening out a plurality of Gaussian matching pairs from the plurality of target 3D Gaussian ellipsoids; optimizing the target 3D Gaussian ellipsoids in all Gaussian matching pairs; and fusing the plurality of optimized target 3D Gaussian ellipsoids to obtain a novel view of the target scene. According to the invention, the rendering effect of the 3DGS under the sparse view angle can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of view synthesis, and in particular, relates to a novel view synthesis method for a three-dimensional scene based on matching light. Background Art

[0002] In the field of 3D computer vision, generating novel views based on a few-shot novel view synthesis has always been a challenging research topic. With the increasing demand for applications such as virtual reality (VR), augmented reality (AR), and autonomous driving, how to reconstruct 3D scenes from sparse 2D images and generate unseen perspectives has become a key technical problem that needs to be solved urgently. In recent years, Neural Radiance Field (NeRF), as an emerging 3D representation method, has achieved remarkable achievements in the field of novel view image generation (NVS) with its excellent rendering effect. However, NeRF usually relies on a large number of training images, and its optimization and rendering speed are slow, which brings a large computational burden in practical applications, especially in scenarios that require fast response such as autonomous driving and robots.

[0003] To solve this problem, 3D Gaussian Splatting (3DGS) is a new 3D representation method that replaces volume rendering in NeRF by using efficient, high-resolution, and fast Gaussian primitives to achieve better performance. 3DGS uses Gaussian points initialized from sparse structured light (SfM) point clouds to explicitly represent 3D scenes, and optimizes its Gaussian parameters through photometric loss, ultimately achieving high-quality and real-time rendering. However, in the case of a small number of view inputs, especially when the scene geometry information is insufficient, the performance of 3DGS will still drop significantly, and the rendering results are prone to problems such as overfitting and geometric artifacts. That is, the current rendering effect of 3DGS under sparse view is poor. Summary of the invention

[0004] The embodiment of the present application provides a novel view synthesis method for a three-dimensional scene based on matching light, which can solve the problem of poor rendering effect of 3DGS under sparse viewing angles.

[0005] The embodiment of the present application provides a novel view synthesis method of a three-dimensional scene based on matching light, including:

[0006] Generate multiple 3D Gaussian ellipsoids according to multiple perspective images of the target scene;

[0007] Randomly select multiple target perspective images from multiple perspective images;

[0008] According to the multiple target view images, multiple target 3D Gaussian ellipsoids are screened out from the multiple 3D Gaussian ellipsoids;

[0009] Based on the matching light, a plurality of Gaussian matching pairs are screened out from the plurality of target 3D Gaussian ellipsoids; the Gaussian matching pairs include two target 3D Gaussian ellipsoids from the plurality of target 3D Gaussian ellipsoids;

[0010] Optimize the target 3D Gaussian ellipsoid in all Gaussian matching pairs;

[0011] The optimized multiple target 3D Gaussian ellipsoids are fused to obtain a novel view of the target scene.

[0012] Optionally, according to the multiple target view images, multiple target 3D Gaussian ellipsoids are screened out from the multiple 3D Gaussian ellipsoids, including:

[0013] For each target view image, the L1 loss and SSIM loss are used to calculate the photometric loss L between the target view image and the rendered image. p ; The rendered image is the rendering result of the process of generating a 3D Gaussian ellipsoid using the target view image;

[0014] The light loss L p The 3D Gaussian ellipsoid corresponding to the target perspective image that is less than the light loss threshold is used as the target 3D Gaussian ellipsoid.

[0015] Optionally, the photometric loss between the target view image and the rendered image is calculated using L1 loss and SSIM loss, including:

[0016] By formula Calculate the photometric loss L between the target view image and the rendered image p ;

[0017] Among them, λ is the weight factor, The target perspective image I and the rendered image The L1 loss between The target perspective image I and the rendered image The SSIM loss between .

[0018] Optionally, based on the matching rays, multiple Gaussian matching pairs are screened out from multiple target 3D Gaussian ellipsoids, including:

[0019] For the i-th target 3D Gaussian ellipsoid G among multiple target 3D Gaussian ellipsoids i and the jth target 3D Gaussian ellipsoid G j , calculate the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G jGaussian position loss; i = 1, ..., Q, j = 1, ..., Q, i ≠ j, Q is the number of target 3D Gaussian ellipsoids;

[0020] If the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j If the Gaussian position loss is less than the Gaussian position loss threshold, the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j as a Gaussian matched pair.

[0021] Optionally, calculate the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian position loss, including:

[0022] The i-th target 3D Gaussian ellipsoid G is calculated by the projection error calculation formula i To the jth target 3D Gaussian ellipsoid G j The projection error and the jth target 3D Gaussian ellipsoid G j To the i-th target 3D Gaussian ellipsoid G i The projection error

[0023] The projection error and projection error The average value of the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian location loss L g_primitives ;

[0024] The projection error calculation formula is:

[0025]

[0026] Among them, p i and p j They are the target perspective images I i and the target perspective image I j The pixel coordinates of the 2D pixels corresponding to the same 3D point, {p i , p j} is the target perspective image I i and the target perspective image I j A pair of matching rays {r i , r j} corresponding pixel coordinates, ray r i Collect target view image I for the camera i The ray from the three-dimensional point to the camera focus, ray rj Collect target view image I for the camera j The ray from the three-dimensional point to the camera focus, p i→j (μ′ i ) is p i Projected to the target perspective image I j The corresponding 2D projection coordinates of the 2D image plane, p j→i (μ′ j ) is p j Projected to the target perspective image I i The corresponding 2D projection coordinates of the 2D image plane, μ′ i =o i +z i d i , μ′ j =o j +z j d j , o i Represents the i-th target 3D Gaussian ellipsoid G i The origin of the coordinate system on the corresponding two-dimensional image plane, z i Represents the i-th target 3D Gaussian ellipsoid G i The corresponding depth of the observation point in three-dimensional space, d i Represents the i-th target 3D Gaussian ellipsoid G i The corresponding unit direction vector on the two-dimensional image plane, o j Represents the j-th target 3D Gaussian ellipsoid G j The origin of the coordinate system on the corresponding two-dimensional image plane, z j Represents the j-th target 3D Gaussian ellipsoid G j The corresponding depth of the observation point in three-dimensional space, d j Represents the j-th target 3D Gaussian ellipsoid G j The corresponding unit direction vector on the two-dimensional image plane, the target perspective image I i is the i-th target 3D Gaussian ellipsoid G i The corresponding target perspective image, target perspective image I j is the jth target 3D Gaussian ellipsoid G j The corresponding target perspective image.

[0027] Optionally, the target 3D Gaussian ellipsoids in all Gaussian matching pairs are optimized, including:

[0028] Calculate the geometric loss L for each Gaussian matching pair separately render_g ;

[0029] Based on the calculated geometric losses L render_g The average value of all Gaussian position losses L g_primitivesThe average value and all luminosity losses L p Calculate the final error;

[0030] Iteratively update the parameters of the target 3D Gaussian ellipsoid based on the final error, and return to calculate the geometric loss L for each Gaussian matching pair separately. render_g The target 3D Gaussian ellipsoid in each Gaussian matching pair when the iteration termination condition is met is used as the optimized target 3D Gaussian ellipsoid.

[0031] Optionally, calculate the geometric loss L for each Gaussian matching pair render_g ,include:

[0032] The target 3D Gaussian ellipsoid G′ of the Gaussian matching pair i and target 3D Gaussian ellipsoid G′ j Projection to a 2D image plane;

[0033] Get the depth value D in the 2D image plane depth_i (p i ) and depth value D depth_j (p j );

[0034] The depth value D depth_i (p i ) is transformed into 3D space and the position v in 3D space is obtained i , and the depth value D depth_j (p j ) is transformed into 3D space and the position v in 3D space is obtained j ;

[0035] The target 3D Gaussian ellipsoid G′ is calculated by the geometric error calculation formula i To the target 3D Gaussian ellipsoid G′ j The geometric error and target 3D Gaussian ellipsoid G′ j To the target 3D Gaussian ellipsoid G′ i The geometric error

[0036] The geometric error and geometric error The average value of the geometric loss L of the Gaussian matching pair render_g .

[0037] Optionally, the geometric error calculation formula is:

[0038]

[0039] Among them, p i→j (v′ i ) is v iProjected to the target perspective image I j The corresponding two-dimensional projection coordinates of the 2D image plane, R i is the rotation matrix of the observation camera under the target perspective i, D i (p i ) is D depth_i (p i ), is the inverse matrix of the intrinsic parameter matrix of the observation camera under the target perspective i, is the homogeneous coordinate of the pixel coordinate on the two-dimensional image plane corresponding to the target view angle i, t i is the translation vector of the observation camera under the target perspective i, p j→i (v′ j ) is ν j Projected to the target perspective image I i The corresponding two-dimensional projection coordinates of the 2D image plane, R j is the rotation matrix of the observation camera under the target perspective j, D j (p j ) is D depth_j (p j ), is the inverse matrix of the intrinsic parameter matrix of the observation camera under the target perspective j, is the homogeneous coordinate of the pixel coordinate on the two-dimensional image plane corresponding to the target view angle j, t j is the translation vector of the observation camera under the target perspective j.

[0040] Optionally, based on the calculated total geometric loss L render_g The average value of all Gaussian position losses L g_primitives The average value and all luminosity losses L p Calculate the final error, including:

[0041] Through the formula Loss = L' p +βL' g_primitives +δL' render_g Calculate the final error;

[0042] Among them, Loss is the final error, L′ p is the calculated total luminosity loss L p The sum of β and δ are weight factors, L' g_primitives is the calculated Gaussian position loss L g_primitives The average value, L′ render_g is the calculated geometric loss L render_g The average value of .

[0043] The above solution of the present application has the following beneficial effects:

[0044] In an embodiment of the present application, a matching light strategy is adopted to capture the correspondence between light and pixels between different perspectives, thereby helping to optimize the model to maintain a consistent 3D structure under multiple perspectives. In this way, these a priori matching lights are introduced as constraints into the optimization process, thereby guiding the model to avoid geometric conflicts caused by multi-perspective inconsistencies or scale problems when rendering depth. Through the guidance of matching priors, the rendered geometric structure can more accurately reflect the real structure of the scene, thereby avoiding the common geometric distortion and detail loss problems in a small number of perspectives, and improving the rendering effect of 3DGS under sparse perspectives.

[0045] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A flowchart of a novel view synthesis method for a three-dimensional scene provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0049] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0050] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0051] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0052] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0054] In response to the problem that the current 3DGS has poor rendering effect under sparse perspectives, an embodiment of the present application provides a novel view synthesis method for a three-dimensional scene based on matching light. By adopting a matching light strategy, the correspondence between light and pixels between different perspectives is captured, helping to optimize the model to maintain a consistent 3D structure under multiple perspectives. In this way, these prior matching lights are introduced into the optimization process as constraints, thereby guiding the model to avoid geometric conflicts caused by multi-perspective inconsistencies or scale problems when rendering depth. Through the guidance of matching priors, the rendered geometric structure can more accurately reflect the real structure of the scene, thereby avoiding the common geometric distortion and detail loss problems in a small number of perspectives, and improving the rendering effect of 3DGS under sparse perspectives.

[0055] The novel view synthesis method of a three-dimensional scene based on matching light provided by the present application is exemplarily described below in conjunction with specific embodiments.

[0056] like Figure 1 As shown, the novel view synthesis method of a three-dimensional scene based on matching light provided in an embodiment of the present application includes the following steps:

[0057] Step 11, generating multiple 3D Gaussian ellipsoids according to multiple viewing angle images of the target scene.

[0058] In some embodiments of the present application, a camera may be used to capture multiple perspective images of a target scene (ie, image information from multiple perspectives), and then these perspective images may be used in combination with internal and external parameters of the camera to generate multiple 3D Gaussian ellipsoids.

[0059] Specifically, a semi-positive covariance matrix ∑ can be defined based on the input camera view information. The covariance matrix is ​​decomposed into two parts: a scaling matrix S and a rotation matrix R. This allows the shape and direction of the Gaussian primitive (i.e., 3D Gaussian ellipsoid) to be adjusted and optimized more efficiently during the optimization process, so that the geometric structure of each Gaussian primitive maintains reasonable physical constraints in three-dimensional space, thereby ensuring the accuracy and stability of the scene representation: ∑ = RSS T R T Then, according to the defined covariance matrix ∑, combined with the Gaussian center vector information and its position x, explicitly parameterizing the 3D Gaussian ellipsoid:

[0060] For the 3D Gaussian ellipsoid, it is necessary to project the 3D Gaussian to the 2D image plane via the Jacobian matrix of the affine approximation of the view transformation matrix W and the projection transformation J: ∑′=JW∑W T J T , for point-based rendering, the color C of a pixel can be calculated by mixing N ordered Gaussians that overlap the pixel:

[0061]

[0062] Similarly, the corresponding depth image D is rendered depth :

[0063]

[0064] In the above formula, N is the number of generated 3D Gaussian ellipsoids, c i is the color value of the i-th Gaussian distribution (i.e., 3D Gaussian ellipsoid), α′ i is the opacity of the ith Gaussian distribution, ranging from [0, 1], which is used to measure the contribution of the distribution to the final pixel color. is the cumulative transmittance, which means the remaining part that is not completely covered when the i-th Gaussian is covered and blocked by the previously generated Gaussian j, d i is the depth value of the i-th observed object, indicating the distance from the observation point to the observed object.

[0065] Step 12: randomly select multiple target perspective images from the multiple perspective images.

[0066] In some embodiments of the present application, a limited number of M training perspectives can be selected by setting a random seed. (i.e., M target view images are selected) to simulate the imaging process under sparse view.

[0067]

[0068] Among them, RandomSubset refers to the process of randomly selecting a part of data from a larger data set as a subset, and V is the Vth perspective image of the target scene.

[0069] Step 13: Screening out multiple target 3D Gaussian ellipsoids from the multiple 3D Gaussian ellipsoids according to the multiple target view images.

[0070] In some embodiments of the present application, the photometric loss L between the target view image and the rendered image (the rendered image is the rendering result in the process of generating a 3D Gaussian ellipsoid using the target view image) can be calculated using the L1 loss and the SSIM loss for each target view image. p ; Then the luminosity loss L p The 3D Gaussian ellipsoid corresponding to the target perspective image that is less than the light loss threshold is used as the target 3D Gaussian ellipsoid.

[0071] In some embodiments of the present application, the formula Calculate the photometric loss L between the target view image and the rendered image p .

[0072] Among them, λ is the weight factor, which can be set to 0.15. The target perspective image I and the rendered image The L1 loss between The target perspective image I and the rendered image The SSIM loss between .

[0073] In order to allow 3DGS to learn a consistent 3D scene structure under sparse view input without SFM (SFM is a method for recovering structure from motion, mainly used for 3D reconstruction) point initialization, after selecting the better Gaussian primitives under sparse view, it is necessary to complete the matching of prior information. The specific steps are as follows:

[0074] (1) Matching is done through prior information. Matching ray pairs represent the 2D pixel positions corresponding to the same 3D point (referring to a point in the target scene) in different views (i.e., perspective images). These matching rays can be used as key multi-view constraints. Given an image I i and I j A pair of matching rays {r i , r j}, and its corresponding pixel coordinates are {p i , p j}, assuming that the surface points intersecting each ray have been calculated to be P i and P j , then we can get: P i =P j ;

[0075] (2) Using the camera's intrinsic parameter matrix {K i , K j} and the external parameter matrix {[R i , t i ],[R j , t j ]}, project the three-dimensional surface point onto another two-dimensional image plane to obtain the corresponding projected pixel coordinates p i→j (P i ):

[0076]

[0077] Get pixel coordinates p j =p i→j , similarly, we have p i =p j→i .

[0078] Among them, r i Refers to a given image I i The corresponding light, r j Refers to a given image I j The corresponding light, K i is the internal parameter matrix of the observation camera under the target perspective i, K j is the internal parameter matrix of the observation camera under the target perspective j, R i is the rotation matrix of the observation camera under the target perspective i, t i is the translation vector of the observation camera under the target perspective i, R j is the rotation matrix of the observation camera under the target perspective j, t j is the translation vector of the observation camera under the target perspective j, p i→j is the viewing angle i (which can be understood as the image I captured by the camera at viewing angle i i ) to the viewing angle j (which can be understood as the image I taken by the camera at the viewing angle j j )’s two-dimensional pixel plane projection pixel coordinates, p j→i is the viewing angle j (which can be understood as the image I taken by the camera at the viewing angle j j ) to the viewing angle i (which can be understood as the image I taken by the camera at the viewing angle i i )'s two-dimensional pixel plane projection pixel coordinates.

[0079] In the matching prior, the positions of matching rays can accurately indicate the regions that are visible in at least two views. These multi-view visible regions are crucial for the reconstruction of 3D scenes because they provide important information about the scene geometry. When there are overlapping regions between different views, the model can use these regions for consistency constraints, thereby effectively optimizing and aligning the 3D structure.

[0080] Step 14: Based on the matching rays, multiple Gaussian matching pairs are selected from multiple target 3D Gaussian ellipsoids.

[0081] The Gaussian matching pair includes two target 3D Gaussian ellipsoids from a plurality of target 3D Gaussian ellipsoids.

[0082] In some embodiments of the present application, the i-th target 3D Gaussian ellipsoid G among the multiple target 3D Gaussian ellipsoids may be i and the jth target 3D Gaussian ellipsoid G j , calculate the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian position loss; if the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j If the Gaussian position loss is less than the Gaussian position loss threshold, the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j As Gaussian matching pairs, where i=1, ..., Q, j=1, ..., Q, i≠j, and Q is the number of target 3D Gaussian ellipsoids.

[0083] Specifically, calculate the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j The Gaussian position loss can be calculated as follows: The i-th target 3D Gaussian ellipsoid G is calculated by the projection error calculation formula i To the jth target 3D Gaussian ellipsoid G j The projection error and the jth target 3D Gaussian ellipsoid G j To the i-th target 3D Gaussian ellipsoid G i The projection error Then the projection error and projection error The average value of is taken as the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian location loss L g_primitives .

[0084] The above projection error calculation formula is:

[0085]

[0086] Among them, p i and p j They are the target perspective images I i and the target perspective image I j The pixel coordinates of the 2D pixels corresponding to the same 3D point, {p i , p j} is the target perspective image I i and the target perspective image I j A pair of matching rays {r i , r j} corresponding pixel coordinates, ray r i Collect target view image I for the camera i The ray from the three-dimensional point to the camera focus, ray r j Collect target view image I for the camera j The ray from the three-dimensional point to the camera focus, p i→j (μ′ i ) is p i Projected to the target perspective image I j The corresponding 2D projection coordinates of the 2D image plane, p j→i (μ′ j ) is p j Projected to the target perspective image I i The corresponding 2D projection coordinates of the 2D image plane, μ′ i =o i +z i d i , μ′ j =o j +z j d j , o i Represents the i-th target 3D Gaussian ellipsoid G i The origin of the coordinate system on the corresponding two-dimensional image plane, z i Represents the i-th target 3D Gaussian ellipsoid G i The depth of the observation point in the corresponding three-dimensional space (the value of the z axis), that is, the distance from the observation point to the projection plane, d i Represents the i-th target 3D Gaussian ellipsoid G i The corresponding unit direction vector on the two-dimensional image plane defines the direction of the projection transformation, o j Represents the j-th target 3D Gaussian ellipsoid G j The origin of the coordinate system on the corresponding two-dimensional image plane, z j Represents the j-th target 3D Gaussian ellipsoid G jThe depth of the observation point in the corresponding three-dimensional space (the value of the z axis), that is, the distance from the observation point to the projection plane, d j Represents the j-th target 3D Gaussian ellipsoid G j The corresponding unit direction vector on the two-dimensional image plane defines the direction of the projection transformation, the target perspective image I i is the i-th target 3D Gaussian ellipsoid G i The corresponding target perspective image, target perspective image I j is the jth target 3D Gaussian ellipsoid G j The corresponding target perspective image.

[0087] The following is the projection error and The derivation process is explained.

[0088] In addition to the common unstructured Gaussian primitives used to restore the visible background area in a single view, the novel view synthesis method for a three-dimensional scene in the embodiment of the present application also introduces ray-based Gaussian primitives (i.e., Gaussian matching pairs). These primitives are bound to matching rays and optimized using the geometric relationship between the rays. The specific implementation scheme is as follows:

[0089] (1) Use ray-based Gaussian primitives for initialization and bind them to matching rays to complete Gaussian initialization and densification. Assume there are N pairs of matching rays For the kth pair of matching rays, initialize N pairs of ray-based Gaussian primitives For the kth Gaussian matching pair, the position of the ray-based Gaussian basis element μ′ is defined as: μ′=0+zd, and the average amplitude of the view space position gradient is used to identify the underreconstructed area, where o represents the origin of the coordinate system on the corresponding two-dimensional image plane, z represents the depth of the observation point in the three-dimensional space (the value of the z-axis), that is, the distance from the observation point to the projection plane, and d represents the unit direction vector on the corresponding two-dimensional image plane, which defines the direction of the projection transformation.

[0090] (2) Optimize the position of the original Gaussian: For image I i and I j A pair of matching rays {r i , r j}, we get a pair of bound Gaussian basis elements {G i , G j}, their positions in 3D space are μ′ i =o i +z i d i and μ′ j =o j +z j d j. Thus, we get the two-dimensional projection coordinates from image i to image j: p i→j (μ′ i ), the two-dimensional projection coordinates from j to i: p j→i (μ′ j ). Therefore, the projection error of this pair of Gaussian basis elements can be obtained as:

[0091]

[0092] Step 15, optimizing the target 3D Gaussian ellipsoids in all Gaussian matching pairs.

[0093] In some embodiments of the present application, the specific implementation method of optimizing the target 3D Gaussian ellipsoid in all Gaussian matching pairs is:

[0094] Step 15.1, calculate the geometric loss L for each Gaussian matching pair separately render_g .

[0095] It should be noted that the geometric loss L of each Gaussian matching pair is render_g The calculation method is the same as that of the geometric loss L for any Gaussian matching pair. render_g Specifically, the geometric loss L of the Gaussian matching pair is render_g The calculation process is:

[0096] The target 3D Gaussian ellipsoid G′ of the Gaussian matching pair i and target 3D Gaussian ellipsoid G′ j Projection to a 2D image plane;

[0097] Get the depth value D in the 2D image plane depth_i (p i ) and depth value D depth_j (p j );D depth_i (p i ) is the target 3D Gaussian ellipsoid G′ i The corresponding depth value in the 2D image plane, D depth_j (p j ) is the target 3D Gaussian ellipsoid G′ j The depth value corresponding to the 2D image plane can be calculated according to the depth image D depth (i.e. the depth value of the 3D Gaussian ellipsoid);

[0098] The depth value D depth_i (p i ) is transformed into 3D space and the position v in 3D space is obtained i , and the depth value D depth_j (p j) is transformed into 3D space and the position v in 3D space is obtained j ;

[0099] The target 3D Gaussian ellipsoid G′ is calculated by the geometric error calculation formula i To the target 3D Gaussian ellipsoid G′ j The geometric error and target 3D Gaussian ellipsoid G′ j To the target 3D Gaussian ellipsoid G′ i The geometric error

[0100] The geometric error and geometric error The average value of the geometric loss L of the Gaussian matching pair render_g .

[0101] The geometric error calculation formula is:

[0102]

[0103] In the above formula, p i→j (v′ i ) is v i Projected to the target perspective image I j The corresponding two-dimensional projection coordinates of the 2D image plane, R i is the target view angle i (i.e., the camera captures the target view angle image I i The rotation matrix of the observation camera under the viewing angle (D i (p i ) is D depth_i (p i ), is the inverse matrix of the intrinsic parameter matrix of the observation camera under the target perspective i, is the homogeneous coordinate of the pixel coordinate on the two-dimensional image plane corresponding to the target view angle i, t i is the translation vector of the observation camera under the target perspective i, p j→i (v′ j ) is ν j Projected to the target perspective image I i The corresponding two-dimensional projection coordinates of the 2D image plane, R j is the target perspective j (i.e., the camera captures the target perspective image I j The rotation matrix of the observation camera under the viewing angle (D j (p j ) is D depth_j (p j ), is the inverse matrix of the intrinsic parameter matrix of the observation camera under the target perspective j, is the homogeneous coordinate of the pixel coordinate on the two-dimensional image plane corresponding to the target view angle j, t j is the target viewing angle, the translation vector of the observing camera.

[0104] Step 15.2, based on the calculated geometric loss L render_g The average value of all Gaussian position losses L g_primitives The average value and all luminosity losses L p Calculate the final error. Specifically, the formula Loss = L' p +βL' g_primitives +δL′ render_g Calculate the final error.

[0105] Among them, Loss is the final error, L′ p is the calculated total luminosity loss L p The sum of β and δ are weight factors, L' g_primitives is the calculated Gaussian position loss L g_primitives The average value, L' render_g is the calculated geometric loss L render_g The average value of .

[0106] Step 15.3, based on the final error, adjust the parameters of the target 3D Gaussian ellipsoid (such as learning rate Ir, spatial position o of the 3D Gaussian ellipsoid, scale d i (indicates the size or diffusion range of the Gaussian), color C and opacity α (controls the degree of light occlusion), etc.) are iteratively updated and returned to calculate the geometric loss L for each Gaussian matching pair respectively. render_g The step (i.e., returning to step 15.1) is performed, and the target 3D Gaussian ellipsoid in each Gaussian matching pair when the iteration termination condition is met is used as the optimized target 3D Gaussian ellipsoid.

[0107] In some embodiments of the present application, the above-mentioned iteration termination condition may be that the final error is less than an error threshold, or the number of iterations reaches a preset number.

[0108] During the training process, β was set to 1.0 to ensure that the various losses were balanced during the optimization process. In order to prevent the model from falling into a suboptimal solution at the beginning of training, δ was first set to 0 and gradually increased to 0.4 after 1000 iterations to gradually introduce more constraints. In the first 1000 iterations, to ensure that the Gaussian primitives could converge to the optimal position, a caching strategy was adopted, that is, the Gaussian primitive position loss (L g_primitives ) is the smallest position. In addition, since there may be mismatched light pairs in the matching prior, we further filter out those positions where the loss L g_primitivesGaussian primitives greater than a certain threshold η. During the optimization process, ray-based Gaussian primitives will not be pruned to ensure the consistency of the model structure.

[0109] For example, the number of iterations can be set to 3000. In the early stage of training, the learning rate of the learnable distance factor z is set to 0.1, and gradually reduced to 1.5×10 -6 , in order to fine-tune the position of the Gaussian primitives.

[0110] It is worth mentioning that by introducing these error terms, various aspects of the scene can be optimized more comprehensively. Photometric loss is mainly used to ensure the consistency of color and brightness between the generated image and the real image, ensuring the high quality of the rendering effect. Gaussian position loss helps optimize the spatial position of Gaussian primitives and ensure their reasonable distribution in three-dimensional space, thereby improving geometric consistency. Rendering geometry loss further constrains the geometric structure of the rendering result to keep it consistent under changes in perspective, especially under sparse perspective conditions, to prevent rendering problems caused by overfitting or geometric distortion. Combining these three losses can better balance the geometric structure and texture details, and improve the stability and accuracy of few-view novel view synthesis.

[0111] Step 16, fusing the optimized multiple target 3D Gaussian ellipsoids to obtain a novel view of the target scene.

[0112] In summary, this application proposes an effective solution to the key challenges in generating novel views from a small number of perspectives (Few-shot Novel View Synthesis), which can maintain the consistency of scene structure and improve rendering quality under sparse perspective conditions. It is mainly reflected in the following aspects:

[0113] 1. Introduction and optimization of matching priors:

[0114] By introducing matching prior information and using the correspondence between light and pixels between perspectives, the geometric structure of the 3D scene is optimized. Matching prior helps ensure that the model can maintain a consistent 3D structure under sparse perspectives, avoiding geometric artifacts or missing details caused by sparse perspectives and inconsistent multi-perspectives. The core value of this strategy is that it can effectively avoid geometric distortion problems under a single perspective or a small number of perspectives, especially the scale inconsistency and geometric inconsistency problems in depth estimation and multi-perspective fusion.

[0115] 2. Mixed representation of Gaussian primitives and light constraints:

[0116] Aiming at the high interdependence between the shape and position of Gaussian primitives, a hybrid Gaussian representation is designed. In addition to the traditional unstructured Gaussian primitives, ray-based Gaussian primitives are introduced. These primitives can be bound to matching rays and optimized along the direction of the rays. This method accurately constrains the position of Gaussian primitives to the real 3D geometric surface by matching and optimizing the rays, thereby improving the geometric accuracy and consistency of the rendering results.

[0117] 3. Double optimization strategy:

[0118] A dual optimization strategy is adopted, that is, optimizing the position and shape of Gaussian primitives at the same time. In this way, the optimization can not only improve the details and quality of the rendered image, but also ensure that the spatial distribution of Gaussian primitives is consistent with the scene structure, especially under a small number of viewing angles, it can avoid overfitting and texture blurring. This dual optimization strategy ensures the stability of the geometric structure under sparse viewing angles and greatly improves the rendering effect.

[0119] 4. Seamless connection from matching priors to deep optimization:

[0120] By leveraging pre-trained matching models (such as geometric constraints for matching light pairs), the limitations of traditional methods that rely on SFM points or random initialization are avoided, allowing effective scene reconstruction without high-quality initial point clouds. This allows the framework to operate stably in data-scarce or complex scenes and has better adaptability.

[0121] The core of the invention is to introduce an innovative strategy of matching prior information and optimizing Gaussian primitives with light constraints, which solves the common problems of geometric inconsistency, detail loss and texture blur under a small number of viewing angles. At the same time, through a dual optimization strategy, it ensures a significant improvement in the accuracy and quality of the rendering results.

[0122] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A novel view synthesis method for a three-dimensional scene based on matching light, characterized in that: include: Generate multiple 3D Gaussian ellipsoids according to multiple perspective images of the target scene; Randomly selecting a plurality of target perspective images from the plurality of perspective images; Filtering a plurality of target 3D Gaussian ellipsoids from the plurality of 3D Gaussian ellipsoids according to the plurality of target perspective images; Based on the matching light, a plurality of Gaussian matching pairs are screened out from the plurality of target 3D Gaussian ellipsoids; the Gaussian matching pairs include two target 3D Gaussian ellipsoids from the plurality of target 3D Gaussian ellipsoids; Optimize the target 3D Gaussian ellipsoid in all Gaussian matching pairs; The optimized multiple target 3D Gaussian ellipsoids are fused to obtain a novel view of the target scene.

2. The novel view synthesis method for a three-dimensional scene according to claim 1, characterized in that: The step of selecting a plurality of target 3D Gaussian ellipsoids from the plurality of 3D Gaussian ellipsoids according to the plurality of target perspective images comprises: For each target perspective image, the photometric loss L between the target perspective image and the rendered image is calculated using the L1 loss and the SSIM loss. p ; The rendered image is a rendering result in the process of generating a 3D Gaussian ellipsoid using the target perspective image; The light loss L p The 3D Gaussian ellipsoid corresponding to the target perspective image that is less than the light loss threshold is used as the target 3D Gaussian ellipsoid.

3. The novel view synthesis method for a three-dimensional scene according to claim 2, characterized in that: The method of calculating the photometric loss between the target perspective image and the rendered image by using the L1 loss and the SSIM loss includes: By formula Calculate the photometric loss L between the target perspective image and the rendered image p ; Among them, λ is the weight factor, The target perspective image I and the rendered image The L1 loss between The target perspective image I and the rendered image The SSIM loss between .

4. The novel view synthesis method for a three-dimensional scene according to claim 2, characterized in that: The step of selecting a plurality of Gaussian matching pairs from the plurality of target 3D Gaussian ellipsoids based on the matching light rays comprises: For the i-th target 3D Gaussian ellipsoid G among the multiple target 3D Gaussian ellipsoids i and the jth target 3D Gaussian ellipsoid G j , calculate the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian position loss; i = 1, ..., Q, j = 1, ..., Q, i ≠ j, Q is the number of target 3D Gaussian ellipsoids; If the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j If the Gaussian position loss is less than the Gaussian position loss threshold, the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j as a Gaussian matched pair.

5. The novel view synthesis method for a three-dimensional scene according to claim 4, characterized in that: The calculation of the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian position loss, including: The i-th target 3D Gaussian ellipsoid G is calculated by the projection error calculation formula i To the Jth target 3D Gaussian ellipsoid G j The projection error and the jth target 3D Gaussian ellipsoid G j To the i-th target 3D Gaussian ellipsoid G i The projection error The projection error and projection error The average value of is taken as the i-th target 3D Gaussian ellipsoid G i and the jth target 3D Gaussian ellipsoid G j Gaussian location loss L g_primitives ; The projection error calculation formula is: Among them, p i and p j They are the target perspective images I i and the target perspective image I j The pixel coordinates of the 2D pixels corresponding to the same 3D point, {p i ,p j } is the target perspective image I i and the target perspective image I j A pair of matching rays {r i ,r j }The corresponding pixel coordinates, ray r i Collect target view image I for the camera i When the three-dimensional point points to the ray of the camera focus, the ray r j Collect target view image I for the camera j When the three-dimensional point points to the ray of the camera focus, p i→j (μ′ i ) is p i Projected to the target perspective image I j The corresponding 2D projection coordinates of the 2D image plane, p j→i (μ′ j ) is p j Projected to the target perspective image I i The corresponding 2D projection coordinates of the 2D image plane, μ′ i =o i +z i d i , μ′ j =o j +z j d j , o i Represents the i-th target 3D Gaussian ellipsoid G i The origin of the coordinate system on the corresponding two-dimensional image plane, z i Represents the i-th target 3D Gaussian ellipsoid G i The corresponding depth of the observation point in three-dimensional space, d i Represents the i-th target 3D Gaussian ellipsoid G i The corresponding unit direction vector on the two-dimensional image plane, o j Represents the j-th target 3D Gaussian ellipsoid G j The origin of the coordinate system on the corresponding two-dimensional image plane, z j Represents the j-th target 3D Gaussian ellipsoid G j The depth of the corresponding observation point in three-dimensional space, d j Represents the j-th target 3D Gaussian ellipsoid G j The corresponding unit direction vector on the two-dimensional image plane, the target perspective image I i is the i-th target 3D Gaussian ellipsoid G i The corresponding target perspective image, target perspective image I j is the jth target 3D Gaussian ellipsoid Gj 对 The corresponding target perspective image.

6. The novel view synthesis method for a three-dimensional scene according to claim 5, characterized in that: The optimization of the target 3D Gaussian ellipsoid in all Gaussian matching pairs includes: Calculate the geometric loss L for each Gaussian matching pair separately render_g ; Based on the calculated geometric losses L render_g The average value of all Gaussian position losses L g_primitives The average value and all luminosity losses L p Calculate the final error; Iteratively update the parameters of the target 3D Gaussian ellipsoid based on the final error, and return to execute the geometric loss L calculated for each Gaussian matching pair respectively. render_g The target 3D Gaussian ellipsoid in each Gaussian matching pair when the iteration termination condition is met is used as the optimized target 3D Gaussian ellipsoid.

7. The novel view synthesis method for a three-dimensional scene according to claim 6, characterized in that: The geometric loss L is calculated for each Gaussian matching pair render_g ,include: The target 3D Gaussian ellipsoid G′ of the Gaussian matching pair i and target 3D Gaussian ellipsoid G′ j Projection to a 2D image plane; Get the depth value D in the 2D image plane depth_i (p i ) and depth value D depth_j (p j ); The depth value D depth_i (p i ) is transformed into 3D space and the position v in 3D space is obtained i , and the depth value D depth_j (p j ) is transformed into 3D space and the position v in 3D space is obtained j ; The target 3D Gaussian ellipsoid G′ is calculated by the geometric error calculation formula i To the target 3D Gaussian ellipsoid G′ j The geometric error and target 3D Gaussian ellipsoid G′ j To the target 3D Gaussian ellipsoid G′ i The geometric error The geometric error and geometric error The average value of the geometric loss L of the Gaussian matching pair render_g .

8. The novel view synthesis method for a three-dimensional scene according to claim 7, characterized in that: The geometric error calculation formula is: Among them, p i→j (v′ i ) is v i Projected to the target perspective image I j The corresponding two-dimensional projection coordinates of the 2D image plane, R i is the rotation matrix of the observation camera under the target perspective i, D i (p i ) is D depth_i (p i ), is the inverse matrix of the intrinsic parameter matrix of the observation camera under the target perspective i, is the homogeneous coordinate of the pixel coordinate on the two-dimensional image plane corresponding to the target view angle i, t i is the translation vector of the observation camera under the target perspective i, p j→i (v′ j ) is v j Projected to the target perspective image I i The corresponding two-dimensional projection coordinates of the 2D image plane, R j is the rotation matrix of the observation camera under the target perspective j, D j (p j ) is D depth_j (p j ), is the inverse matrix of the intrinsic parameter matrix of the observation camera under the target perspective j, is the homogeneous coordinate of the pixel coordinate on the two-dimensional image plane corresponding to the target view angle j, t j is the translation vector of the observation camera under the target perspective j.

9. The novel view synthesis method of a three-dimensional scene according to claim 6, characterized in that: The calculated geometric losses L render_g The average value of all Gaussian position losses L g_primitives The average value and all luminosity losses L p Calculate the final error, including: Through the formula Loss = L' p +βL' g_primitives +δL' render_g Calculate the final error; Among them, Loss is the final error, L′ p is the calculated total luminosity loss L p The sum of β and δ are weight factors, L' g_primitives is the calculated Gaussian position loss L g_primitives The average value, L' render_g is the calculated geometric loss L render_g The average value of .

Images