Image processing method and device, equipment, storage medium and program product
By combining matching priors and 3D Gaussian splattering techniques, the performance decay caused by sparse images in new perspective synthesis in few-sample new perspectives is solved, achieving higher rendering accuracy and lower risk of overfitting.
Patent Information
- Application Number
- CN202411525837.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-30
- Publication Date
- 2025-02-25
AI Technical Summary
In the new perspective synthesis task with few samples, the input of sparse images leads to the performance of the new perspective synthesis model deterioration, the rendering effect is not ideal, and the multi-view consistency of the monocular depth model is insufficient, making it difficult to learn accurate scene structures.
Using a method combining matching priors and three-dimensional Gaussian splattering technology, the matching priors between different viewing angle images of the same scene are extracted through the pre-trained matching model, the mixed Gaussian representation is initialized, and the position and shape of the Gaussian primitives are optimized to improve the rendering accuracy of new viewing angle images.
This improves the rendering accuracy of new perspective images, reduces the risk of overfitting under sparse image input, speeds up the convergence speed of model, and makes the optimization of scene structure more consistent and accurate.
Smart Images
Figure CN119516075A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image processing method, device, equipment, storage medium and program product. Background Art
[0002] Few-shot New View Synthesis (FNVS) is a type of technology that reconstructs the radiation field of a target scene from a given set of sparse images. In the prior art, the three-dimensional Gaussian splattering (3DGS) technology can be used to implement FNVS. Three-dimensional Gaussian splattering (3DGS) is a method that uses a set of Gaussian primitives with shapes and positions to explicitly model three-dimensional scenes. It uses differentiable Gaussian rasterization to quickly render the entire image. However, the 3DGS technology relies on dense image input. In the Few-shot New View Synthesis (FNVS) task, as the image becomes sparse, the new view synthesis model (such as 3DGS, NeRF, etc.) will have serious performance degradation, which in turn affects the rendering effect.
[0003] In order to overcome the impact of sparse images input in the process of few-shot new view synthesis (FNVS) based on 3DGS, the existing technology mainly relies on adding monocular depth prior to guide the optimization of the new view synthesis model, and uses the relative depth relationship in the monocular depth as a supervisory signal to supervise the relative depth relationship of the depth rendered by the model, so as to achieve geometric regularization of the model. However, the monocular depth map obtained by the monocular depth model has the problems of unknown scale and inconsistency in multiple views, which makes it difficult to ensure the consistency of multiple views, and still cannot allow the new view synthesis model to learn the accurate scene structure, resulting in the rendering result is still not ideal. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide an image processing method, device, equipment, storage medium and program product, which use a combination of matching prior and three-dimensional Gaussian splashing technology to achieve a few-sample new perspective synthesis task and improve the accuracy of the rendered new perspective image.
[0005] To achieve the above object, an embodiment of the present invention provides an image processing method, comprising:
[0006] Input two images of the same scene at different viewing angles into a preset matching model to extract a matching prior between the two images; wherein the matching prior is a matching relationship of a matching ray pair, and the matching ray pair is composed of matching rays corresponding to the same pixel point in the two images;
[0007] Initializing a mixed Gaussian representation according to the matching prior; wherein the mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives; the ray-based Gaussian primitives are restricted to move in the direction of the matching ray;
[0008] The position and shape of the Gaussian primitives are optimized and trained according to the matching prior to obtain a trained new perspective synthesis model; the new perspective synthesis model is used to realize new perspective image rendering.
[0009] As an improvement of the above solution, the matching relationship includes the ray correspondence relationship of the matching ray pair and the ray position of the matching ray.
[0010] As an improvement of the above solution, the initializing the mixed Gaussian representation according to the matching prior includes:
[0011] Initializing a ray-based Gaussian primitive corresponding to each matching ray; wherein the position of the ray-based Gaussian primitive is determined according to the ray position of the matching ray;
[0012] Splitting or cloning at least one unstructured Gaussian primitive from the ray-based Gaussian primitive;
[0013] The mixed Gaussian representation is formed based on the ray-based Gaussian primitives and the unstructured Gaussian primitives.
[0014] As an improvement of the above solution, the position and shape of the Gaussian primitives are optimized and trained according to the matching prior to obtain a trained new perspective synthesis model, including:
[0015] Calculating position loss, rendering loss and color loss based on the matching prior;
[0016] The position of the Gaussian primitive is optimized and trained according to the position loss, and the shape of the Gaussian primitive is optimized and trained according to the rendering loss and the color loss, so as to obtain a trained new perspective synthesis model.
[0017] As an improvement of the above solution, the calculating the position loss according to the matching prior includes:
[0018] Binding the ray-based Gaussian primitives to matching rays in the matching prior;
[0019] According to the ray correspondence of the matching ray pair, a matching correspondence between two ray-based Gaussian primitives in a ray-based Gaussian primitive pair is obtained; wherein the ray-based Gaussian primitive pair is two ray-based Gaussian primitives bound to two matching rays in the matching ray pair;
[0020] Calculating a projection error between two ray-based Gaussian primitives in the ray-based Gaussian primitive pair according to the ray position of the matching ray and the matching correspondence;
[0021] The position loss is calculated based on the projection errors of all ray-based Gaussian basis pairs.
[0022] As an improvement of the above solution, the calculation of rendering loss and color loss according to the matching prior includes:
[0023] Inputting the mixed Gaussian representation and the camera poses of the two images into a Gaussian rasterizer to obtain a rendered depth map of the two images corresponding to the viewing angle, so as to obtain a depth value of the matching ray pair;
[0024] According to the depth value of the matching ray pair, two pixel points corresponding to the matching ray pair are mapped to a three-dimensional space, and the three-dimensional coordinates of the two pixel points are calculated;
[0025] Calculating a projection error based on the three-dimensional coordinates and the ray positions of the matching ray pairs;
[0026] Calculate the rendering loss according to the projection errors of all the matching ray pairs;
[0027] The real images of the two images are rendered by Gaussian rasterization to obtain a rendering color map;
[0028] The loss between the rendered color map and the real image is calculated to obtain a color loss.
[0029] The embodiment of the present invention further provides an image processing device, comprising:
[0030] A matching prior extraction module is used to input two images of the same scene at different viewing angles into a preset matching model to extract a matching prior between the two images; wherein the matching prior is a matching relationship of a matching ray pair, and the matching ray pair is composed of matching rays corresponding to the same pixel point in the two images;
[0031] A Gaussian primitive initialization module, configured to initialize a mixed Gaussian representation according to the matching prior; wherein the mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives; and the ray-based Gaussian primitives are restricted to move in the direction of the matching ray;
[0032] The Gaussian primitive optimization module is used to optimize the position and shape of the Gaussian primitive according to the matching prior to obtain a trained new perspective synthesis model; the new perspective synthesis model is used to realize new perspective image rendering.
[0033] An embodiment of the present invention further provides an image processing device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements any one of the above-described image processing methods when executing the computer program.
[0034] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the image processing methods described above.
[0035] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, the image processing method as described in any one of the above is implemented.
[0036] Compared with the prior art, the image processing method, device, equipment, storage medium and program product disclosed in the present invention use a pre-trained matching model to extract the matching prior between different images of the same scene, obtain the projection error between the images through the matching prior, and explicitly add multi-view consistency constraints to the model based on the projection error. By constructing a mixed Gaussian representation including ray-based Gaussian primitives and unstructured Gaussian primitives, the model convergence space is reduced, the model convergence speed is accelerated, and the risk of overfitting is reduced under sparse image input. In addition, the strategy of binding the matching prior and the Gaussian primitives is used to simultaneously optimize the position and shape of the Gaussian primitives, so that the optimization of the scene structure is more consistent and accurate. The present invention uses a combination of matching priors and three-dimensional Gaussian splashing technology to achieve a few-sample new perspective synthesis task, which is conducive to improving the accuracy of the rendered new perspective image. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present invention;
[0038] Figure 2 is a schematic diagram of the principle of the image processing method in an embodiment of the present invention;
[0039] Figure 3 is a structural schematic diagram of an image processing device provided by an embodiment of the present invention;
[0040] Figure 4 It is a structural schematic diagram of an image processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] In the description of the present application, it should be understood that the terms "center", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0043] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.
[0044] See also Figure 1 , is a flow chart of an image processing method provided by an embodiment of the present invention. The embodiment of the present invention provides an image processing method, including steps S11 to S13:
[0045] S11, inputting two images of the same scene at different viewing angles into a preset matching model, and extracting a matching prior between the two images; wherein the matching prior is a matching relationship of a matching ray pair, and the matching ray pair is composed of matching rays corresponding to the same pixel point in the two images;
[0046] S12, initializing a mixed Gaussian representation according to the matching prior; wherein the mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives; the ray-based Gaussian primitives are restricted to move in the direction of the matching ray;
[0047] S13. Optimize and train the position and shape of the Gaussian primitives according to the matching prior to obtain a trained new perspective synthesis model; the new perspective synthesis model is used to implement new perspective image rendering.
[0048] It should be noted that the challenge of the three-dimensional Gaussian splash model in the task of synthesizing new perspectives with few samples mainly comes from the inability to learn a consistent scene structure from multiple perspectives, that is, the position of the Gaussian primitives cannot be optimized to the exact surface vicinity. When there are only sparse input images, the supervision signal of the model comes only from the sparse training perspectives. This insufficient supervision signal makes it difficult for the model to learn a consistent scene structure, resulting in poor rendering results at the test perspective. At the same time, due to the cross-correlation of Gaussian attributes, the three-dimensional Gaussian splash model has ambiguity between the optimization of the position attribute and the shape attribute of the Gaussian primitives. The three-dimensional Gaussian splash model is more inclined to increase the size of the Gaussian primitives and it is difficult to directly optimize the position of the Gaussian primitives. It can be seen that in the few-sample new perspective synthesis scene of the prior art, there are mainly problems such as sparse input images and poor rendering results when synthesizing new perspectives due to the inability to ensure consistency among multiple perspectives.
[0049] In order to solve this problem, an embodiment of the present invention proposes a few-sample new perspective synthesis method based on scene structure consistency Gaussian splashing, which extracts a matching prior without scale problem between two images through a matching model, and is used to characterize the matching relationship of matching ray pairs composed of matching rays corresponding to the same pixel points in the two images. The matching prior obtains the projection error between the images through the corresponding relationship of the matching rays, and explicitly adds multi-perspective consistency constraints to the model based on the projection error.
[0050] Specifically, a matching model is pre-trained to extract matching priors between images. The matching model is a prior art, and its training samples are image pairs from different perspectives of the same scene, and the correspondence between the real pixels between the image pairs is pre-labeled. By inputting any two images, the pre-trained matching model is used to extract the matching priors between the two images. The two images need to be images generated from different perspectives of the same scene, and there is a certain degree of overlap between the perspectives. Of course, in actual application, multiple images can also be input, and the matching model will pair the images in pairs to extract the matching priors between any two images.
[0051] Furthermore, the mixed Gaussian representation is initialized according to the matching prior to accelerate the convergence speed and reduce the risk of overfitting. The mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives. Subsequently, the positions and shapes of the ray-based Gaussian primitives and unstructured Gaussian primitives are optimized according to the matching prior, and finally a new perspective synthesis model is obtained, so that the model can optimize the multi-perspective consistent scene structure even when the input image is very sparse, and render an accurate new perspective image.
[0052] By adopting the technical means of the embodiment of the present invention, a pre-trained matching model is used to extract the matching prior between different images of the same scene, the projection error between the images is obtained through the matching prior, and multi-view consistency constraints are explicitly added to the model based on the projection error. In addition, by constructing a mixed Gaussian representation including ray-based Gaussian primitives and unstructured Gaussian primitives, the model convergence space is reduced, the model convergence speed is accelerated, and the risk of overfitting of the model is reduced under sparse image input. In addition, the strategy of binding the matching prior and the Gaussian primitives is used to simultaneously optimize the position and shape of the Gaussian primitives, so that the optimization of the scene structure is more consistent and accurate. The present invention combines matching priors with three-dimensional Gaussian splashing technology to achieve the task of synthesizing new perspectives with a small number of samples, which is conducive to improving the accuracy of the rendered new perspective images.
[0053] As a preferred embodiment, the matching relationship includes the ray correspondence relationship of the matching ray pair and the ray position of the matching ray, that is, the matching prior includes the ray correspondence relationship and ray position between the matching rays (i.e., matching ray pairs) of the two images. The pixel point in the two images corresponding to the matching ray pair is the same, and the matching ray refers to the ray from the pixel point to the camera.
[0054] Specifically, the ray correspondence relationship refers to a pair of rays in two images that are matched to represent the 2D corresponding points of a 3D surface point at different viewing angles, which can be used as an important multi-view constraint, that is, a pair of matched rays should intersect at the same 3D surface point. i and I j The matching ray {r i ,r j}, and its corresponding pixel 2D coordinates are {p i ,p j}, that is, the ray position in the matching prior, represents the coordinates of the 2D corresponding points of the 3D surface points represented by this set of matching rays at different viewing angles. Assume that the intersection point of the matching ray and the surface has been calculated as X i and X j , which represents the Gaussian points randomly initialized when the new perspective synthesis model is initialized, is the 3D coordinate, corresponding to the 3D surface point, and as the model training is continuously optimized, the following corresponding relationship can be obtained:
[0055] X i =X j ;
[0056] At the same time, given the camera intrinsic parameters {K i ,K j} and camera extrinsics {[R i ,t i ],[R j ,tj ]}, the 3D surface point can be further projected into another 2D image plane and the corresponding projected pixel coordinates can be obtained. For example, the formula for projecting from view i to view j is as follows:
[0057]
[0058] Among them, π([x,y,z] T )=[x / z,y / z] T Therefore, the following relationship can be obtained on the pixel coordinates: j =p i→j , similarly, we can also get p i =p j→i .
[0059] In the matching prior, the matched rays are exactly those areas that are visible from at least two perspectives. These multi-view visible areas play an important role in the reconstruction model, because those areas that are only visible from a single perspective have little value for reconstruction. In sparse scenes, the stereo correspondence is not obvious, and those areas that are not covered by multiple perspectives may even affect the training of the model. The importance of multi-view visible areas in sparse reconstruction is further increased.
[0060] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of any of the above embodiments, step S12, that is, the initialization of the mixed Gaussian representation according to the matching prior, includes steps S121 to S123:
[0061] S121, initializing a ray-based Gaussian primitive corresponding to each matching ray; wherein the position of the ray-based Gaussian primitive is determined according to the ray position of the matching ray;
[0062] S122, splitting or cloning at least one unstructured Gaussian primitive from the ray-based Gaussian primitive;
[0063] S123. Form the mixed Gaussian representation according to the ray-based Gaussian primitives and the unstructured Gaussian primitives.
[0064] It should be noted that in the original three-dimensional Gaussian splash model, the Gaussian primitive is an unstructured object, that is, an unstructured Gaussian primitive, which can move in any direction in three-dimensional space, has a large convergence space and poor controllability, and it is difficult to achieve display constraints on the Gaussian primitive. At the same time, when the input image is relatively sparse, the large convergence space easily causes the model to fall into overfitting.
[0065] In order to improve the controllability of Gaussian primitives and make the three-dimensional Gaussian splash model converge to the optimal solution on a sparse image, an embodiment of the present invention proposes a mixed Gaussian representation, which is composed of ray-based Gaussian primitives and original unstructured Gaussian primitives. The position of the ray-based Gaussian primitive is restricted to move in the direction of the ray, and the unstructured Gaussian primitive can move in any direction in the three-dimensional space.
[0066] Among them, the ray-based Gaussian primitives greatly reduce the space for model convergence by restricting the Gaussian primitives to move only in the ray direction, accelerate the model convergence speed, and reduce the risk of overfitting of the model in sparse scenes. Unstructured Gaussian primitives can move freely in three-dimensional space, which is conducive to recovering background areas that cannot be covered by ray-based Gaussian primitives. The unstructured Gaussian primitives are obtained from ray-based Gaussian primitives by splitting or cloning.
[0067] Each Gaussian primitive has the following properties: position μ, scaling factor s, rotation factor r, opacity α, and spherical harmonic coefficients sh. The only difference between ray-based Gaussian primitives and unstructured Gaussian primitives is the representation of position μ. These parameters are randomly initialized when the model is defined.
[0068] When the model is initialized, it will only initialize the ray-based Gaussian primitives based on the ray positions in the matching prior. Then, during the training process, these ray-based Gaussian primitives are cloned or split to obtain unstructured Gaussian primitives.
[0069] The specific process is as follows: Based on the matching prior, the matching rays between images can be obtained, and a ray-based Gaussian basis element is initialized at the position of each matched ray. Its properties are defined as described above, including position, scaling factor, rotation factor, opacity, and spherical harmonic coefficients. Except for the position, which is determined based on the ray position in the matching prior, the other parameters are obtained through random initialization and will be continuously updated as the model is trained.
[0070] The position of the ray-based Gaussian primitive is constrained to move in the direction of the ray, and its position is defined as follows:
[0071] μ′=o+zd;
[0072] Among them, o and d represent the camera center and ray direction of a ray, respectively, determined based on the ray position in the matching prior, and z is a learnable distance factor. In the densification stage, the same scheme as the original 3D Gaussian splatter model is adopted, using the moving average of the gradient in the view space to determine the under-reconstructed candidate Gaussian primitives from the Gaussian primitives of the primitive rays, and splitting the unstructured Gaussian primitives from these candidate Gaussian primitives.
[0073] Split and clone process: The moving average of the gradient at the current moment = the moving average of the gradient at the previous moment (initial value is 0) + the gradient at the current moment * the preset factor, which is between 0 and 1. If the gradient of the Gaussian primitive is greater than the preset first threshold, the model will clone the Gaussian primitive. During the cloning process, all attribute values of the Gaussian primitive will be updated according to the gradient descent algorithm. If the scaling factor of a Gaussian primitive is greater than the preset second threshold, the Gaussian primitive is considered to be too large, and the model will split the Gaussian primitive.
[0074] During model training, according to the cloning and splitting process described above, unstructured Gaussian primitives are continuously split and cloned from ray-based Gaussian primitives. The properties of these cloned unstructured Gaussian primitives are inherited from the properties of their corresponding ray-based Gaussian primitives. The splitting stop condition can be that the model running time reaches 70% of the training cycle.
[0075] For each ray, there is a corresponding ray-based Gaussian primitive and a plurality of unstructured Gaussian primitives cloned from the ray-based Gaussian primitive.
[0076] By adopting the technical means of the embodiments of the present invention, a mixed Gaussian representation including ray-based Gaussian primitives and original unstructured Gaussian primitives is proposed. The position of the ray-based Gaussian primitives is restricted to move in the direction of the ray, and the unstructured Gaussian primitives are obtained by splitting and cloning the ray-based Gaussian primitives. The mixed Gaussian primitive representation of the present invention effectively reduces the space for model convergence, accelerates the model convergence speed, and reduces the risk of overfitting of the model in sparse scenarios.
[0077] As a preferred embodiment, see Figure 2 , is a schematic diagram of the principle of the image processing method in an embodiment of the present invention. The embodiment of the present invention is further implemented on the basis of any of the above embodiments. Step S13, that is, optimizing the position and shape of the Gaussian primitives according to the matching prior to obtain a trained new perspective synthesis model, includes steps S131 to S132:
[0078] S131, calculating position loss, rendering loss and color loss according to the matching prior;
[0079] S132. Optimize and train the position of the Gaussian primitives according to the position loss, and optimize and train the shape of the Gaussian primitives according to the rendering loss and the color loss, to obtain a trained new perspective synthesis model.
[0080] It should be noted that the main factors affecting the rendering quality are the accuracy of the position and shape of the Gaussian primitives. Therefore, by optimizing the position of the ray-based Gaussian primitives and optimizing the rendering geometry, the position and shape of the Gaussian primitives are optimized respectively, thereby improving the synthesis quality of the new perspective.
[0081] During the training iteration, the position loss L is calculated based on the matching prior. gp , rendering loss L rg and color loss L photo , and then according to the position loss L gp Optimize the position of Gaussian primitives based on the rendering loss L rg and color loss L photo Optimize the shape of the Gaussian primitives until the end of training and save the final mixed Gaussian primitive representation.
[0082] Preferably, the calculating the position loss according to the matching prior comprises:
[0083] Binding the ray-based Gaussian primitives to matching rays in the matching prior;
[0084] According to the ray correspondence of the matching ray pair, a matching correspondence between two ray-based Gaussian primitives in a ray-based Gaussian primitive pair is obtained; wherein the ray-based Gaussian primitive pair is two ray-based Gaussian primitives bound to two matching rays in the matching ray pair;
[0085] Calculating a projection error between two ray-based Gaussian primitives in the ray-based Gaussian primitive pair according to the ray position of the matching ray and the matching correspondence;
[0086] The position loss is calculated based on the projection errors of all ray-based Gaussian basis pairs.
[0087] In the embodiment of the present invention, the ray-based Gaussian primitives are bound to the ray positions of the matching rays in the matching prior, and the positions of the ray-based Gaussian primitives and the rendering geometry are explicitly optimized according to the corresponding relationship of the rays.
[0088] Binding corresponds to initializing the ray-based Gaussian primitives according to the matching prior in the previous step, that is, according to the matching rays between the images obtained in the matching prior, a ray-based Gaussian primitive is initialized on these matched rays, which is equivalent to binding the ray-based Gaussian primitives and the matching rays. This binding relationship can make the ray-based Gaussian primitives also have a matching correspondence between matching rays. The model training process is the subsequent optimization process.
[0089] Based on the binding relationship, the matching correspondence between ray-based Gaussian primitives can be obtained. Based on this correspondence, the projection error between ray-based Gaussian primitives can be obtained to explicitly optimize the position of the ray-based Gaussian primitives. The specific process is as follows:
[0090] For a pair in image I i and I j The matching ray {r i ,r j}, and their corresponding pair of ray-based Gaussian basis elements are After initialization, their positions in 3D space are μ i ′ =o i +z i d i and μ j ′ =o j +z j d j . Calculate the 2D projection coordinates p from view i to view j i→j (μ i ′ ) and the projection coordinates p from view j to view i j→i (μ j ′ ). Therefore, the projection error of this pair of ray-based Gaussian primitives can be obtained as follows:
[0091]
[0092] The final ray-based Gaussian primitive position loss L gp is the average of the projection errors of all bound ray-based Gaussian primitive pairs.
[0093] Based on this position loss, the position of the ray-based Gaussian primitives can be explicitly optimized to make them as accurate as possible to the surface attachment, which helps with multi-view consistency of new view synthesis.
[0094] Preferably, the calculating the rendering loss and the color loss according to the matching prior comprises:
[0095] Inputting the mixed Gaussian representation and the camera poses of the two images into a Gaussian rasterizer to obtain a rendered depth map of the two images corresponding to the viewing angle, so as to obtain a depth value of the matching ray pair;
[0096] According to the depth value of the matching ray pair, two pixel points corresponding to the matching ray pair are mapped to a three-dimensional space, and the three-dimensional coordinates of the two pixel points are calculated;
[0097] Calculating a projection error based on the three-dimensional coordinates and the ray positions of the matching ray pairs;
[0098] Calculate the rendering loss according to the projection errors of all the matching ray pairs;
[0099] The real images of the two images are rendered by Gaussian rasterization to obtain a rendering color map;
[0100] The loss between the rendered color map and the real image is calculated to obtain a color loss.
[0101] In an embodiment of the present invention, the rendering geometry is optimized by matching the ray correspondence of the rays. This step is mainly used to optimize the shape of the Gaussian primitives. Due to the cross-correlation of Gaussian properties, the rendering geometry of the model is not consistent with the Gaussian position. Even when the Gaussian primitive position is accurate, if the scale and rotation of the Gaussian primitive are inaccurate, it will still lead to incorrect rendering results and affect the new viewpoint synthesis effect. This method further uses the rendering loss and color loss of the rendering depth to constrain the shape of the Gaussian primitives.
[0102] After obtaining the mixed Gaussian representation, the mixed Gaussian representation and the camera poses corresponding to the two images are input into the Gaussian rasterizer to obtain the rendered depth map D corresponding to the perspective of the two images. i and D j , and get the depth value of the matching ray {D i (p i ),D j (p j )}. The specific process is as follows:
[0103] Given a set of Gaussian primitives (represented by mixed Gaussian primitives) and the camera pose of the desired rendering perspective. Based on the spherical harmonic coefficient factor sh in the attributes of each Gaussian primitive, the color value c of each Gaussian primitive in the direction of the given camera pose can be obtained. At the same time, the position, rotation, and translation factors of the Gaussian primitives are updated to obtain the depth value d of each Gaussian primitive in the given camera pose. The 3DGS model will first sort the Gaussian primitives according to their depth, and then obtain the rendered image through the following formula:
[0104]
[0105] The same is true for getting the rendering depth, except that c is replaced by d, that is:
[0106]
[0107] Where N is the total number of all Gaussian basis points, c i refers to the color of the i-th Gaussian basis element, d i is the depth of the i-th Gaussian primitive, α′ iIt refers to the opacity of the i-th Gaussian primitive at a given camera pose, which is determined by the opacity in the Gaussian primitive attributes together with the position, rotation, and scale of the Gaussian primitive.
[0108] After obtaining the depth value of the matching ray {D i (p i ),D j (p j )}, the pixel coordinates are projected back into 3D space according to the following formula:
[0109]
[0110] in Refers to the homogeneous coordinate form of p. Similarly, the 3D coordinate v corresponding to ray j can also be obtained j Further projection calculations can be performed to obtain the projected 2D coordinates. Therefore, the projection error based on the rendering depth can be calculated as:
[0111]
[0112] The average of the errors of all ray pairs is taken as the final rendering loss L rg .
[0113] As with the depth rendering process, given a camera pose and its real image I, the corresponding color map is obtained by Gaussian rasterization rendering Calculate the weighted sum of the L1 loss and SSIM loss of the rendered image and the real image as the color rendering loss, and the final color loss L photo The formula is as follows:
[0114]
[0115] Based on the rendering loss L rg and color loss L photo To optimize the model, better optimization of the Gaussian primitive shape can be achieved.
[0116] Therefore, the overall loss of the embodiment of the present invention is a combination of position loss, rendering loss and color loss:
[0117] L=L photo +βL gp +βL rg ;
[0118] During the training iteration, according to the position loss L gp Optimize the position of Gaussian primitives based on the rendering loss L rg and color loss L photo Optimize the shape of the Gaussian primitives until the end of training and save the final mixed Gaussian primitive representation.
[0119] By adopting the technical means of the embodiments of the present invention, the position and shape of the Gaussian primitives are optimized simultaneously by binding the matching rays and Gaussian primitives, so that the optimization of the scene structure is more consistent and accurate, so that the model can optimize the multi-perspective consistent scene structure and render accurate new perspective images even when the input image is very sparse.
[0120] As a preferred implementation, the embodiment of the present invention is further implemented on the basis of any of the above embodiments, and the method further includes step S14:
[0121] S14: Input the camera posture of the image to be processed into the new perspective synthesis model for rendering processing to obtain a new perspective image of the image to be processed.
[0122] In an embodiment of the present invention, the new perspective synthesis model used in this scheme integrates a matching model, a three-dimensional Gaussian splatter (3DGS) model and a Gaussian rasterizer. After the model training is completed, the final optimized mixed Gaussian primitive representation is saved.
[0123] During the use stage of the new perspective synthesis model, the camera poses of different perspective images of the same object as those when the new perspective synthesis model was trained are input. The new perspective synthesis model uses the position μ, scaling factor s, and rotation factor in the basic attributes of each primitive in the optimized hybrid Gaussian primitive representation to project the Gaussian primitives in the three-dimensional space back to the two-dimensional space. Then, similar to the prior art, the Gaussian primitives are sorted based on the depth relationship of each Gaussian primitive, and finally a point cloud-based rendering method is used to render the new perspective image.
[0124] The overall principle of the embodiment of the present invention is as follows Figure 2 As shown, is the input K sparse images, i is the image index subscript, μ' 1 ′ and μ' 2 ′ represent the positions of two ray-based Gaussian primitives, L1 refers to the absolute error between the rendered image and the real image, and SSIM refers to the structural similarity loss between the rendered image and the real image.
[0125] The technical means of the embodiments of the present invention are adopted to solve the problem that in a few-sample new perspective synthesis scenario, the input image is sparse and the multi-perspective consistency cannot be guaranteed, resulting in poor rendering results when synthesizing the new perspective. The model can optimize the scene structure with consistent multi-perspectives even when the input image is very sparse, thereby improving the accuracy of new perspective image rendering.
[0126] See also Figure 3 , is a schematic diagram of the structure of an image processing device provided by an embodiment of the present invention. The embodiment of the present invention provides an image processing device 20, including:
[0127] A matching prior extraction module 21 is used to input two images of the same scene at different viewing angles into a preset matching model to extract a matching prior between the two images; wherein the matching prior is a matching relationship of a matching ray pair, and the matching ray pair is composed of matching rays corresponding to the same pixel point in the two images;
[0128] A Gaussian primitive initialization module 22, configured to initialize a mixed Gaussian representation according to the matching prior; wherein the mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives; the ray-based Gaussian primitives are restricted to move in the direction of the matching ray;
[0129] The Gaussian primitive optimization module 23 is used to optimize the position and shape of the Gaussian primitive according to the matching prior to obtain a trained new perspective synthesis model; the new perspective synthesis model is used to realize new perspective image rendering.
[0130] As a preferred implementation, the matching relationship includes a ray correspondence relationship of the matching ray pair and a ray position of the matching ray.
[0131] As a preferred implementation, the initializing the mixed Gaussian representation according to the matching prior includes:
[0132] Initializing a ray-based Gaussian primitive corresponding to each matching ray; wherein the position of the ray-based Gaussian primitive is determined according to the ray position of the matching ray;
[0133] Splitting or cloning at least one unstructured Gaussian primitive from the ray-based Gaussian primitive;
[0134] The mixed Gaussian representation is formed based on the ray-based Gaussian primitives and the unstructured Gaussian primitives.
[0135] As a preferred implementation, the position and shape of the Gaussian primitives are optimized and trained according to the matching prior to obtain a trained new perspective synthesis model, including:
[0136] Calculating position loss, rendering loss and color loss based on the matching prior;
[0137] The position of the Gaussian primitive is optimized and trained according to the position loss, and the shape of the Gaussian primitive is optimized and trained according to the rendering loss and the color loss, so as to obtain a trained new perspective synthesis model.
[0138] Preferably, the calculating the position loss according to the matching prior comprises:
[0139] Binding the ray-based Gaussian primitives to matching rays in the matching prior;
[0140] According to the ray correspondence of the matching ray pair, a matching correspondence between two ray-based Gaussian primitives in a ray-based Gaussian primitive pair is obtained; wherein the ray-based Gaussian primitive pair is two ray-based Gaussian primitives bound to two matching rays in the matching ray pair;
[0141] Calculating a projection error between two ray-based Gaussian primitives in the ray-based Gaussian primitive pair according to the ray position of the matching ray and the matching correspondence;
[0142] The position loss is calculated based on the projection errors of all ray-based Gaussian basis pairs.
[0143] Preferably, the calculating the rendering loss and the color loss according to the matching prior comprises:
[0144] Inputting the mixed Gaussian representation and the camera poses of the two images into a Gaussian rasterizer to obtain a rendered depth map of the two images corresponding to the viewing angle, so as to obtain a depth value of the matching ray pair;
[0145] According to the depth value of the matching ray pair, two pixel points corresponding to the matching ray pair are mapped to a three-dimensional space, and the three-dimensional coordinates of the two pixel points are calculated;
[0146] Calculating a projection error based on the three-dimensional coordinates and the ray positions of the matching ray pairs;
[0147] Calculate the rendering loss according to the projection errors of all the matching ray pairs;
[0148] The real images of the two images are rendered by Gaussian rasterization to obtain a rendering color map;
[0149] The loss between the rendered color map and the real image is calculated to obtain a color loss.
[0150] By adopting the technical means of the embodiment of the present invention, a pre-trained matching model is used to extract the matching prior between different images of the same scene, the projection error between the images is obtained through the matching prior, and multi-view consistency constraints are explicitly added to the model based on the projection error. In addition, by constructing a mixed Gaussian representation including ray-based Gaussian primitives and unstructured Gaussian primitives, the model convergence space is reduced, the model convergence speed is accelerated, and the risk of overfitting of the model is reduced under sparse image input. In addition, the strategy of binding the matching prior and the Gaussian primitives is used to simultaneously optimize the position and shape of the Gaussian primitives, so that the optimization of the scene structure is more consistent and accurate. The present invention combines matching priors with three-dimensional Gaussian splashing technology to achieve the task of synthesizing new perspectives with a small number of samples, which is conducive to improving the accuracy of the rendered new perspective images.
[0151] It should be noted that an image processing device provided in an embodiment of the present invention is used to execute all process steps of an image processing method in the above embodiment, and the working principles and beneficial effects of the two correspond one to one, so they will not be described in detail.
[0152] See also Figure 4 , is a structural diagram of an image processing device provided by an embodiment of the present invention. The embodiment of the present invention provides an image processing device 30, including a processor 31, a memory 32, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the image processing method described in any one of the above embodiments.
[0153] An embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the image processing method as described in any one of the above embodiments.
[0154] An embodiment of the present invention further provides a computer program product, which includes a computer program or computer instructions. When the computer program or the computer instructions are executed by a processor, the image processing method as described in any one of the above embodiments is implemented.
[0155] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0156] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.
Claims
1. An image processing method, characterized in that: include: Input two images of the same scene at different viewing angles into a preset matching model to extract a matching prior between the two images; wherein the matching prior is a matching relationship of a matching ray pair, and the matching ray pair is composed of matching rays corresponding to the same pixel point in the two images; Initializing a mixed Gaussian representation according to the matching prior; wherein the mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives; the ray-based Gaussian primitives are restricted to move in the direction of the matching ray; The position and shape of the Gaussian primitives are optimized and trained according to the matching prior to obtain a trained new perspective synthesis model; the new perspective synthesis model is used to realize new perspective image rendering.
2. The image processing method according to claim 1, characterized in that: The matching relationship includes the ray correspondence relationship of the matching ray pair and the ray position of the matching ray.
3. The image processing method according to claim 2, characterized in that: Initializing the mixed Gaussian representation according to the matching prior includes: Initializing a ray-based Gaussian primitive corresponding to each matching ray; wherein the position of the ray-based Gaussian primitive is determined according to the ray position of the matching ray; Splitting or cloning at least one unstructured Gaussian primitive from the ray-based Gaussian primitive; The mixed Gaussian representation is formed based on the ray-based Gaussian primitives and the unstructured Gaussian primitives.
4. The image processing method according to claim 2, wherein: The step of optimizing and training the position and shape of the Gaussian primitives according to the matching prior to obtain a trained new perspective synthesis model includes: Calculating position loss, rendering loss and color loss based on the matching prior; The position of the Gaussian primitive is optimized and trained according to the position loss, and the shape of the Gaussian primitive is optimized and trained according to the rendering loss and the color loss, so as to obtain a trained new perspective synthesis model.
5. The image processing method according to claim 4, characterized in that: The calculating the position loss according to the matching prior includes: Binding the ray-based Gaussian primitives to matching rays in the matching prior; According to the ray correspondence of the matching ray pair, a matching correspondence between two ray-based Gaussian primitives in a ray-based Gaussian primitive pair is obtained; wherein the ray-based Gaussian primitive pair is two ray-based Gaussian primitives bound to two matching rays in the matching ray pair; Calculating a projection error between two ray-based Gaussian primitives in the ray-based Gaussian primitive pair according to the ray position of the matching ray and the matching correspondence; The position loss is calculated based on the projection errors of all ray-based Gaussian basis pairs.
6. The image processing method according to claim 4, characterized in that: The calculating the rendering loss and the color loss according to the matching prior includes: Inputting the mixed Gaussian representation and the camera poses of the two images into a Gaussian rasterizer to obtain a rendered depth map of the two images corresponding to the viewing angle, so as to obtain a depth value of the matching ray pair; According to the depth value of the matching ray pair, two pixel points corresponding to the matching ray pair are mapped to a three-dimensional space, and the three-dimensional coordinates of the two pixel points are calculated; Calculating a projection error based on the three-dimensional coordinates and the ray positions of the matching ray pairs; Calculate the rendering loss according to the projection errors of all the matching ray pairs; The real images of the two images are rendered by Gaussian rasterization to obtain a rendering color map; The loss between the rendered color map and the real image is calculated to obtain a color loss.
7. An image processing device, characterized in that: include: A matching prior extraction module is used to input two images of the same scene at different viewing angles into a preset matching model to extract a matching prior between the two images; wherein the matching prior is a matching relationship of a matching ray pair, and the matching ray pair is composed of matching rays corresponding to the same pixel point in the two images; A Gaussian primitive initialization module, configured to initialize a mixed Gaussian representation according to the matching prior; wherein the mixed Gaussian representation includes ray-based Gaussian primitives and unstructured Gaussian primitives; and the ray-based Gaussian primitives are restricted to move in the direction of the matching ray; The Gaussian primitive optimization module is used to optimize the position and shape of the Gaussian primitive according to the matching prior to obtain a trained new perspective synthesis model; the new perspective synthesis model is used to realize new perspective image rendering.
8. An image processing device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the image processing method according to any one of claims 1 to 6 when executing the computer program.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 6.
10. A computer program product, characterized in that The computer program product comprises a computer program or a computer instruction, and when the computer program or the computer instruction is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.