Gaussian-splatting based multi-person novel view rendering method and system with large baseline

Abstract

The application provides a large-baseline multi-person new view rendering method and system based on Gaussian splashing. The method comprises the following steps: acquiring a plurality of target viewpoints, and using a plurality of cameras to collect multi-person images to form a plurality of image pairs, then performing stereo correction on the plurality of image pairs to obtain a plurality of second image pairs; inputting the plurality of second image pairs into a pre-trained disparity prediction model to obtain a predicted disparity map of each second image pair; determining a depth map of each second image pair, and inputting each depth map and each second image pair into a pre-trained Gaussian point parameter prediction model to obtain a Gaussian point prediction parameter map of each second image pair and a color corresponding to the Gaussian point; determining a Gaussian point cloud of each second image pair, and rendering a plurality of target viewpoints based on each Gaussian point cloud to obtain a plurality of rendered images. The technical scheme provided by the application can achieve the purpose of efficient and rapid rendering by combining estimated depth and Gaussian splashing rendering.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and system for rendering new multi-viewpoint rendering with large baselines based on Gaussian splashing. Background Technology

[0002] Real-time rendering from novel viewpoints is a crucial area in computer vision, with broad prospects for academic and industry-academia collaboration. Achievements in this field have already seen numerous applications in holographic communication and video conferencing. The main challenge lies in generating high-quality, real-time multi-object viewpoints. This requires rendering methods with generalizability and the ability to rapidly render multiple viewpoints. However, existing techniques, such as neural radiation field methods, do not perform well in terms of generalizability and real-time performance. Therefore, there is an urgent need to propose a rendering scheme that excels in both generalizability and real-time performance. Summary of the Invention

[0003] This application provides a new multi-viewpoint rendering method and system based on Gaussian splashing with large baselines, in order to at least solve the technical problems of poor generalization and real-time performance of existing technologies.

[0004] The first aspect of this application proposes a new multi-viewpoint rendering method based on Gaussian splashing with a large baseline, the method comprising:

[0005] Multiple target viewpoints are acquired, and multiple images of multiple people are collected to form multiple first image pairs. Stereo correction is performed on each of the multiple first image pairs to obtain multiple second image pairs. The multiple second image pairs are respectively input into a pre-trained disparity prediction model to obtain a predicted disparity map for each second image pair; The depth map of each second image pair is determined based on the predicted disparity map of each second image pair. The depth map of each second image pair and the second image pair corresponding to each depth map are then input into a pre-trained Gaussian point parameter prediction model to obtain the Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair. Based on each second image pair and its depth map, Gaussian point prediction parameter map, and the color corresponding to the Gaussian point, the Gaussian point cloud of each second image pair is determined, and the Gaussian point cloud of each second image pair is used to render multiple target viewpoints to obtain the rendered image of each first image pair under multiple target viewpoints.

[0006] Preferably, the step of using multiple cameras to acquire images of multiple people to form multiple first image pairs includes: Multiple cameras arranged horizontally are used to simultaneously and uniformly acquire images of multiple people on a large baseline. Images of multiple people captured by two adjacent cameras are grouped together to form the first image pair.

[0007] Further, determining the depth map of each of the second image pairs based on the predicted disparity maps of each of the image pairs includes: Obtain each disparity value in the predicted disparity map of each of the second image pairs; The depth value of each second image pair is determined based on the disparity values ​​in the predicted disparity map of each second image pair, the baseline length between the two cameras of each second image pair, the focal length of the reference camera, and the principal point offset value of the two cameras after stereo correction. Depth maps for each of the second image pairs are determined based on the depth values ​​of each second image pair.

[0008] Furthermore, the training process of the disparity prediction model includes: Obtain multiple corrected image pairs and the true disparity map of each image pair; The initial recurrent neural network is trained by taking the multiple corrected image pairs as inputs, the true disparity maps of each image pair as outputs, and the absolute value loss as the loss function, to obtain a trained disparity prediction model.

[0009] Furthermore, the training process of the Gaussian point parameter prediction model includes: Acquire multiple corrected image pairs, depth maps of each image pair, and the true image of the target viewpoint; The initial convolutional network is trained by taking multiple corrected image pairs and the depth maps of each image pair as inputs, and the real images of the target viewpoints of each image pair as outputs. The structural similarity loss and absolute value loss are weighted as the loss function to obtain the trained Gaussian point parameter prediction model.

[0010] Furthermore, the parameter map includes: a Gaussian point rotation parameter map, a Gaussian point size parameter map, and a Gaussian point transparency parameter map.

[0011] Furthermore, determining the Gaussian point cloud of each second image pair based on each second image pair, its depth map, Gaussian point prediction parameter map, and the color corresponding to the Gaussian point includes: Back-projection is performed on each of the second image pairs and their depth maps, Gaussian point prediction parameter maps, and the colors corresponding to the Gaussian points to obtain the Gaussian point clouds of each of the second image pairs.

[0012] A second aspect of this application proposes a large-baseline multi-viewpoint rendering system based on Gaussian splashing, comprising: The acquisition module is used to acquire multiple target viewpoints and collect images of multiple people to form multiple first image pairs. Stereo correction is performed on the multiple first image pairs to obtain multiple second image pairs. The disparity determination module is used to input the plurality of second image pairs into a pre-trained disparity prediction model to obtain a predicted disparity map for each second image pair. The parameter determination module is used to determine the depth map of each second image pair based on the predicted disparity map of each second image pair, and input the depth map of each second image pair and the second image pair corresponding to each depth image into the pre-trained Gaussian point parameter prediction model to obtain the Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair. The rendering module is used to determine the Gaussian point cloud of each second image pair based on the depth map, Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair, and to render the multiple target viewpoints based on the Gaussian point cloud of each second image pair to obtain the rendered image of each first image pair under the multiple target viewpoints.

[0013] In this embodiment of the disclosure, the acquisition module is further configured to: Multiple cameras arranged horizontally are used to simultaneously and uniformly acquire images of multiple people on a large baseline. Images of multiple people captured by two adjacent cameras are grouped together to form an image pair.

[0014] Furthermore, the parameter determination module is also used for: Obtain each disparity value in the predicted disparity map of each of the second image pairs; The depth value of each second image pair is determined based on the disparity values ​​in the predicted disparity map of each second image pair, the baseline length between the two cameras of each second image pair, the focal length of the reference camera, and the principal point offset value of the two cameras after stereo correction. Depth maps for each of the second image pairs are determined based on the depth values ​​of each second image pair.

[0015] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects: This application proposes a large-baseline multi-viewpoint rendering method and system based on Gaussian splashing. The method includes: acquiring multiple target viewpoints and collecting multi-person images to form multiple first image pairs; performing stereo correction on the multiple first image pairs to obtain multiple second image pairs; inputting the multiple second image pairs into a pre-trained disparity prediction model to obtain a predicted disparity map for each second image pair; determining the depth map of each second image pair based on the predicted disparity map, and inputting the depth map of each second image pair and the second image pair corresponding to each depth map into a pre-trained Gaussian point parameter prediction model to obtain a Gaussian point prediction parameter map and the color corresponding to the Gaussian point for each second image pair; determining the Gaussian point cloud of each second image pair based on the depth map, Gaussian point prediction parameter map, and the color corresponding to the Gaussian point for each second image pair; and rendering the multiple target viewpoints based on the Gaussian point clouds of each second image pair to obtain the rendered image of each first image pair under the multiple target viewpoints. The technical solution proposed in this application, which combines depth estimation and Gaussian splash rendering, can achieve efficient and fast rendering, and also has good generalization on various types of data.

[0016] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0017] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a large baseline multi-viewpoint rendering method based on Gaussian splashing according to an embodiment of this application; Figure 2 This is a schematic diagram of the rendering results of one target viewpoint generated by each of three pairs of cameras according to an embodiment of this application; Figure 3 This is a structural diagram of a large baseline multi-view rendering system based on Gaussian splashing provided in one embodiment of this application. Detailed Implementation

[0018] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0019] This application proposes a method and system for rendering new multi-viewpoint images with a large baseline based on Gaussian splashing. The method includes: acquiring multiple target viewpoints and collecting multiple images of multiple people to form multiple first image pairs; performing stereo correction on the multiple first image pairs to obtain multiple second image pairs; inputting the multiple second image pairs into a pre-trained disparity prediction model to obtain a predicted disparity map for each second image pair; determining the depth map of each second image pair based on the predicted disparity map of each second image pair, and inputting the depth map of each second image pair and the second image pair corresponding to each depth map into a pre-trained Gaussian point parameter prediction model to obtain a Gaussian point prediction parameter map and the color corresponding to the Gaussian point for each second image pair; determining the Gaussian point cloud of each second image pair based on the depth map, Gaussian point prediction parameter map, and the color corresponding to the Gaussian point for each second image pair, and rendering the multiple target viewpoints based on the Gaussian point clouds of each second image pair to obtain the rendered image of each first image pair under the multiple target viewpoints. The technical solution proposed in this application, which combines depth estimation and Gaussian splash rendering, can achieve efficient and fast rendering, and also has good generalization on various types of data.

[0020] The following describes, with reference to the accompanying drawings, a method and system for rendering large baseline multi-person new viewpoints based on Gaussian splashing, according to embodiments of this application.

[0021] Example 1 Figure 1 The flowchart below shows a method for rendering new multi-viewpoint rendering based on Gaussian splashing with a large baseline according to an embodiment of this application. Figure 1 As shown, the method includes: Step 1: Acquire multiple target viewpoints and collect images of multiple people to form multiple first image pairs. Perform stereo correction on each of the multiple first image pairs to obtain multiple second image pairs.

[0022] In this embodiment of the disclosure, the step of using multiple cameras to acquire images of multiple people to form multiple first image pairs includes: Multiple cameras arranged horizontally are used to simultaneously and uniformly acquire images of multiple people on a large baseline. Images of multiple people captured by two adjacent cameras are grouped together to form an image pair.

[0023] For example, four horizontally arranged RGB cameras uniformly acquire images of multiple people on a large baseline and form three image pairs.

[0024] It should be noted that stereo correction is performed on each pair of images to ensure that only horizontal parallax exists in each pair of images.

[0025] Step 2: Input the multiple second image pairs into the pre-trained disparity prediction model to obtain the predicted disparity map of each second image pair.

[0026] It should be noted that the training process of the disparity prediction model includes: Obtain multiple corrected image pairs and the true disparity map of each image pair, wherein the corrected image pairs are stereo-verified image pairs; The initial recurrent neural network is trained by taking the multiple corrected image pairs as inputs, the true disparity maps of each image pair as outputs, and the absolute value loss as the loss function, to obtain a trained disparity prediction model. The absolute value is the absolute error between the predicted disparity map output by the disparity prediction model and its corresponding true disparity map during the training and verification process.

[0027] Step 3: Determine the depth map of each second image pair based on the predicted disparity map of each second image pair, and input the depth map of each second image pair and the second image pair corresponding to each depth image into the pre-trained Gaussian point parameter prediction model to obtain the Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair.

[0028] In this embodiment of the disclosure, determining the depth map of each of the second image pairs based on the predicted disparity maps of each of the image pairs includes: Obtain each disparity value in the predicted disparity map of each of the second image pairs; The depth value of each second image pair is determined based on the disparity values ​​in the predicted disparity map of each second image pair, the baseline length between the two cameras of each second image pair, the focal length of the reference camera, and the principal point offset value of the two cameras after stereo correction. Depth maps for each of the second image pairs are determined based on the depth values ​​of each second image pair.

[0029] It should be noted that using depth values With disparity value The relationship between the two can lead to the formula for calculating the depth value as follows:

[0030] In the formula, The baseline length between two adjacent cameras. For reference camera focal length, This is to correct the offset value of the principal points of two adjacent cameras.

[0031] In this embodiment of the disclosure, the parameter map includes: a Gaussian point rotation parameter map, a Gaussian point size parameter map, and a Gaussian point transparency parameter map.

[0032] It should be noted that the depth map of each of the second image pairs and the second image corresponding to each of the depth image pairs are respectively input into the pre-trained Gaussian point parameter prediction model for encoding and decoding to obtain the Gaussian point prediction parameter map of each of the second image pairs and the color corresponding to the Gaussian point.

[0033] It should be noted that the training process of the Gaussian point parameter prediction model includes: Obtaining a plurality of corrected image pairs, a depth map of each image pair, and a real image of a target viewpoint, wherein the corrected image pairs are stereoscopic verification image pairs. The initial convolutional network is trained by taking the plurality of corrected image pairs and the depth map of each image pair as input, taking the real image of the target viewpoint of each image pair as output, and taking the structural similarity loss and the absolute value loss as a loss function, to obtain the trained Gaussian point parameter prediction model.

[0034] Step 4: determining the Gaussian point cloud of each of the second image pairs based on each of the second image pairs, the depth map of each of the second image pairs, the Gaussian point prediction parameter map, and the color corresponding to the Gaussian point, and rendering each of the target viewpoints based on the Gaussian point cloud of each of the second image pairs to obtain the rendered images of each of the first image pairs at the plurality of target viewpoints.

[0035] In the embodiments of the present disclosure, the determination of the Gaussian point cloud of each of the second image pairs based on each of the second image pairs, the depth map of each of the second image pairs, the Gaussian point prediction parameter map, and the color corresponding to the Gaussian point includes: The Gaussian point cloud of each of the second image pairs is obtained by back-projection based on each of the second image pairs, the depth map of each of the second image pairs, the Gaussian point prediction parameter map, and the color corresponding to the Gaussian point.

[0036] For example, when the number of target viewpoints of each image pair is 15, the 15 target viewpoints are rendered based on the Gaussian point cloud of each image pair, and 45 horizontally distributed rendered images can be obtained, and further, 45 horizontally distributed rendered results are provided for the holographic display. Figure 2 As shown in FIG. 6, 1 target viewpoint rendered result is generated for each of the 3 pairs of cameras.

[0037] In summary, the large-baseline multi-person new viewpoint rendering method based on Gaussian splatting proposed in the embodiments can achieve efficient and fast rendering by combining estimated depth and Gaussian splatting rendering, and has good generalization on various data.

[0038] Embodiment Two Figure 3 The structure diagram of a large-baseline multi-person new viewpoint rendering system based on Gaussian splatting provided according to an embodiment of the present application is shown in FIG. 7.Figure 3 The system comprises: An acquisition module 100 is configured to acquire a plurality of target viewpoints and collect multi-person images to form a plurality of first image pairs, and perform stereo rectification on the plurality of first image pairs to obtain a plurality of second image pairs; A disparity determination module 200 is configured to input the plurality of second image pairs into a pre-trained disparity prediction model to obtain a predicted disparity map of each second image pair; A parameter determination module 300 is configured to determine a depth map of each second image pair according to the predicted disparity map of each second image pair, and input the depth map of each second image pair and the corresponding second image pair into a pre-trained Gaussian point parameter prediction model to obtain a Gaussian point prediction parameter map of each second image pair and a color corresponding to the Gaussian point. A rendering module 400 is configured to determine a Gaussian point cloud of each second image pair based on each second image pair, the depth map of each second image pair, the Gaussian point prediction parameter map of each second image pair, and the color corresponding to the Gaussian point, and render each target viewpoint based on the Gaussian point cloud of each second image pair to obtain a rendered image of each first image pair at the plurality of target viewpoints.

[0039] The parameter map comprises a Gaussian point rotation parameter map, a Gaussian point size parameter map, and a Gaussian point transparency parameter map.

[0040] In the embodiments of the present disclosure, the acquisition module 100 is further configured to: Collect multi-person images on a large baseline simultaneously and uniformly by using a plurality of horizontally arranged cameras; Divide the multi-person images collected by each two adjacent cameras into a group to form an image pair.

[0041] Further, the parameter determination module 300 is further configured to: Obtain each disparity value in the predicted disparity map of each second image pair; Determine a depth value of each second image pair based on each disparity value in the predicted disparity map of each second image pair, a baseline length between two cameras of each second image pair, a focal length of a reference camera, and a principal point offset value of the two cameras after stereo rectification; Determine a depth map of each second image pair based on the depth value of each second image pair.

[0042] It should be noted that the training process of the disparity prediction model comprises: Obtain a plurality of corrected image pairs and a real disparity map of each image pair; The initial recurrent neural network is trained by taking the plurality of corrected image pairs as input respectively, taking the real parallax map of each image pair as output, and taking the absolute value loss as a loss function.

[0043] It should be noted that the training process of the Gaussian point parameter prediction model comprises: A plurality of corrected image pairs, a depth map of each image pair, and a real image of a target viewpoint are obtained. The initial convolutional network is trained by taking the plurality of corrected image pairs and the depth map of each image pair as input, taking the real image of the target viewpoint of each image pair as output, and taking the structural similarity loss and the absolute value loss as a loss function, to obtain the trained Gaussian point parameter prediction model.

[0044] Further, the rendering module 400 is further configured to: The Gaussian point cloud of each second image pair is obtained by back-projection based on each second image pair, the depth map of each second image pair, the Gaussian point prediction parameter map, and the color corresponding to the Gaussian point, respectively.

[0045] In summary, the large-baseline multi-person new viewpoint rendering system based on Gaussian splashing proposed in the embodiment can achieve efficient and fast rendering by combining estimated depth and Gaussian splashing rendering, and has good generalization on various data.

[0046] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0047] Any process or method descriptions in flow charts or described elsewhere herein can be understood as representing one or more steps in a set of steps performed in one or more processes or methods. The scope of preferred embodiments of the present application encompasses numerous additional implementation sequences, examples of which have been provided for the sake of parenthetical description. The number, sequence, and / or organization of steps in any combination of processes or methods should not be limited to that described in this document, but can be understood in the context of the claims.

[0048] Although the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are exemplary and are not to be construed as limiting the present application, and that changes, modifications, substitutions and variations can be made by those skilled in the art without departing from the scope of the present application.

Claims

1. A new multi-viewpoint rendering method based on Gaussian splashing with large baselines, characterized in that, The method includes: Multiple target viewpoints are acquired, and multiple images of multiple people are collected to form multiple first image pairs. Stereo correction is performed on each of the multiple first image pairs to obtain multiple second image pairs. The multiple second image pairs are respectively input into a pre-trained disparity prediction model to obtain a predicted disparity map for each second image pair; The depth map of each second image pair is determined based on the predicted disparity map of each second image pair. The depth map of each second image pair and the second image pair corresponding to each depth map are then input into a pre-trained Gaussian point parameter prediction model to obtain the Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair. Based on each second image pair and its depth map, Gaussian point prediction parameter map, and the color corresponding to the Gaussian point, the Gaussian point cloud of each second image pair is determined, and the Gaussian point cloud of each second image pair is used to render multiple target viewpoints to obtain the rendered image of each first image pair under multiple target viewpoints. The acquired images of multiple people constitute multiple first image pairs, including: Multiple cameras arranged horizontally are used to simultaneously and uniformly acquire images of multiple people on a large baseline. The images of multiple people captured by every two adjacent cameras are grouped together to form the first image pair; Determining the depth map of each of the second image pairs based on the predicted disparity maps of each of the image pairs includes: Obtain each disparity value in the predicted disparity map of each of the second image pairs; The depth value of each second image pair is determined based on the disparity values ​​in the predicted disparity map of each second image pair, the baseline length between the two cameras of each second image pair, the focal length of the reference camera, and the principal point offset value of the two cameras after stereo correction. Depth maps for each of the second image pairs are determined based on the depth values ​​of each second image pair.

2. The method as described in claim 1, characterized in that, The training process of the disparity prediction model includes: Obtain multiple corrected image pairs and the true disparity map of each image pair; The initial recurrent neural network is trained by taking the multiple corrected image pairs as inputs, the true disparity maps of each image pair as outputs, and the absolute value loss as the loss function, to obtain a trained disparity prediction model.

3. The method as described in claim 2, characterized in that, The training process of the Gaussian point parameter prediction model includes: Acquire multiple corrected image pairs, depth maps of each image pair, and the true image of the target viewpoint; The initial convolutional network is trained by taking multiple corrected image pairs and the depth maps of each image pair as inputs, and the real images of the target viewpoints of each image pair as outputs. The structural similarity loss and absolute value loss are weighted as the loss function to obtain the trained Gaussian point parameter prediction model.

4. The method as described in claim 3, characterized in that, The parameter graphs include: Gaussian point rotation parameter graph, Gaussian point size parameter graph, and Gaussian point transparency parameter graph.

5. The method as described in claim 4, characterized in that, The step of determining the Gaussian point cloud of each second image pair based on each second image pair, its depth map, Gaussian point prediction parameter map, and the color corresponding to the Gaussian point includes: Back-projection is performed on each of the second image pairs and their depth maps, Gaussian point prediction parameter maps, and the colors corresponding to the Gaussian points to obtain the Gaussian point clouds of each of the second image pairs.

6. A large-baseline multi-viewpoint rendering system based on Gaussian splashing, characterized in that, The system includes: The acquisition module is used to acquire multiple target viewpoints and collect images of multiple people to form multiple first image pairs. Stereo correction is performed on the multiple image pairs to obtain multiple second image pairs. The disparity determination module is used to input the plurality of second image pairs into a pre-trained disparity prediction model to obtain a predicted disparity map for each second image pair. The parameter determination module is used to determine the depth map of each second image pair based on the predicted disparity map of each second image pair, and input the depth map of each second image pair and the second image pair corresponding to each depth image into the pre-trained Gaussian point parameter prediction model to obtain the Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair. The rendering module is used to determine the Gaussian point cloud of each second image pair based on the depth map, Gaussian point prediction parameter map and the color corresponding to the Gaussian point of each second image pair, and to render the multiple target viewpoints based on the Gaussian point cloud of each second image pair to obtain the rendered image of each first image pair under the multiple target viewpoints. The acquisition module is also used for: Multiple cameras arranged horizontally are used to simultaneously and uniformly acquire images of multiple people on a large baseline. The images of multiple people captured by every two adjacent cameras are grouped together to form the first image pair; The parameter determination module is also used for: Obtain each disparity value in the predicted disparity map of each of the second image pairs; The depth value of each second image pair is determined based on the disparity values ​​in the predicted disparity map of each second image pair, the baseline length between the two cameras of each second image pair, the focal length of the reference camera, and the principal point offset value of the two cameras after stereo correction. Depth maps for each of the second image pairs are determined based on the depth values ​​of each second image pair.

Images