New-view image generation method, and training method and apparatus for new-view generation model

Abstract

A new-view image generation method, and a training method and apparatus for a new-view generation model. The new-view image generation method comprises: acquiring a plurality of first initial images, wherein the plurality of first initial images are collected by means of a plurality of image collection apparatuses, which are distributed in a dispersed manner and image collection areas of which comprise overlapping areas; acquiring human eye coordinate information; inputting the first initial images and the human eye coordinate information into a new-view generation model, so as to obtain a new-view image corresponding to the human eye coordinate information, wherein the new-view generation model comprises a depth calculation module, a projection prediction module, a resize module and a super-resolution module, the depth calculation module is configured to obtain a depth image according to the plurality of first initial images, the projection prediction module is configured to project the depth image to a new-view position, so as to obtain a predicted new-view image, the resize module is configured to resize the predicted new-view image, and the super-resolution module is configured to perform resolution improvement on the resized new-view image, so as to obtain a final new-view image.

Description

New viewpoint image generation method, new viewpoint generation model training method and device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on July 24, 2023, with application number 202310913040.4 and invention name “New viewpoint image generation method, training method and device for new viewpoint generation model”, the content of which should be understood as incorporated into this application by reference. Technical Field

[0002] The embodiments of the present disclosure relate to, but are not limited to, the field of image processing technology, and in particular to a method for generating a new viewpoint image, and a method and device for training a new viewpoint generation model. Background Art

[0003] 3D displays have long been considered the ultimate dream in display technology development. After years of development, two major 3D display technology systems currently exist on the market: glasses-based and autostereoscopic. Autostereoscopic 3D display, also known as autostereoscopic, is the future trend in 3D display technology.

[0004] Summary of the Invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The present disclosure provides a method for generating a new viewpoint image, including:

[0007] Acquiring a plurality of first initial images, wherein the plurality of first initial images are simultaneously acquired by a plurality of dispersed image acquisition devices, wherein image acquisition areas of the plurality of image acquisition devices include overlapping areas;

[0008] Obtain the eye coordinate information of the user on the display end;

[0009] The first initial image and the human eye coordinate information are input into a new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information. The new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate depth information based on multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

[0010] An embodiment of the present disclosure also provides a new viewpoint generation device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the new viewpoint image generation method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0011] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the new viewpoint image generation method as described in any embodiment of the present disclosure.

[0012] The present disclosure also provides a method for training a new viewpoint generation model, including:

[0013] Constructing a three-dimensional reconstruction model and a first training data set;

[0014] Using the first training data set to train the three-dimensional reconstruction model to obtain the trained three-dimensional reconstruction model;

[0015] Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model;

[0016] Taking two-dimensional images of the three-dimensional model at multiple different angles using a virtual camera, and constructing a second training data set using the two-dimensional images;

[0017] The new viewpoint generation model is trained using the second training dataset.

[0018] An embodiment of the present disclosure also provides a training device for a new viewpoint generation model, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the training method for the new viewpoint generation model as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0019] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a new viewpoint generation model as described in any embodiment of the present disclosure.

[0020] Still other aspects will become apparent upon reading and understanding the accompanying drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are intended to provide a further understanding of the technical solutions of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solutions of the present disclosure and do not constitute a limitation of the technical solutions of the present disclosure. The shapes and sizes of one or more components in the accompanying drawings do not reflect the actual scale and are intended only to illustrate the contents of the present disclosure.

[0022] FIG1 is a schematic flow chart of a method for generating a new viewpoint image according to an exemplary embodiment of the present disclosure;

[0023] FIG2 is a schematic diagram of an application scenario of a new viewpoint image generation method according to an exemplary embodiment of the present disclosure;

[0024] FIG3 is a schematic diagram of the data transmission process of the application scenario shown in FIG2 ;

[0025] FIG4 is a schematic diagram of a neural network modeling process according to an exemplary embodiment of the present disclosure;

[0026] FIG5 is a schematic diagram of a three-dimensional reconstruction process of a three-dimensional reconstruction model according to an exemplary embodiment of the present disclosure;

[0027] FIG6 is a schematic diagram of a new viewpoint image generation process of a new viewpoint generation model according to an exemplary embodiment of the present disclosure;

[0028] FIG7 is a flow chart of a method for training a new viewpoint generation model according to an exemplary embodiment of the present disclosure;

[0029] FIG8 is a schematic structural diagram of a new viewpoint image generation device according to an exemplary embodiment of the present disclosure;

[0030] FIG9 is a schematic structural diagram of a training device for a new viewpoint generation model according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. The embodiments can be implemented in a variety of different forms. A person skilled in the art can easily understand that the method and content can be transformed into other forms without departing from the purpose and scope of the present disclosure. Therefore, the present disclosure should not be interpreted as being limited to the contents described in the following embodiments. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other in any manner.

[0032] In the drawings, the size of one or more components, layer thicknesses, or regions may be exaggerated for clarity. Therefore, one embodiment of the present disclosure is not necessarily limited to these dimensions, and the shapes and sizes of one or more components in the drawings do not reflect true to scale. Furthermore, the drawings schematically illustrate idealized examples, and one embodiment of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0033] In this specification, ordinal numbers such as "first," "second," and "third" are provided to avoid confusion among constituent elements, and are not intended to limit the number. "Multiple" in this disclosure means two or more.

[0034] Holographic video conferencing is a relatively important application scenario in 3D display and is relatively easy to implement. In holographic video conferencing, generating new viewpoint images has become a key research problem. This problem aims to generate an image of the scene from a specific viewpoint given one or more images of the same scene.

[0035] As shown in FIG1 , an embodiment of the present disclosure provides a method for generating a new viewpoint image, including:

[0036] Step 101: Acquire multiple first initial images, where the multiple first initial images are simultaneously acquired by multiple dispersed image acquisition devices, and image acquisition areas of the multiple image acquisition devices include overlapping areas;

[0037] Step 102: Obtain eye coordinate information of the user at the display end;

[0038] Step 103: Input the multiple first initial images and the human eye coordinate information into a new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information. The new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate the depth information based on the multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to the new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

[0039] The disclosed embodiment establishes a new viewpoint generation model, inputs multiple first initial images (these multiple first initial images constitute a frame of image) and human eye coordinate information into the new viewpoint generation model, and obtains a new viewpoint image corresponding to the human eye coordinate information. With the help of the general parallel computing capabilities of a high-performance graphics processing unit (GPU) and the representation capabilities of a neural network, an excellent new viewpoint image generation effect can be achieved.

[0040] In some exemplary embodiments, the image acquisition device is located at the acquisition end, and the acquisition end is connected to the display end via a network.

[0041] Figure 2 is a schematic diagram of an application scenario for the new viewpoint image generation method of an embodiment of the present disclosure. As shown in Figure 2, the system is an end-to-end single-person chat system. The user sits in front of a display. The system at each end runs a new viewpoint generation model. One end is the acquisition end (and the display end), and the other end is the display end (and the acquisition end). The image acquisition device at the acquisition end captures the initial image and sends it to the display end via the network. The display end system runs the new viewpoint generation model to generate a new viewpoint image corresponding to the eye coordinate information of the user at the display end.

[0042] In some exemplary embodiments, the image acquisition device may be a camera or any other image acquisition device.

[0043] Figure 3 is a schematic diagram of the data transmission process for the application scenario shown in Figure 2. As shown in Figure 3, after the acquisition-end camera captures the real-scene image, it is first transmitted to the acquisition-end central processing unit (CPU). The acquisition-end CPU transmits the data to the acquisition-end GPU via the bus system. The acquisition-end GPU performs image preprocessing on the raw captured data. Image preprocessing can include image signal processing (ISP), color correction, and interpolation into RGB data. The RGB data is then directly encoded using the GPU hardware encoding chip and uploaded to the network. The display-end processor pulls the video stream from the network and decodes it using the GPU hardware decoding chip to obtain the source data captured by the acquisition-end camera. This source data serves as input for the human eye positioning algorithm and viewpoint generation algorithm.

[0044] In some exemplary embodiments, there may be four image acquisition devices, which are respectively arranged in the left area, right area, upper area and lower area of ​​the acquisition end display, and the image acquisition areas of the four image acquisition devices are all facing the front area of ​​the acquisition end display.

[0045] In the disclosed embodiments, the number of image acquisition devices is not fixed; it can be four, six, or even more. Theoretically, a greater number of image acquisition devices results in better results, but this also increases the data generated and the cost, hindering productization and promotion. Therefore, the new viewpoint generation method in the disclosed embodiments utilizes a sparse number of image acquisition devices. The image acquisition devices are not fixed in position; they can be placed around the acquisition-end display as much as possible, with their lenses facing the center of the acquisition-end display. Each image acquisition device is designed to capture the user sitting in front of the acquisition-end display.

[0046] The application of holographic video communication technology in video conferencing will become more and more popular, and the present disclosure mainly optimizes the technology for this application scenario. This system comes with multiple cameras and uses neural network modeling technology to generate the multi-viewpoint images required for light field displays. From a technical point of view, if you want to obtain image data of people at any angle in front of the display by only inputting images taken by a few cameras, the neural network must have seen enough image data from any angle before. Therefore, a mature neural network system must first undergo a large amount of data training to learn before it is possible to obtain the optimal network weight parameters. When we train the new viewpoint generation model, we should try to let the new viewpoint generation model see rich image data from any angle on the front of the display. In this way, when we only input a sparse number of camera images, the new viewpoint generation model can also output images from any angle.

[0047] However, our system only has a sparse number of cameras, and it is impossible to collect enough rich image data from any perspective. Therefore, as shown in Figure 4, the present disclosure constructs two neural network models: a three-dimensional reconstruction model and a new viewpoint generation model. The first neural network model (i.e., the three-dimensional reconstruction model) generates a three-dimensional model based on a frame of input image data (each frame of image data includes images collected simultaneously by multiple image acquisition devices). Based on the three-dimensional model, rich image data from any perspective can be generated, and then the rich image data is used for learning and training the second neural network model (i.e., the new viewpoint generation model). In this way, enough training data can be provided for the new viewpoint generation model. This is a method of exchanging time for space, so that the second neural network model can meet the real-time characteristics of the video chat scenario as much as possible. The second neural network model is the network model when the system is running in real time. In Figure 4, the training networks of the 3D reconstruction model and the new viewpoint generation model include loss calculation and back-propagation modules, while the inference networks of the 3D reconstruction model and the new viewpoint generation model do not include loss calculation and back-propagation modules. In addition, knowledge distillation is performed through the teacher model and the student model in the training network of the new viewpoint generation model, while only the corresponding student model is included in the inference network of the new viewpoint generation model, thereby achieving model acceleration while ensuring model effectiveness.

[0048] Theoretically, only a 3D reconstruction model can generate image data from any perspective. However, the method of generating viewpoint maps based on the 3D reconstruction model is time-consuming and difficult to meet the real-time requirements of the system. The 3D reconstruction model is the training network of the new viewpoint generation model. As shown in Figure 4, for our system, the first three stages are all preparation stages and serve to serve the final stage. The new viewpoint generation model inference network in the final stage is the algorithm module that runs in real time when the system is in use. This algorithm module only needs to input images captured by a sparse number of cameras to output images of the front of the display from any perspective. That is, when the system is in use, only the new viewpoint generation model is deployed on the processors (computers) at both ends of Figure 2. By inputting images captured by a sparse number of cameras on the acquisition end into the new viewpoint generation model, images of the front of the display from any perspective can be output.

[0049] In some exemplary embodiments, the new viewpoint generation model is trained by the following method:

[0050] Constructing a three-dimensional reconstruction model and a first training data set;

[0051] Using the first training data set to train the three-dimensional reconstruction model to obtain a trained three-dimensional reconstruction model;

[0052] Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into a trained three-dimensional reconstruction model to obtain a three-dimensional model;

[0053] Using a virtual camera to capture two-dimensional images of the three-dimensional model at multiple different angles, and using the two-dimensional images to construct a second training data set;

[0054] The new viewpoint generation model is trained using the second training dataset.

[0055] Holographic video conferencing applications theoretically require a large number of cameras to capture real-world images, but having too many cameras significantly impacts both cost and power consumption. The new viewpoint image generation method in the disclosed embodiments utilizes two neural network algorithms: a 3D reconstruction model and a new viewpoint generation model. The 3D reconstruction model trades time for space, generating a sufficient number of images from any perspective for training the new viewpoint generation model, which is then used for real-time system operation. This approach meets the system's requirements for both sparse camera capabilities and real-time performance.

[0056] In some exemplary embodiments, the first training data set includes a system-acquired data set and a virtual data set. The system-acquired data set includes images acquired by multiple image acquisition devices, and the virtual data set includes images of different people performing different actions downloaded from the Internet.

[0057] The modeling process of the 3D reconstruction model includes:

[0058] 1) Constructing a 3D reconstruction model (training network) as shown in FIG5 ;

[0059] 2) Dataset Preparation: Construct a virtual dataset, which includes professional image data downloaded online containing people in various clothing and performing various actions; construct a system acquisition dataset, which includes data of multiple different people captured using our system; merge the two datasets together to form the first training dataset;

[0060] 3) Training process: The first training dataset is divided into a first training set and a first test set. The first training set is used to train the 3D reconstruction model. During the training process, the backpropagation algorithm is used to calculate the model parameters and adjust the model weights and biases to achieve optimal model performance.

[0061] 4) Use the first test set to verify the trained model to see if the dataset has generalization capabilities. If the first test set input model works well, it means the model training is successful. If not, adjust the training strategy based on experience.

[0062] 5) The trained 3D reconstruction model's network parameters are fed into the 3D reconstruction model inference network shown in Figure 4 (excluding the loss calculation and backpropagation modules in Figure 5). This network takes a sparse number of camera images as input, performs 3D modeling based on the input camera images, and outputs a 3D model that includes depth and color. A virtual camera is placed in the space where the 3D model resides and photographs the 3D model from various viewpoints, generating many 2D images from different perspectives. These 2D images can then be used to train the new viewpoint generation model.

[0063] In some exemplary embodiments, the 3D reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module, and a weighted summation module, wherein:

[0064] The feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract the feature map of the input image, and project the N1*N2 sampling points onto the feature map according to the shooting parameters to obtain the color feature of each sampling point, where N1 is greater than 1 and N2 is greater than 1;

[0065] A directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, where the target geometric model includes a distance field value and a second feature value of each sampling point;

[0066] A color extraction module is configured to obtain a color value of each sampling point according to the second eigenvalue of each sampling point and the light direction;

[0067] A weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point;

[0068] The weighted summation module is configured to perform weighted summation based on the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light ray on the image.

[0069] For example, N1 may be equal to 1024*1024, and N2 may be equal to 128, however, the embodiments of the present disclosure are not limited thereto.

[0070] In some exemplary embodiments, the weight of the sampling point is exponentially related to the distance field value of the sampling point.

[0071] In some exemplary embodiments, when training a three-dimensional reconstruction model using the first training data set, a loss function is established, where the loss function includes the difference between the color value of each pixel initially sampled and the color value obtained by weighted summation of each pixel.

[0072] Figure 5 is a schematic diagram of the 3D reconstruction process of the 3D reconstruction model. The 3D reconstruction model uses continuous color images under limited and sparse viewpoints (for example, 4 viewpoints) to restore the 3D model and color information of the human body, providing high-quality and sufficient training data for the new viewpoint generation model.

[0073] In the embodiment of the present disclosure, the 3D reconstruction model includes the following modules:

[0074] Feature extraction module (not shown in the figure): mainly configured to extract feature maps of the input image;

[0075] Directed Distance Field Network (SDFNetwork): This module covers the core concepts of four-dimensional representation of continuous video frames, projective decomposition of feature vectors, and guided convergence of MVS. Its main function is to predict the geometric information of the human body model.

[0076] Color extraction module (ColorNetwork): mainly configured to predict the color information (color and transparency, neural radiation field) of the human body model photographed at a specified position;

[0077] Weight calculation module (Single Variance Network): configured to establish a connection between the neural radiation field and the SDF field in order to use image information as supervision of the geometric model;

[0078] The weighted summation module (Σ in FIG5 ) is configured to perform weighted summation based on the color value of each sampling point and the weight of each sampling point to obtain the color value of each pixel point.

[0079] For each input image, the valid point range in the image is obtained based on its mask (commonly used for cropping images to obtain the outline of a person). Each pixel in the image corresponds to a ray in space originating from the camera's optical center. During training, N1 rays are randomly selected, and N2 sampling points are sampled on each ray, resulting in N1*N2 sampling points (pts) in space. The four input images are convolved using a feature extraction module (not shown in the figure, which can be a convolutional network) to extract four feature maps. Using the camera's intrinsic and extrinsic parameters, the N1*N2 sampling points are projected onto these four feature maps to obtain the color features Projs for each sampling point. The N1*N2 sampling points pts, the color features Projs for each sampling point, and the video frame number Time in which this set of images occurs, together constitute the input of SDFNetWork. SDFNetWork can be composed of a series of convolutional network structures, the purpose of which is to obtain the geometric model of the target (person) (represented by the distance field sdf). The output of SDFNetWork contains two parts: one is the distance field value corresponding to each sampling point, and the other is the second eigenvalue features (the second eigenvalue contains color information). The second eigenvalue of each sampling point plus the light direction (view angle) corresponding to this point at that time dirs), which together constitute the input of ColorNetwork. The function of ColorNetwork is to obtain the color in the neural radiation field, that is, the color value corresponding to this sampling point. SingleVarianceNetwork is a single-parameter network. We believe that there is an exponential relationship between the distance field value corresponding to each sampling point pt and the transparency or weight in its neural radiation field. This single-parameter network is to obtain the exponential parameter sigma_f in this exponential relationship. That is to say, the distance field value can obtain the transparency or weight corresponding to this sampling point through the exponential relationship of sigma_f. The weighted sum of the color and transparency of all sampling points on N rays can obtain the color value of the pixel corresponding to this ray on the image. The difference between this color value and the color value corresponding to the pixel position when we first sampled (this difference is not necessarily the absolute value of the simple difference) is the loss (Loss). Using this Loss to supervise the back propagation of each module is a training process.

[0080] In some exemplary embodiments, the second training data set includes a first three-dimensional model data set and a second portrait model data set, the first three-dimensional model data set includes two-dimensional images of the three-dimensional model taken at multiple different angles by a virtual camera, and the second portrait model data set includes two-dimensional images of the three-dimensional portrait model downloaded from the Internet taken at multiple different angles by a virtual camera.

[0081] The modeling process of the new viewpoint generation model includes:

[0082] 1) Construct a new viewpoint generation model as shown in Figure 6;

[0083] 2) Using a large number of multi-view images generated by the 3D reconstruction model, a first 3D model dataset is constructed. Using portrait models downloaded from the Internet, a second portrait model dataset is constructed. The two datasets are merged to construct a second training dataset that is ultimately used to train the new viewpoint generation model.

[0084] 3) Training process: The second training dataset is divided into a second training set and a second test set, and the new viewpoint generation model is trained using the second training set. During the training process, the backpropagation algorithm is used to calculate the model parameters and adjust the model weights and biases to achieve optimal model performance;

[0085] 4) Use the second test set to verify the trained model to see if the dataset has generalization capabilities. If the second test set input model works well, it means the model training is successful. If not, adjust the training strategy based on experience.

[0086] 5) Use the trained network model parameters to input the new viewpoint generation model inference network in Figure 4. This network is used to input images collected by a sparse number of cameras and can render and output 2D images of any perspective.

[0087] Since the final output image is displayed on a 3D display, the specific field of view angles for rendering images can be combined with human eye positioning. The eye coordinates calculated by the eye positioning algorithm can be transmitted to the new viewpoint generation model inference network, and the viewpoint images corresponding to the left and right eyes can be rendered in real time (combined with the display principle of the 3D display).

[0088] Figure 6 is a schematic diagram of the new viewpoint image generation process of the new viewpoint generation model. As shown in Figure 6, the left, right and upper three viewpoint color images are input, the disparity map of the upper left two perspective images and the upper right two perspective images are calculated to obtain the depth map of the left and right two perspectives, and the color image of the new viewpoint (512*512) is predicted. Then, the super-resolution module is used to output 1K, 2K and other resolution high-definition images.

[0089] In the embodiment of the present disclosure, the new viewpoint generation model includes the following modules:

[0090] Depth Calculation Module (MVS): This module can be a non-training network module that uses the open source RMVS algorithm and weight files. Its main function is to calculate depth information based on multi-view color images;

[0091] Projection prediction module (ColorNet): Based on the left and right viewpoint color images, the depth map is projected to the new viewpoint position to predict the accurate new viewpoint image (512*512);

[0092] Stretching module (Resize): configured to stretch the predicted new viewpoint image;

[0093] Super-resolution module (ColorHrNet): This module is configured to increase the resolution of the stretched new viewpoint image to obtain the final new viewpoint image. The super-resolution module can super-resolve the 512*512 new viewpoint image to a size of 1K*1K or larger.

[0094] As shown in Figure 6, the images taken by the left camera and the upper camera are input, and the depth image of the left camera position is obtained through MVS. Similarly, the images taken by the right camera and the upper camera are input, and the depth image of the right camera position is obtained through MVS. Based on the color images taken by the left and right camera positions and the calculated depth image, the ColorNet network is used to project (Projection) to the new viewpoint position to predict an accurate new viewpoint image (size: 512*512). The new viewpoint image obtained above is stretched to 1024*1024 size, and then passed through the super-resolution module to improve the clarity of the output new viewpoint image.

[0095] As shown in Figure 6, when training the new viewpoint generation model, it also includes a diffusion model. ColorHrNet is the student model, and the Diffusion Model is the teacher model. Together, they form the super-resolution module. The Diffusion Model is configured to generate data similar to the training data. Fundamentally, the Diffusion Model works by continuously adding Gaussian noise to corrupt the training data, then recovering the data by learning the inverse denoising process. After training, the Diffusion Model can be used to pass randomly sampled noise into the model, and generate data using the learned denoising process. The Diffusion Model is a large and complex model, while the ColorHrNet is a small and simple model. During training, the Diffusion Model supervises the ColorHrNet training, but only the ColorHrNet is used during operation. Because the ColorHrNet is a simple model and the network model that actually runs the entire algorithm, this achieves model acceleration while maintaining model performance.

[0096] In some exemplary embodiments, when the new viewpoint generation model is trained using the second training dataset, the new viewpoint generation model further includes a diffusion model, wherein the diffusion model is a teacher model and the super-resolution module is a student model.

[0097] In some exemplary embodiments, the depth calculation module is specifically configured to:

[0098] Depth information is calculated based on the first initial image captured by each two image capture devices, wherein each two image capture devices include two image capture devices arranged in the left area and the upper area of ​​the first display, and two image capture devices arranged in the right area and the upper area of ​​the first display.

[0099] In the embodiment of the present disclosure, the depth calculation module can also calculate the depth information based on the image acquisition devices at other locations, and the embodiment of the present disclosure does not limit this.

[0100] As shown in FIG7 , the embodiment of the present disclosure further provides a training method for a new viewpoint generation model, including:

[0101] Step 701: construct a three-dimensional reconstruction model and a first training data set;

[0102] Step 702: Use the first training data set to train the 3D reconstruction model to obtain the trained 3D reconstruction model;

[0103] Step 703: Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into the trained 3D reconstruction model to obtain a 3D model;

[0104] Step 704: Use a virtual camera to capture two-dimensional images of the three-dimensional model at multiple different angles, and use the two-dimensional images to construct a second training data set;

[0105] Step 705: Use the second training dataset to train the new viewpoint generation model.

[0106] In some exemplary embodiments, the first training data set includes a system-acquired data set and a virtual data set, the system-acquired data set includes images acquired by the multiple image acquisition devices, and the virtual data set includes images of different people performing different actions downloaded from the Internet.

[0107] In some exemplary embodiments, the second training data set includes a first three-dimensional model data set and a second portrait model data set, the first three-dimensional model data set including two-dimensional images of the three-dimensional model taken at multiple different angles by a virtual camera, and the second portrait model data set including two-dimensional images of a three-dimensional portrait model downloaded from the Internet taken at multiple different angles by a virtual camera.

[0108] In some exemplary embodiments, the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module, and a super-resolution module, wherein:

[0109] The depth calculation module is configured to calculate depth information based on the multiple first initial images to obtain a depth image;

[0110] The projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image;

[0111] The stretching module is configured to stretch the predicted new viewpoint image;

[0112] The super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

[0113] In some exemplary embodiments, the image acquisition device is located at an acquisition end, and the acquisition end is connected to the display end via a network.

[0114] In some exemplary embodiments, the image acquisition devices include four, and the four image acquisition devices are respectively arranged in the left area, right area, upper area and lower area of ​​the first display, and the image acquisition areas of the four image acquisition devices face the front area of ​​the first display.

[0115] In some exemplary embodiments, the depth calculation module is specifically configured to:

[0116] Depth information is calculated based on the first initial image captured by each two image capture devices, wherein each two image capture devices include two image capture devices arranged in the left area and upper area of ​​the first display, and two image capture devices arranged in the right area and upper area of ​​the first display.

[0117] In some exemplary embodiments, when the new viewpoint generation model is trained using the second training dataset, the new viewpoint generation model further includes a diffusion model, the diffusion model is a teacher model, and the super-resolution module is a student model.

[0118] In some exemplary embodiments, the three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module, and a weighted summation module, wherein:

[0119] The feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract a feature map of the input image, and project N1*N2 sampling points onto the feature map according to the shooting parameters to obtain a color feature of each sampling point, where N1 is greater than 1 and N2 is greater than 1;

[0120] The directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, wherein the target geometric model includes the distance field value and the second feature value of each sampling point;

[0121] The color extraction module is configured to obtain the color value of each sampling point according to the second eigenvalue of each sampling point and the light direction;

[0122] The weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point;

[0123] The weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light ray on the image.

[0124] In some exemplary embodiments, the weight of a sampling point is exponentially related to the distance field value of the sampling point.

[0125] In some exemplary embodiments, when the first training data set is used to train the three-dimensional reconstruction model, a loss function is established, where the loss function includes the difference between the color value of each pixel initially sampled and the color value obtained by weighted summation of each pixel.

[0126] An embodiment of the present disclosure also provides a new viewpoint generation device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the new viewpoint image generation method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0127] As shown in Figure 8, in one example, the new viewpoint generation device may include: a first processor 810, a first memory 820, a first bus system 830 and a first transceiver 840, wherein the first processor 810, the first memory 820 and the first transceiver 840 are connected through the first bus system 830, the first memory 820 is used to store instructions, and the first processor 810 is used to execute the instructions stored in the first memory 820 to control the first transceiver 840 to send and receive signals. Specifically, the first transceiver 840 can obtain multiple first initial images under the control of the first processor 810, and the multiple first initial images are simultaneously acquired by multiple dispersed image acquisition devices, and the image acquisition areas of the multiple image acquisition devices include overlapping areas. The first processor 810 obtains the human eye coordinate information of the display end user; the first initial image and the human eye coordinate information are input into the new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information, and the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate depth information based on the multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to the new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image; the final new viewpoint image is output to the display interface of the display end through the first transceiver 840.

[0128] It should be understood that the first processor 810 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0129] The first memory 820 may include a read-only memory and a random access memory, and provides instructions and data to the first processor 810. A portion of the first memory 820 may also include a non-volatile random access memory. For example, the first memory 820 may also store device type information.

[0130] In addition to the data bus, the first bus system 830 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the first bus system 830 in FIG.

[0131] During implementation, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the first processor 810 or by instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the first memory 820, and the first processor 810 reads the information in the first memory 820 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0132] The present disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the new viewpoint image generation method described in any of the embodiments of the present disclosure. The method for driving new viewpoint image generation by executing executable instructions is substantially the same as the new viewpoint image generation method described in the aforementioned embodiments of the present disclosure and is not further described here.

[0133] In some possible implementations, various aspects of the new viewpoint image generation method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the new viewpoint image generation method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the new viewpoint image generation method recorded in the embodiments of the present application.

[0134] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0135] An embodiment of the present disclosure also provides a training device for a new viewpoint generation model, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the training method for the new viewpoint generation model as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0136] As shown in FIG9 , in one example, a training apparatus for a new viewpoint generation model may include: a second processor 910, a second memory 920, a second bus system 930, and a second transceiver 940. The second processor 910, the second memory 920, and the second transceiver 940 are connected via the second bus system 930. The second memory 920 is used to store instructions, and the second processor 910 is used to execute the instructions stored in the second memory 920 to control the second transceiver 940 to transmit and receive signals. Specifically, under the control of the second processor 910, the second transceiver 940 may acquire second initial images captured by multiple image acquisition devices. The second processor 910 constructs a three-dimensional reconstruction model and a first training dataset; trains the three-dimensional reconstruction model using the first training dataset to obtain the trained three-dimensional reconstruction model; inputs the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model; captures two-dimensional images of the three-dimensional model at multiple different angles using a virtual camera, and constructs a second training dataset using the two-dimensional images; and trains the new viewpoint generation model using the second training dataset.

[0137] It should be understood that the second processor 910 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0138] The second memory 920 may include a read-only memory and a random access memory, and provide instructions and data to the second processor 910. A portion of the second memory 920 may also include a non-volatile random access memory. For example, the second memory 920 may also store device type information.

[0139] In addition to the data bus, the second bus system 930 may also include a power bus, a control bus, a status signal bus, etc. However, for the sake of clarity, various buses are labeled as the second bus system 930 in FIG.

[0140] During the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the second processor 910 or by instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the second memory 920, and the second processor 910 reads the information in the second memory 920 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0141] An embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a new viewpoint generation model as described in any embodiment of the present disclosure.

[0142] In some possible implementations, various aspects of the training method for a new viewpoint generation model provided in the present application may also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the training method for a new viewpoint generation model according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the training method for a new viewpoint generation model described in the embodiments of the present application.

[0143] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0144] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0145] It should be noted that the above-described embodiments or implementations are merely illustrative and not restrictive. Therefore, the present disclosure is not limited to what is specifically shown and described herein. Various modifications, substitutions, or omissions may be made to the forms and details of the implementations without departing from the scope of the present disclosure.

[0146] The drawings of the embodiments of the present disclosure only involve the structures involved in the embodiments of the present disclosure, and other structures may refer to general designs.

[0147] For the sake of clarity, the thickness and size of layers or microstructures are exaggerated in the drawings used to describe the embodiments of the present disclosure. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element can be "directly on" or "under" the other element, or intervening elements may be present.

Claims

1. A method for generating a new viewpoint image, comprising: Acquire a plurality of first initial images, wherein the plurality of first initial images are simultaneously acquired by a plurality of dispersed image acquisition devices, and image acquisition areas of the plurality of image acquisition devices include overlapping areas; Obtain eye coordinate information of the user at the display end; The first initial image and the human eye coordinate information are input into a new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information, wherein the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate depth information according to multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

2. The method for generating a new viewpoint image according to claim 1, wherein: The new viewpoint generation model is trained by the following method: Constructing a three-dimensional reconstruction model and a first training data set; Using the first training data set to train the three-dimensional reconstruction model to obtain the trained three-dimensional reconstruction model; Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model; Taking two-dimensional images of the three-dimensional model at multiple different angles by a virtual camera, and using the two-dimensional images to construct a second training data set; The new viewpoint generation model is trained using the second training dataset.

3. The method for generating a new viewpoint image according to claim 2, wherein: The three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module and a weighted summation module, wherein: The feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract a feature map of the input image, and project N1*N2 sampling points onto the feature map according to the shooting parameters to obtain a color feature of each sampling point, where N1 is greater than 1 and N2 is greater than 1; The directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point. The target geometric model includes a distance field value and a second eigenvalue of each sampling point; The color extraction module is configured to obtain the color value of each sampling point according to the second eigenvalue of each sampling point and the light direction; The weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point; The weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light ray on the image.

4. The method for generating a new viewpoint image according to claim 3, wherein: The weight of the sampling point is in exponential correspondence with the distance field value of the sampling point.

5. The method for generating a new viewpoint image according to claim 2, wherein: When the first training data set is used to train the three-dimensional reconstruction model, a loss function is established, where the loss function includes the difference between the color value initially sampled from each pixel and the color value obtained by weighted summation of each pixel.

6. The method for generating a new viewpoint image according to claim 2, wherein: When the new viewpoint generation model is trained using the second training data set, the new viewpoint generation model also includes a diffusion model, the diffusion model is a teacher model, and the super-resolution module is a student model.

7. The method for generating a new viewpoint image according to claim 2, wherein: The first training data set includes a system acquisition data set and a virtual data set. The system acquisition data set includes images acquired by the multiple image acquisition devices, and the virtual data set includes images of different people performing different actions downloaded from the Internet.

8. The method for generating a new viewpoint image according to claim 2, wherein: The second training data set includes a first three-dimensional model data set and a second portrait model data set, the first three-dimensional model data set includes two-dimensional images of the three-dimensional model taken at multiple different angles by a virtual camera, and the second portrait model data set includes two-dimensional images of a three-dimensional portrait model downloaded from the Internet taken at multiple different angles by a virtual camera.

9. The method for generating a new viewpoint image according to claim 1, wherein: The image acquisition device is located at the acquisition end, and the acquisition end is connected to the display end through a network.

10. The method for generating a new viewpoint image according to claim 1, wherein: The image acquisition devices include four, and the four image acquisition devices are respectively arranged in the left area, right area, upper area and lower area of ​​the first display. The image acquisition areas of the four image acquisition devices are all facing the front area of ​​the first display, and the first display is located at the acquisition end.

11. The method for generating a new viewpoint image according to claim 10, wherein: The depth calculation module is specifically configured as follows: Depth information is calculated based on the first initial images captured by every two of the image capture devices, and every two of the image capture devices include two of the image capture devices arranged in the left area and the upper area of ​​the first display, and two of the image capture devices arranged in the right area and the upper area of ​​the first display.

12. A new viewpoint generation device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the new viewpoint image generation method as described in any one of claims 1 to 11 based on the instructions stored in the memory.

13. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for generating a new viewpoint image according to any one of claims 1 to 11 is implemented.

14. A training method for a new viewpoint generation model, comprising: Constructing a three-dimensional reconstruction model and a first training data set; Using the first training data set to train the three-dimensional reconstruction model to obtain the trained three-dimensional reconstruction model; Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model; Taking two-dimensional images of the three-dimensional model at multiple different angles by a virtual camera, and using the two-dimensional images to construct a second training data set; The new viewpoint generation model is trained using the second training dataset.

15. The training method for generating a new viewpoint model according to claim 14, wherein: The new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein: The depth calculation module is configured to calculate depth information according to the plurality of first initial images to obtain a depth image; The projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image; The stretching module is configured to stretch the predicted new viewpoint image; The super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

16. The training method for generating a new viewpoint model according to claim 15, wherein: When the new viewpoint generation model is trained using the second training data set, the new viewpoint generation model also includes a diffusion model, the diffusion model is a teacher model, and the super-resolution module is a student model.

17. The training method for generating a new viewpoint model according to claim 14, wherein: The three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module and a weighted summation module, wherein: The feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract a feature map of the input image, and project N1*N2 sampling points onto the feature map according to the shooting parameters to obtain a color feature of each sampling point, where N1 is greater than 1 and N2 is greater than 1; The directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, wherein the target geometric model includes a distance field value and a second feature value of each sampling point; The color extraction module is configured to obtain the color value of each sampling point according to the second eigenvalue of each sampling point and the light direction; The weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point; The weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light ray on the image.

18. The training method for generating a new viewpoint model according to claim 17, wherein: The weight of the sampling point is in exponential correspondence with the distance field value of the sampling point.

19. A training device for a new viewpoint generation model, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the training method for the new viewpoint generation model as described in any one of claims 14 to 18 based on the instructions stored in the memory.

20. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a new viewpoint generation model as claimed in any one of claims 14 to 18.

Images