New viewpoint image generation method, new viewpoint generation model training method and device

Abstract

A new viewpoint image generation method, a training method and device for a new viewpoint generation model, the new viewpoint image generation method comprising: obtaining a plurality of first initial images, the plurality of first initial images being acquired by a plurality of dispersed image acquisition devices whose image acquisition areas include overlapping areas; acquiring human eye coordinate information; inputting the first initial images and the human eye coordinate information into a new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information, the new viewpoint generation model comprising a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, the depth calculation module being configured to obtain a depth image according to the plurality of first initial images, the projection prediction module being configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image, the stretching module being configured to stretch the predicted new viewpoint image, and the super-resolution module being configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to, but are not limited to, the field of image processing technology, and in particular to a new viewpoint image generation method, a new viewpoint generation model training method and device. Background Art

[0002] 3D display has been recognized as the ultimate dream of display technology development. After years of technological development, there are two major stereoscopic display technology systems on the market: glasses-type and naked-eye-type. Among them, naked-eye 3D display technology is also called automatic stereoscopic display technology, which is the future development trend of 3D display technology. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The present disclosure provides a method for generating a new viewpoint image, including:

[0005] Acquire a plurality of first initial images, wherein the plurality of first initial images are simultaneously acquired by a plurality of dispersed image acquisition devices, and image acquisition areas of the plurality of image acquisition devices include overlapping areas;

[0006] Obtain eye coordinate information of the user at the display end;

[0007] The first initial image and the human eye coordinate information are input into a new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information, wherein the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate depth information according to multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

[0008] An embodiment of the present disclosure also provides a new viewpoint generation device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the new viewpoint image generation method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0009] The embodiments of the present disclosure further provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method for generating a new viewpoint image as described in any embodiment of the present disclosure is implemented.

[0010] The present disclosure also provides a method for training a new viewpoint generation model, including:

[0011] Constructing a three-dimensional reconstruction model and a first training data set;

[0012] Using the first training data set to train the three-dimensional reconstruction model to obtain the trained three-dimensional reconstruction model;

[0013] Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model;

[0014] Taking two-dimensional images of the three-dimensional model at multiple different angles by a virtual camera, and using the two-dimensional images to construct a second training data set;

[0015] The new viewpoint generation model is trained using the second training dataset.

[0016] An embodiment of the present disclosure also provides a training device for a new viewpoint generation model, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the training method for the new viewpoint generation model as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0017] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a new viewpoint generation model as described in any embodiment of the present disclosure.

[0018] Other aspects will be apparent upon reading and understanding the drawings and detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure. The shape and size of one or more components in the accompanying drawings do not reflect the actual proportions and are only intended to illustrate the contents of the present disclosure.

[0020] Figure 1 A schematic diagram of a flow chart of a method for generating a new viewpoint image according to an exemplary embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of an application scenario of a new viewpoint image generation method according to an exemplary embodiment of the present disclosure;

[0022] Figure 3 for Figure 2 Schematic diagram of data transmission process in the application scenario shown;

[0023] Figure 4 A schematic diagram of a neural network modeling process of an exemplary embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a three-dimensional reconstruction process of a three-dimensional reconstruction model according to an exemplary embodiment of the present disclosure;

[0025] Figure 6 A schematic diagram of a new viewpoint image generation process of a new viewpoint generation model according to an exemplary embodiment of the present disclosure;

[0026] Figure 7 A flowchart of a method for training a new viewpoint generation model according to an exemplary embodiment of the present disclosure;

[0027] Figure 8 A schematic structural diagram of a new viewpoint image generating device according to an exemplary embodiment of the present disclosure;

[0028] Fig. 9 The present invention is a schematic diagram of the structure of a training device for a new viewpoint generation model according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. The embodiments can be implemented in a plurality of different forms. A person skilled in the art can easily understand the fact that the method and content can be transformed into other forms without departing from the purpose and scope of the present disclosure. Therefore, the present disclosure should not be interpreted as being limited to the contents described in the following embodiments. In the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.

[0030] In the drawings, the size of one or more components, the thickness of a layer, or an area is sometimes exaggerated for the sake of clarity. Therefore, one embodiment of the present disclosure is not necessarily limited to this size, and the shape and size of one or more components in the drawings do not reflect the true proportion. In addition, the drawings schematically show ideal examples, and one embodiment of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0031] The ordinal numbers such as "first", "second", and "third" in this specification are provided to avoid confusion of constituent elements, and are not intended to limit the quantity. The "plurality" in this disclosure means two or more.

[0032] Holographic video conferencing is an important application scenario in 3D display and is an application that is relatively easy to implement. In holographic video conferencing, the generation of new viewpoint images has become one of the important research issues. This problem aims to provide one or more images of the same scene and generate an image of the scene at a certain viewpoint.

[0033] like Figure 1 As shown, the embodiment of the present disclosure provides a method for generating a new viewpoint image, comprising:

[0034] Step 101, acquiring a plurality of first initial images, wherein the plurality of first initial images are simultaneously acquired by a plurality of dispersed image acquisition devices, and image acquisition areas of the plurality of image acquisition devices include overlapping areas;

[0035] Step 102: obtaining eye coordinate information of the user at the display end;

[0036] Step 103: Input the multiple first initial images and the human eye coordinate information into a new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information. The new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate the depth information according to the multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to the new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

[0037] The disclosed embodiment establishes a new viewpoint generation model, inputs multiple first initial images (the multiple first initial images constitute one frame of image) and human eye coordinate information into the new viewpoint generation model, and obtains a new viewpoint image corresponding to the human eye coordinate information. With the help of the general parallel computing capability of a high-performance graphics processing unit (GPU) and the representation capability of a neural network, a very good new viewpoint image generation effect can be achieved.

[0038] In some exemplary embodiments, the image acquisition device is located at the acquisition end, and the acquisition end is connected to the display end via a network.

[0039] Figure 2 FIG. 1 is a schematic diagram of an application scenario of the new viewpoint image generation method of the embodiment of the present disclosure. Figure 2 As shown, the system is an end-to-end single-person chat system. The user sits in front of the display. The system at each end runs a new viewpoint generation model. One end is the acquisition end (and the display end), and the other end is the display end (and the acquisition end). After the image acquisition device at the acquisition end acquires the initial image, it sends it to the display end through the network. The display end system generates a new viewpoint image corresponding to the human eye coordinate information of the display end user by running the new viewpoint generation model.

[0040] In some exemplary embodiments, the image acquisition device may be a camera or any other image acquisition device.

[0041] Figure 3 for Figure 2 Schematic diagram of data transmission process in the application scenario shown in FIG. Figure 3 As shown, after the camera at the acquisition end acquires the real scene image, it is first transmitted to the central processing unit (CPU) at the acquisition end. The CPU at the acquisition end transmits the data to the GPU at the acquisition end through the bus system. The GPU at the acquisition end performs image preprocessing on the original shooting data. The image preprocessing may include image signal processing (ISP), color correction, and interpolation into RGB data. Then, the RGB data is directly encoded using the GPU hardware encoding chip, and the encoded data is uploaded to the network. The display end processor pulls the video stream from the network, and decodes it through the GPU hardware decoding chip to obtain the source data collected by the camera at the acquisition end. The source data is used as the input of the human eye positioning algorithm and the viewpoint generation algorithm.

[0042] In some exemplary embodiments, there may be four image acquisition devices, which are respectively arranged in the left area, right area, upper area and lower area of ​​the acquisition end display, and the image acquisition areas of the four image acquisition devices are all facing the front area of ​​the acquisition end display.

[0043] In the disclosed embodiment, the number of image acquisition devices is not fixed, and can be four, six, or even more. In theory, the more image acquisition devices there are, the better the effect, but the more data and cost generated, which is not conducive to productization and promotion. Therefore, the new viewpoint generation method of the disclosed embodiment adopts a sparse number of image acquisition devices. The placement of the image acquisition device is not fixed, and it can be placed around the acquisition end display as much as possible, and the lens is facing the center of the acquisition end display to shoot, and try to ensure that each image acquisition device can capture the user sitting in front of the acquisition end display.

[0044] The application of holographic video communication technology in video conferencing will become more and more popular, and the present disclosure mainly optimizes the technology for this application scenario. This system comes with multiple cameras and uses neural network modeling technology to generate the multi-viewpoint images required by the light field display. From a technical point of view, if you want to obtain image data of people at any angle in front of the display by inputting only images taken by a few cameras, the neural network must have seen enough image data from any angle before. Therefore, a mature neural network system must first undergo a large amount of data training to learn before it is possible to obtain the optimal network weight parameters. When we train the new viewpoint generation model, we should try to let the new viewpoint generation model see rich image data from any angle on the front of the display, so that when we only input a sparse number of camera images, the new viewpoint generation model can also output images from any angle.

[0045] However, our system only has a sparse number of cameras and it is impossible to collect enough rich image data from any perspective. Figure 4 As shown, the present invention constructs two neural network models: a three-dimensional reconstruction model and a new viewpoint generation model. The first neural network model (i.e., the three-dimensional reconstruction model) generates a three-dimensional model based on a frame of input image data (each frame of image data includes images simultaneously captured by multiple image acquisition devices). Based on the three-dimensional model, rich image data of any perspective can be generated, and then the rich image data is used for learning and training of the second neural network model (i.e., the new viewpoint generation model). In this way, enough training data can be provided for the new viewpoint generation model. This is a method of exchanging time for space, so that the second neural network model can meet the real-time characteristics of the video chat scenario as much as possible. The second neural network model is the network model when the system is running in real time. Figure 4 In the present invention, the training networks of the 3D reconstruction model and the new viewpoint generation model include loss calculation and back-propagation modules, while the inference networks of the 3D reconstruction model and the new viewpoint generation model do not include loss calculation and back-propagation modules. In addition, knowledge distillation is performed through the teacher model and the student model in the training network of the new viewpoint generation model, while only the corresponding student model is included in the inference network of the new viewpoint generation model, thereby achieving model acceleration while ensuring the model effect.

[0046] Theoretically, only the 3D reconstruction model can generate image data of any viewpoint, but the method of generating viewpoint graph based on the 3D reconstruction model is time-consuming and difficult to meet the real-time requirements of the system. The 3D reconstruction model is the training network of the new viewpoint generation model. Figure 4 As shown in the figure, for our system, the first three stages are all preparation stages, serving the last stage. The new viewpoint generation model inference network in the last stage is the algorithm module that runs in real time when the system is used. This algorithm module only needs to input images taken by a sparse number of cameras to output images of any angle of view on the front of the display. That is, when the system is used, Figure 2 Only the new viewpoint generation model is deployed on the processors (computers) at both ends. By inputting the images collected by a sparse number of cameras at the acquisition end into the new viewpoint generation model, images of any viewing angle on the front of the display can be output.

[0047] In some exemplary embodiments, the new viewpoint generation model is trained by the following method:

[0048] Constructing a three-dimensional reconstruction model and a first training data set;

[0049] Using the first training data set to train the 3D reconstruction model, to obtain a trained 3D reconstruction model;

[0050] Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into a trained three-dimensional reconstruction model to obtain a three-dimensional model;

[0051] Using a virtual camera to take two-dimensional images of the three-dimensional model at multiple different angles, and using the two-dimensional images to construct a second training data set;

[0052] The new viewpoint generation model is trained using the second training dataset.

[0053] In theory, the holographic video conferencing application scenario requires a large number of cameras to capture actual images, but too many cameras have a huge impact on cost and power consumption. The new viewpoint image generation method of the disclosed embodiment constructs two sets of neural network algorithms: a 3D reconstruction model and a new viewpoint generation model. The 3D reconstruction model uses time to exchange space to generate enough images of any perspective for the new viewpoint generation model to train. The new viewpoint generation model is used for real-time system operation. This method meets the sparse camera characteristics and real-time characteristics requirements of our system.

[0054] In some exemplary embodiments, the first training data set includes a system-acquired data set and a virtual data set. The system-acquired data set includes images acquired by multiple image acquisition devices, and the virtual data set includes images of different people performing different actions downloaded from the Internet.

[0055] The modeling process of the 3D reconstruction model includes:

[0056] 1) Build Figure 5 The 3D reconstruction model (training network) shown;

[0057] 2) Dataset preparation: construct a virtual dataset, where the virtual dataset includes: downloading professional image data of people in different clothes and doing different actions from the Internet; construct a system acquisition dataset, where the system acquisition dataset includes: using our system to take data of multiple different people; merge the two datasets together to form the first training dataset;

[0058] 3) Training process: The first training data set is divided into a first training set and a first test set, and the first training set is used to train the 3D reconstruction model. During the training process, the back propagation algorithm is used to calculate the model parameters and adjust the model weights and biases to achieve the best model performance;

[0059] 4) Use the first test set to verify the trained model to see if the data set has generalization capabilities. If the first test set input into the model works well, it means that the model training is relatively successful. If not, the training strategy needs to be adjusted based on experience.

[0060] 5) Use the trained 3D reconstruction model network model parameters to input Figure 4 The 3D reconstruction model inference network (i.e., removing Figure 5 The loss calculation and back propagation module in the network) is used to input a sparse number of camera images. The 3D reconstruction model inference network performs 3D modeling based on the input camera images and outputs a 3D model containing depth and color. The virtual camera is placed in the space where the 3D model is located, and the 3D model is photographed from certain perspectives to generate many 2D images from different perspectives. These 2D images can be used to train the new viewpoint generation model.

[0061] In some exemplary embodiments, the three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module, and a weighted summation module, wherein:

[0062] The feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract a feature map of the input image, and project the N1*N2 sampling points onto the feature map according to the shooting parameters to obtain a color feature of each sampling point, where N1 is greater than 1 and N2 is greater than 1;

[0063] The directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, wherein the target geometric model includes a distance field value and a second feature value of each sampling point;

[0064] A color extraction module is configured to obtain a color value of each sampling point according to a second eigenvalue of each sampling point and a light direction;

[0065] A weight calculation module, configured to calculate the weight of each sampling point according to the distance field value of each sampling point;

[0066] The weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light ray on the image.

[0067] Exemplarily, N1 may be equal to 1024*1024, and N2 may be equal to 128, however, the embodiments of the present disclosure are not limited thereto.

[0068] In some exemplary embodiments, the weight of the sampling point is exponentially related to the distance field value of the sampling point.

[0069] In some exemplary embodiments, when training a three-dimensional reconstruction model using the first training data set, a loss function is established, where the loss function includes the difference between the color value initially sampled at each pixel and the color value obtained by weighted summation of each pixel.

[0070] Figure 5Schematic diagram of the 3D reconstruction process of the 3D reconstruction model. The 3D reconstruction model uses continuous color images under limited and sparse viewpoints (exemplarily, 4 viewpoints) to restore the 3D model and color information of the human body, providing high-quality and sufficient training data for the new viewpoint generation model.

[0071] In the embodiment of the present disclosure, the 3D reconstruction model includes the following modules:

[0072] Feature extraction module (not shown in the figure): mainly configured to extract feature maps of input images;

[0073] Directed distance field network module (SDFNetwork): covers the core ideas of four-dimensional expression of continuous video frames, projection decomposition of feature vectors, and guided convergence of MVS. Its main function is to predict the geometric information of the human body model;

[0074] Color extraction module (ColorNetwork): mainly configured to predict the color information (color and transparency, neural radiation field) of the human body model photographed at a specified position;

[0075] Weight calculation module (SingleVarianceNetwork): configured to establish a connection between the neural radiation field and the SDF field in order to use image information as supervision of the geometric model;

[0076] Weighted sum module ( Figure 5 Σ in ): is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of each pixel.

[0077] For each input image, the effective point range in the image is obtained according to its mask (commonly used for cutting out images to obtain the outline of a person). Each pixel on the image corresponds to a ray in space starting from the optical center of the camera. During training, N1 rays are randomly selected, and N2 sampling points are sampled on each ray to obtain N1*N2 sampling points (pts) in space. The four input images are convolved through a feature extraction module (not shown in the figure, it can be a convolutional network) to extract four feature maps. Using the camera's internal and external parameters, the N1*N2 sampling points are projected onto the four feature maps respectively, and the color feature Projs of each sampling point can be obtained. The N1*N2 sampling point pts, the color feature Projs of each sampling point, and the video frame number Time where this group of images are located together constitute the input of SDFNetWork. SDFNetWork can be composed of a series of convolutional network structures, the purpose of which is to obtain the geometric model of the target (person) (represented by the distance field sdf). The output of SDFNetWork contains two parts, one is the distance field value corresponding to each sampling point, and the other is the second eigenvalue features (the second eigenvalue contains color information). The second eigenvalue of each sampling point plus the corresponding light direction (view dirs), together constitute the input of ColorNetwork. The function of ColorNetwork is to obtain the color in the neural radiation field, that is, the color value corresponding to this sampling point. SingleVarianceNetwork is a single-parameter network. We believe that the distance field value corresponding to each sampling point pt and its transparency or weight in the neural radiation field have an exponential relationship. This single-parameter network is to obtain the exponential parameter sigma_f in this exponential relationship. In other words, the distance field value can obtain the transparency or weight corresponding to this sampling point through the exponential relationship sigma_f. The weighted sum of the color and transparency of all sampling points on N rays can obtain the color value of the pixel corresponding to this ray on the image. The difference between this color value and the color value corresponding to the pixel position when we first sampled (this difference is not necessarily the simple absolute value of the difference) is the loss (Loss). Using this Loss to supervise the back propagation of each module is a training process.

[0078] In some exemplary embodiments, the second training data set includes a first three-dimensional model data set and a second portrait model data set, the first three-dimensional model data set includes two-dimensional images of the three-dimensional model taken at multiple different angles by a virtual camera, and the second portrait model data set includes two-dimensional images of the three-dimensional portrait model downloaded from the Internet taken at multiple different angles by a virtual camera.

[0079] The modeling process of the new viewpoint generation model includes:

[0080] 1) Build Figure 6 The new viewpoint generation model shown;

[0081] 2) Using a large number of multi-view images generated by the 3D reconstruction model, construct a first 3D model dataset, using a portrait model downloaded from the Internet to construct a second portrait model dataset, and merging the two datasets to construct a second training dataset that is ultimately used to train the new viewpoint generation model;

[0082] 3) Training process: The second training data set is divided into a second training set and a second test set, and the second training set is used to train the new viewpoint generation model. During the training process, the back propagation algorithm is used to calculate the model parameters and adjust the model weights and biases to achieve the best model performance;

[0083] 4) Use the second test set to verify the trained model to see whether the data set has generalization ability. If the second test set input model has a good effect, it means that the model training is relatively successful. If the effect is not good, it is necessary to adjust the training strategy based on experience;

[0084] 5) Use the trained network model parameters to input Figure 4 The new viewpoint generation model inference network in . This network is used to input images collected by a sparse number of cameras and can render and output 2D images of arbitrary perspectives.

[0085] Since the final output image is displayed on a 3D display, the specific field of view angles for rendering images can be combined with eye positioning. The eye coordinates calculated by the eye positioning algorithm can be transmitted to the new viewpoint generation model inference network, and the viewpoint images corresponding to the left and right eyes can be rendered in real time (combined with the display principle of the 3D display).

[0086] Figure 6 It is a schematic diagram of the new viewpoint image generation process of the new viewpoint generation model, such as Figure 6 As shown, the left, right and upper three viewpoint color images are input, the disparity maps of the upper left and upper right view images are calculated to obtain the depth maps of the left and right view points, the color image of the new viewpoint (512*512) is predicted, and then the super-resolution module is used to output 1K, 2K and other resolution high-definition images.

[0087] In the embodiment of the present disclosure, the new viewpoint generation model includes the following modules:

[0088] Depth Calculation Module (MVS): This module can be a non-training network module, using the open source RMVS algorithm and weight files, and its main function is to calculate depth information based on multi-view color images;

[0089] Projection prediction module (ColorNet): Based on the left and right viewpoint color images, the depth map is projected to the new viewpoint position to predict the accurate new viewpoint image (512*512);

[0090] Stretching module (Resize): configured to stretch the predicted new viewpoint image;

[0091] Super-resolution module (ColorHrNet): is configured to increase the resolution of the stretched new viewpoint image to obtain the final new viewpoint image. The super-resolution module can super-resolve the 512*512 new viewpoint image to more than 1K*1K.

[0092] like Figure 6 As shown in the figure, the images taken by the left camera and the upper camera are input, and the depth image of the left camera position is obtained through MVS. Similarly, the images taken by the right camera and the upper camera are input, and the depth image of the right camera position is obtained through MVS. According to the color images taken by the left and right camera positions and the calculated depth image, the ColorNet network is used to project (Projection) to the new viewpoint position to predict the accurate new viewpoint image (size: 512*512); the new viewpoint image obtained above is stretched to 1024*1024 size, and then passed through the super-resolution module to improve the clarity of the output new viewpoint image.

[0093] like Figure 6 As shown in the figure, when the new viewpoint generation model is trained, the new viewpoint generation model also includes a diffusion model, in which ColorHrNet is a student model and Diffusion Model is a teacher model, and the two together constitute a super-resolution module. Diffusion Model is configured to generate data similar to the training data. Fundamentally, the working principle of Diffusion Model is to destroy the training data by continuously adding Gaussian noise, and then restore the data by learning the inverse denoising process. After training, we can use Diffusion Model to pass randomly sampled noise into the model and generate data through the learned denoising process. Diffusion Model is a large and complex model, and ColorHrNet is a small and simple model. During training, Diffusion Model supervises ColorHrNet for training, and only ColorHrNet is used when it is used. Since ColorHrNet is a simple small model and is also the network model for the actual operation of the entire algorithm, model acceleration is achieved while ensuring the model effect.

[0094] In some exemplary embodiments, when the new viewpoint generation model is trained using the second training data set, the new viewpoint generation model further includes a diffusion model, wherein the diffusion model is a teacher model and the super-resolution module is a student model.

[0095] In some exemplary embodiments, the depth calculation module is specifically configured as follows:

[0096] Depth information is calculated based on the first initial image captured by each two image capture devices, wherein each two image capture devices include two image capture devices arranged in the left area and the upper area of ​​the first display, and two image capture devices arranged in the right area and the upper area of ​​the first display.

[0097] In the embodiment of the present disclosure, the depth calculation module may also calculate the depth information based on the image acquisition devices at other locations, and the embodiment of the present disclosure is not limited to this.

[0098] like Figure 7 As shown, the embodiment of the present disclosure also provides a training method for a new viewpoint generation model, including:

[0099] Step 701: construct a three-dimensional reconstruction model and a first training data set;

[0100] Step 702: Use the first training data set to train the 3D reconstruction model to obtain the trained 3D reconstruction model;

[0101] Step 703: acquiring a second initial image acquired by a plurality of image acquisition devices, and inputting the second initial image into the trained 3D reconstruction model to obtain a 3D model;

[0102] Step 704: Use a virtual camera to capture two-dimensional images of the three-dimensional model at multiple different angles, and use the two-dimensional images to construct a second training data set;

[0103] Step 705: Use the second training data set to train the new viewpoint generation model.

[0104] In some exemplary embodiments, the first training data set includes a system-acquired data set and a virtual data set, the system-acquired data set includes images acquired by the plurality of image acquisition devices, and the virtual data set includes images of different people performing different actions downloaded from the Internet.

[0105] In some exemplary embodiments, the second training data set includes a first three-dimensional model data set and a second portrait model data set, the first three-dimensional model data set including two-dimensional images of the three-dimensional model taken at multiple different angles by a virtual camera, and the second portrait model data set including two-dimensional images of a three-dimensional portrait model downloaded from the Internet taken at multiple different angles by a virtual camera.

[0106] In some exemplary embodiments, the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein:

[0107] The depth calculation module is configured to calculate depth information according to the plurality of first initial images to obtain a depth image;

[0108] The projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image;

[0109] The stretching module is configured to stretch the predicted new viewpoint image;

[0110] The super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

[0111] In some exemplary embodiments, the image acquisition device is located at an acquisition end, and the acquisition end is connected to the display end via a network.

[0112] In some exemplary embodiments, the image acquisition devices include four, and the four image acquisition devices are respectively arranged in the left area, right area, upper area and lower area of ​​the first display, and the image acquisition areas of the four image acquisition devices face the front area of ​​the first display.

[0113] In some exemplary embodiments, the depth calculation module is specifically configured as follows:

[0114] Depth information is calculated based on the first initial images captured by every two of the image capture devices, and every two of the image capture devices include two of the image capture devices arranged in the left area and the upper area of ​​the first display, and two of the image capture devices arranged in the right area and the upper area of ​​the first display.

[0115] In some exemplary embodiments, when the new viewpoint generation model is trained using the second training data set, the new viewpoint generation model further includes a diffusion model, the diffusion model is a teacher model, and the super-resolution module is a student model.

[0116] In some exemplary embodiments, the three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module and a weighted summation module, wherein:

[0117] The feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract a feature map of the input image, and project N1*N2 sampling points onto the feature map according to the shooting parameters to obtain a color feature of each sampling point, where N1 is greater than 1 and N2 is greater than 1;

[0118] The directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, wherein the target geometric model includes a distance field value and a second feature value of each sampling point;

[0119] The color extraction module is configured to obtain the color value of each sampling point according to the second eigenvalue of each sampling point and the light direction;

[0120] The weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point;

[0121] The weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light ray on the image.

[0122] In some exemplary embodiments, the weight of a sampling point is exponentially corresponding to the distance field value of the sampling point.

[0123] In some exemplary embodiments, when the first training data set is used to train the three-dimensional reconstruction model, a loss function is established, where the loss function includes the difference between the color value initially sampled at each pixel and the color value obtained by weighted summation of each pixel.

[0124] An embodiment of the present disclosure also provides a new viewpoint generation device, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the new viewpoint image generation method as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0125] like Figure 8As shown, in one example, a new viewpoint generating device may include: a first processor 810, a first memory 820, a first bus system 830 and a first transceiver 840, wherein the first processor 810, the first memory 820 and the first transceiver 840 are connected via the first bus system 830, the first memory 820 is used to store instructions, and the first processor 810 is used to execute the instructions stored in the first memory 820 to control the first transceiver 840 to send and receive signals. Specifically, the first transceiver 840 can obtain multiple first initial images under the control of the first processor 810, and the multiple first initial images are simultaneously acquired by multiple dispersed image acquisition devices, and the image acquisition areas of the multiple image acquisition devices include overlapping areas. The first processor 810 obtains the human eye coordinate information of the user on the display end; the first initial image and the human eye coordinate information are input into the new viewpoint generation model to obtain a new viewpoint image corresponding to the human eye coordinate information, and the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate the depth information according to the multiple first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to the new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image; and the final new viewpoint image is output to the display interface of the display end through the first transceiver 840.

[0126] It should be understood that the first processor 810 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0127] The first memory 820 may include a read-only memory and a random access memory, and provide instructions and data to the first processor 810. A portion of the first memory 820 may also include a nonvolatile random access memory. For example, the first memory 820 may also store information on the device type.

[0128] The first bus system 830 may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. Figure 8 In the figure, various buses are labeled as a first bus system 830 .

[0129] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the first processor 810 or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the first memory 820, and the first processor 810 reads the information in the first memory 820 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0130] The embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method for generating a new viewpoint image as described in any embodiment of the present disclosure is implemented. The method for driving the generation of a new viewpoint image by executing executable instructions is basically the same as the method for generating a new viewpoint image provided in the above embodiment of the present disclosure, and will not be described in detail here.

[0131] In some possible implementations, various aspects of the new viewpoint image generation method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the new viewpoint image generation method according to various exemplary implementations of the present application described above in this specification. For example, the computer device may execute the new viewpoint image generation method recorded in the embodiments of the present application.

[0132] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0133] An embodiment of the present disclosure also provides a training device for a new viewpoint generation model, comprising a memory; and a processor connected to the memory, wherein the memory is used to store instructions, and the processor is configured to execute the steps of the training method for the new viewpoint generation model as described in any embodiment of the present disclosure based on the instructions stored in the memory.

[0134] like Fig. 9 As shown, in one example, the training device for the new viewpoint generation model may include: a second processor 910, a second memory 920, a second bus system 930, and a second transceiver 940, wherein the second processor 910, the second memory 920, and the second transceiver 940 are connected via the second bus system 930, the second memory 920 is used to store instructions, and the second processor 910 is used to execute the instructions stored in the second memory 920 to control the second transceiver 940 to send and receive signals. Specifically, the second transceiver 940 can obtain the second initial image acquired by multiple image acquisition devices under the control of the second processor 910, and the second processor 910 constructs a three-dimensional reconstruction model and a first training data set; the three-dimensional reconstruction model is trained using the first training data set to obtain the trained three-dimensional reconstruction model; the second initial image is input into the trained three-dimensional reconstruction model to obtain a three-dimensional model; the two-dimensional images of the three-dimensional model at multiple different angles are taken by a virtual camera, and the two-dimensional images are used to construct a second training data set; and the new viewpoint generation model is trained using the second training data set.

[0135] It should be understood that the second processor 910 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0136] The second memory 920 may include a read-only memory and a random access memory, and provide instructions and data to the second processor 910. A portion of the second memory 920 may also include a nonvolatile random access memory. For example, the second memory 920 may also store information on the device type.

[0137] In addition to the data bus, the second bus system 930 may also include a power bus, a control bus, and a status signal bus. Fig. 9 In the figure, various buses are labeled as a second bus system 930.

[0138] In the implementation process, the processing performed by the processing device can be completed by the hardware integrated logic circuit in the second processor 910 or the instructions in the form of software. That is, the method steps of the embodiment of the present disclosure can be embodied as being executed by a hardware processor, or being executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the second memory 920, and the second processor 910 reads the information in the second memory 920 and completes the steps of the above method in combination with its hardware. To avoid repetition, it will not be described in detail here.

[0139] The embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method for a new viewpoint generation model as described in any embodiment of the present disclosure.

[0140] In some possible implementations, various aspects of the training method for a new viewpoint generation model provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of the training method for a new viewpoint generation model according to various exemplary implementations of the present application described above in this specification. For example, the computer device may execute the training method for a new viewpoint generation model described in the embodiments of the present application.

[0141] The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0142] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0143] Although the embodiments disclosed in the present disclosure are as above, the contents described are only embodiments adopted to facilitate understanding of the present disclosure and are not intended to limit the present invention. Any technician in the relevant field can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present disclosure, but the patent protection scope of the present invention shall still be subject to the scope defined by the attached claims.

[0144] The drawings of the embodiments of the present disclosure only involve the structures involved in the embodiments of the present disclosure, and other structures may refer to the general design.

[0145] For the sake of clarity, in the drawings used to describe the embodiments of the present disclosure, the thickness and size of the layer or microstructure are exaggerated. It will be understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element, or there may be intermediate elements.

[0146] Although the embodiments disclosed in the present disclosure are as above, the contents described are only embodiments adopted to facilitate understanding of the present disclosure and are not intended to limit the present disclosure. Any technician in the field to which the present disclosure belongs can make any modifications and changes in the form and details of implementation without departing from the spirit and scope disclosed in the present disclosure, but the scope of patent protection of the present disclosure shall still be subject to the scope defined in the attached claims.

Claims

1. A method for generating a new viewpoint image, characterized in that: include: Acquire a plurality of first initial images, wherein the plurality of first initial images are simultaneously acquired by a plurality of dispersed image acquisition devices, and image acquisition areas of the plurality of image acquisition devices include overlapping areas; Obtain eye coordinate information of the user at the display end; Inputting the first initial image and the eye coordinate information into a new viewpoint generation model to obtain a new viewpoint image corresponding to the eye coordinate information, wherein the new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein the depth calculation module is configured to calculate depth information according to a plurality of the first initial images to obtain a depth image, the projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image, the stretching module is configured to stretch the predicted new viewpoint image, and the super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image; The new viewpoint generation model is trained by the following method: constructing a three-dimensional reconstruction model and a first training data set; using the first training data set to train the three-dimensional reconstruction model to obtain the trained three-dimensional reconstruction model; obtaining a second initial image acquired by multiple image acquisition devices, and inputting the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model; using a virtual camera to shoot two-dimensional images of the three-dimensional model at multiple different angles, and using the two-dimensional images to construct a second training data set; using the second training data set to train the new viewpoint generation model; the three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module and a weighted summation module, wherein: the feature extraction module is configured to select N1 rays, set N on each ray 2 sampling points, extract the feature map of the input image, and project N1*N2 sampling points onto the feature map according to the shooting parameters to obtain the color feature of each sampling point, N1 is greater than 1, and N2 is greater than 1; the directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, and the target geometric model includes the distance field value and the second eigenvalue of each sampling point; the color extraction module is configured to obtain the color value of each sampling point according to the second eigenvalue of each sampling point and the light direction; the weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point; the weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each light on the image.

2. The method for generating a new viewpoint image according to claim 1, characterized in that: The weight of the sampling point is in exponential correspondence with the distance field value of the sampling point.

3. The method for generating a new viewpoint image according to claim 1, characterized in that: When using the first training data set to train the three-dimensional reconstruction model, a loss function is established, where the loss function includes the difference between the color value initially sampled from each pixel in the input image and the color value obtained by weighted summation of each pixel.

4. The method for generating a new viewpoint image according to claim 1, characterized in that: When the new viewpoint generation model is trained using the second training data set, the new viewpoint generation model also includes a diffusion model, the diffusion model is a teacher model, and the super-resolution module is a student model.

5. The new viewpoint image generation method according to claim 1, characterized in that: The first training data set includes a system acquisition data set and a virtual data set. The system acquisition data set includes images acquired by the multiple image acquisition devices, and the virtual data set includes images of different people performing different actions downloaded from the Internet.

6. The method for generating a new viewpoint image according to claim 1, characterized in that: The second training data set includes a first three-dimensional model data set and a second portrait model data set, the first three-dimensional model data set includes two-dimensional images of the three-dimensional model taken at multiple different angles by a virtual camera, and the second portrait model data set includes two-dimensional images of a three-dimensional portrait model downloaded from the Internet taken at multiple different angles by a virtual camera.

7. The new viewpoint image generation method according to claim 1, characterized in that: The image acquisition device is located at the acquisition end, and the acquisition end is connected to the display end through a network.

8. The method for generating a new viewpoint image according to claim 7, characterized in that: The image acquisition devices include four, and the four image acquisition devices are respectively arranged in the left area, right area, upper area and lower area of ​​the first display. The image acquisition areas of the four image acquisition devices are all facing the front area of ​​the first display, and the first display is located at the acquisition end.

9. The method for generating a new viewpoint image according to claim 8, characterized in that: The depth calculation module is specifically configured as follows: Depth information is calculated based on the first initial images captured by every two of the image capture devices, and every two of the image capture devices include two of the image capture devices arranged in the left area and the upper area of ​​the first display, and two of the image capture devices arranged in the right area and the upper area of ​​the first display.

10. A new viewpoint generation device, characterized in that: It comprises a memory; and a processor connected to the memory, the memory is used to store instructions, and the processor is configured to execute the steps of the new viewpoint image generation method according to any one of claims 1 to 9 based on the instructions stored in the memory.

11. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the new viewpoint image generation method according to any one of claims 1 to 9 is implemented.

12. A training method for a new viewpoint generation model, characterized in that: include: Constructing a three-dimensional reconstruction model and a first training data set; The three-dimensional reconstruction model includes a feature extraction module, a directed distance field network module, a color extraction module, a weight calculation module and a weighted summation module, wherein: the feature extraction module is configured to select N1 rays, set N2 sampling points on each ray, extract the feature map of the input image, and project N1*N2 sampling points onto the feature map according to the shooting parameters to obtain the color feature of each sampling point, N1 is greater than 1, and N2 is greater than 1; the directed distance field network module is configured to generate a target geometric model according to the color feature of each sampling point, and the target geometric model includes the distance field value and the second eigenvalue of each sampling point; the color extraction module is configured to obtain the color value of each sampling point according to the second eigenvalue of each sampling point and the direction of the light; the weight calculation module is configured to calculate the weight of each sampling point according to the distance field value of each sampling point; the weighted summation module is configured to perform weighted summation according to the color value of each sampling point and the weight of each sampling point to obtain the color value of the pixel point corresponding to each ray on the image; Using the first training data set to train the three-dimensional reconstruction model to obtain the trained three-dimensional reconstruction model; Acquire a second initial image acquired by a plurality of image acquisition devices, and input the second initial image into the trained three-dimensional reconstruction model to obtain a three-dimensional model; Taking two-dimensional images of the three-dimensional model at multiple different angles by a virtual camera, and using the two-dimensional images to construct a second training data set; The new viewpoint generation model is trained using the second training dataset.

13. The training method for generating a new viewpoint model according to claim 12, characterized in that: The new viewpoint generation model includes a depth calculation module, a projection prediction module, a stretching module and a super-resolution module, wherein: The depth calculation module is configured to calculate depth information according to the plurality of first initial images to obtain a depth image; The projection prediction module is configured to project the depth image to a new viewpoint position to obtain a predicted new viewpoint image; The stretching module is configured to stretch the predicted new viewpoint image; The super-resolution module is configured to improve the resolution of the stretched new viewpoint image to obtain a final new viewpoint image.

14. The training method for generating a new viewpoint model according to claim 13, characterized in that: When the new viewpoint generation model is trained using the second training data set, the new viewpoint generation model also includes a diffusion model, the diffusion model is a teacher model, and the super-resolution module is a student model.

15. The training method for generating a new viewpoint model according to claim 12, characterized in that: The weight of the sampling point is in exponential correspondence with the distance field value of the sampling point.

16. A training device for a new viewpoint generation model, characterized in that: It comprises a memory; and a processor connected to the memory, the memory being used to store instructions, the processor being configured to execute the steps of the training method for a new viewpoint generation model as described in any one of claims 12 to 15 based on the instructions stored in the memory.

17. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the training method for a new viewpoint generation model as described in any one of claims 12 to 15 is implemented.

Images