Video generation method

Abstract

The embodiment of the invention provides a video generation method, and the method comprises the steps: obtaining at least one image frame in single-viewpoint video data, and dividing the pixel points of the image frame into static pixel points and dynamic pixel points; obtaining static pixel point information and dynamic pixel point information; respectively inputting the static pixel point information and the dynamic pixel point information into a preset neural network model for implicit modeling to obtain a modeling result; and generating multi-view video data based on a modeling result. The dynamic and static point segmentation is performed on the image, and then the dynamic and static point segmentation is respectively used as the input of the preset neural network model to perform implicit modeling, so that the calculation overhead can be reduced, and the free viewpoint video of any viewpoint can be rendered, thereby solving the problem that global modeling is performed by taking a plurality of viewpoints as the input in the related technology. The problems of data acquisition cost increase, application scene limitation, calculation overhead increase and rendering result ill-conditioned caused by insufficient monocular input information are solved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communications, and in particular, to a video generation method. Background Art

[0002] 3D scene reconstruction has been commercialized in many fields, especially the application of indoor 3D scenes, which is involved in real estate, museums, architecture and other fields. Currently, almost all application scenarios are static, but with the increasing demand for efficient and high-precision modeling, its potential application areas include but are not limited to: virtual reality, game entertainment, film and television communication, terrain exploration, etc. Its application methods include but are not limited to: video perspective conversion, real scene dynamic simulation, human-computer interaction and other recommended scenarios for dynamic scene reconstruction.

[0003] When the dynamic scene 3D reconstruction method uses a monocular viewpoint as input, the rendering effect of the dynamic scene is poor due to insufficient input information. The dynamic scene 3D reconstruction method of the related art mostly relies on the input of multiple viewpoints for global modeling, which requires multiple fixed camera devices, which not only increases the data collection cost, limits the application scenarios, but also increases the computing overhead, which is not feasible for most users. Summary of the invention

[0004] An embodiment of the present application provides a video generation method to at least solve the problem that the dynamic scene three-dimensional reconstruction method in the related art relies on the input of multiple viewpoints to perform global modeling, which increases data acquisition costs, limits application scenarios, and increases computing overhead.

[0005] According to an embodiment of the present application, a video generation method is provided, including:

[0006] Acquire at least one image frame in the single-viewpoint video data, and divide the pixels of the image frame into static pixels and dynamic pixels;

[0007] Obtain static pixel information and dynamic pixel information;

[0008] Inputting the static pixel point information and the dynamic pixel point information into a preset neural network model for implicit modeling to obtain a modeling result;

[0009] Based on the modeling result, multi-viewpoint video data is generated.

[0010] According to another embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when run.

[0011] According to another embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0012] In the embodiment of the present application, at least one image frame in the single-view video data is obtained, and the pixels of the image frame are divided into static pixels and dynamic pixels; static pixel information and dynamic pixel information are obtained; the static pixel information and dynamic pixel information are respectively input into the preset neural network model for implicit modeling to obtain the modeling result; based on the modeling result, multi-view video data is generated. By segmenting the image into dynamic and static points, and then using them as the input of the preset neural network model for implicit modeling, the computational overhead can be reduced, and a free viewpoint video of any viewpoint can be rendered, thereby solving the problem in the related art of using multiple viewpoints as input for global modeling, increasing data acquisition costs, limiting application scenarios, increasing computational overhead, and insufficient monocular input information leading to pathological rendering results. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a hardware structure block diagram of a mobile terminal according to a video generation method of an embodiment of the present application;

[0014] Figure 2 is a flowchart of a video generation method according to an embodiment of the present application;

[0015] Figure 3 It is a schematic diagram of a process for completing three-dimensional reconstruction of a dynamic scene based on an input video according to an embodiment of the present application. DETAILED DESCRIPTION

[0016] The embodiments of the present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0018] In view of the problem that the dynamic scene 3D reconstruction method in the related art relies on the input of multiple viewpoints to perform global modeling, which increases the data acquisition cost, limits the application scenarios, and increases the computing overhead, the embodiment of the present application provides a method for video generation, which can generate arbitrary camera viewpoints and new views for the input dynamic scene video, thereby achieving the effect of free viewpoint video and stereo rendering, thereby providing an immersive and almost lifelike viewing experience. The embodiment of the present application can be applied to multiple scenarios, such as live broadcast of sports events, generation of bullet time effects in movies, virtual reality, etc.

[0019] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 1 is a hardware structure block diagram of a mobile terminal of a video generation method according to an embodiment of the present application. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is for illustration only and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0020] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the video generation method in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0021] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0022] In this embodiment, a video generation method is provided. Figure 2 is a flowchart of a video generation method according to an embodiment of the present application, such as Figure 2As shown, the process includes the following steps:

[0023] Step S201: acquiring at least one image frame in single-view video data, and dividing the pixels of the image frame into static pixels and dynamic pixels.

[0024] In an embodiment of the present application, for input single-view video data, the pixels of each frame image in the single-view video data can be divided into dynamic pixels and static pixels, so as to model dynamic scenes and static scenes separately.

[0025] The following describes how to segment image frames into dynamic pixels and static pixels:

[0026] In an exemplary embodiment, dividing the pixels of the image frame into static pixels and dynamic pixels includes: obtaining the color value of the pixel of the current image frame, and the color value of the pixel of the image frame at the previous moment and the color value of the pixel of the image frame at the next moment corresponding to the pixel of the current image frame; determining the color value variance of the pixel according to the color value of the pixel of the current image frame, and the color value of the pixel of the image frame at the previous moment and the color value of the pixel of the image frame at the next moment corresponding to the pixel of the current image frame; determining the pixel as a static pixel when the color value variance of the pixel is less than the preset variance threshold; determining the pixel as a dynamic pixel when the color value variance of the pixel is greater than or equal to the preset variance threshold. .

[0027] Exemplarily, the color value of the pixel in the current image frame, the color value of the pixel in the image frame at the previous moment corresponding to the position of the pixel in the current image frame, and the color value of the pixel in the image frame at the next moment corresponding to the position of the pixel in the current image frame can be obtained, and the average color value of the pixel in the time dimension can be calculated based on the color value of the pixel in the current image frame, the color value of the pixel in the image frame at the previous moment corresponding to the position of the pixel in the current image frame, and the color value of the pixel in the image frame at the next moment corresponding to the position of the pixel in the current image frame.

[0028] The color value variance of the pixel can be calculated based on the average color value, the color value of the pixel in the current image frame, the color value of the pixel in the image frame at the previous moment corresponding to the pixel in the current image frame, and the color value of the pixel in the image frame at the next moment corresponding to the pixel in the current image frame.

[0029] Whether the pixel is a static pixel or a dynamic pixel can be determined based on the color value variance of the pixel and a preset variance threshold.

[0030] As an example, assume that the input monocular video has a total of N frames, and the current frame is recorded as f i(i=1,…,N), take the image frame of the previous moment as f i-1 , take the next moment image frame and record it as f i+1 , (x, y) represents the position index of a pixel point in the current frame, and (x i-1 ,y i-1 ) and (x i+1 ,y i+1 ) represent the position index of the pixel in the previous image frame and the next image frame respectively, and the color value of the pixel in the fth image frame is i-1 、f i 、f i+1 Frame is denoted as C i+t (t=-1,…1), therefore, the color value variance based on the time dimension at the pixel level can be:

[0031]

[0032] in, Indicates the pixel (x, y) at f i-1 、f i 、f i+1 The average pixel value of the frame is obtained by observing the pixel (x, y) at f i-1 、f i 、f i+1 The variance value of the three frames can be used to determine whether the pixel is a dynamic pixel or a static pixel.

[0033] As an example, assuming that Ψ(x, y) represents the static attribute or dynamic attribute of the pixel point (x, y), when Ψ(x, y) = 0, it means that the pixel point is a static pixel point, and when Ψ(x, y) = 1, it means that the current pixel point is a dynamic pixel point. Specifically, it can be expressed as:

[0034]

[0035] Wherein, κ is a preset threshold used to determine the attributes of dynamic pixels or static pixels.

[0036] As an example, the κ value can be selected based on the quality of the video. Since the relative position and color of the static point in the world do not change, the standard deviation is 0. In the RGB three-channel 0-255 range, in an ideal scene without noise and light changes, κ can be any constant greater than 0. When there is noise or disturbance and only parallel light in the scene, the optimal value of κ can vary with the scene. Among them, experiments have found that the choice of κ is most affected by light changes. If there are multiple sources of changing light points in the scene, the κ value can be increased to adapt to the scene requirements.

[0037] Step S202, obtaining static pixel point information and dynamic pixel point information.

[0038] Step S203, inputting the static pixel point information and the dynamic pixel point information into a preset neural network model for implicit modeling to obtain a modeling result.

[0039] Exemplarily, static pixel point information and dynamic pixel point information of an image frame can be obtained, and the static pixel point information and dynamic pixel point information can be respectively input into a preset neural network model for implicit modeling, so as to achieve separate modeling of dynamic scenes and static scenes, which can greatly reduce computational overhead compared to the global dynamic modeling method.

[0040] In an exemplary embodiment, the preset neural network includes a static neural network and a dynamic neural network, and the static pixel point information and the dynamic pixel point information are respectively input into the preset neural network model for implicit modeling to obtain a modeling result, including: inputting the static pixel point information into the static neural network, and inputting the dynamic pixel point information into the dynamic neural network, and performing implicit modeling respectively to obtain a static modeling result and a dynamic modeling result.

[0041] For example, distributed modeling can be performed, where the static pixels of each frame of the image are input into a static neural network, and the dynamic pixels are input into a dynamic neural network, and implicit modeling is performed separately to obtain static modeling results and dynamic modeling results. Distributed modeling can avoid repeated calculations of inefficient data, thereby reducing computational overhead.

[0042] In an exemplary embodiment, the static pixels of the image frame constitute a static scene of the image frame, and the static pixel information includes collecting first camera light information corresponding to the static pixels of the image frame; inputting the static pixels into a preset static neural network for implicit modeling to obtain a static modeling result also includes: inputting the first camera light information into the preset static neural network for implicit modeling to obtain a rendered image color value of the static scene.

[0043] In the embodiment of the present application, the static scene of the image frame may be composed of static pixels of the image frame, and the static pixel information may include light information of the first camera corresponding to the static pixels of the image frame.

[0044] As an example, for a static scene, the following can be obtained using the Neural Radiance Field (NeRF) model:

[0045] (σ s ,c s )=MLP θ (r(t))

[0046] Among them, σ s Represents the density value of the rendered image at a static pixel, cs represents the color value of the rendered image of a static pixel, MLP is a multi-layer perceptron (MLP), θ is the camera angle, and r(t) represents the first camera light corresponding to the static pixel.

[0047] r(t)=o+td

[0048] o represents the starting point of the first camera light, and d represents the direction of the first camera light.

[0049] At the nearest boundary t n , the farthest boundary is t f In the case of , the color value C(r) corresponding to the first camera light r(t) corresponding to the static pixel is:

[0050]

[0051] Among them, t n Indicates the nearest boundary of sampling along the d direction at the starting point 0; t f represents the farthest sampling boundary along the d direction at the starting point 0; c(r(t)) represents the sum of the color values ​​of all sampling points on the first camera ray r(t); C(r) represents the color obtained by the camera ray passing through each pixel, that is, the final imaging color value of the pixel along the ray; T(t) represents the cumulative transparency corresponding to the sampling point at a distance t from the starting point o on this ray; σ(r(t)) represents the density of the sampling points on the first camera ray. Among them,

[0052]

[0053] The function T(t) represents the distance along the ray from t n to f The accumulated transparency, that is, the light from t n Departure to f , the probability of passing through the path, the cumulative transparency of a sampling point is affected by the density of the previous sampling points, s represents the distance between the sampling point and the starting point o, the farther the distance, the greater the influence of the previous sampling point on the sampling point; σ(r(s)) represents the density of the previous sampling point at a distance s on the ray r.

[0054] Exemplarily, the first camera ray r(t) may be input into the above NeRF model for implicit modeling to obtain a rendered image color value and a rendered image density value of a static scene.

[0055] In an exemplary embodiment, the method further includes: obtaining a true color value of a static scene of the image frame; and determining a loss function of the static scene based on a rendered image color value of the static scene and the true color value of the static scene.

[0056] Exemplarily, the true color value of a static scene in an image frame may be obtained, and the loss function of the static scene may be determined based on the color value of a rendered image of the static scene and the true color value of the static scene.

[0057] As an example, the loss function l for a static scene is s It can be expressed as:

[0058]

[0059] Among them, C gt is the true value of the color.

[0060] In an exemplary embodiment, the dynamic pixels of the image frame constitute a dynamic scene of the image frame, and the dynamic pixel point information includes light information of a second camera corresponding to the dynamic pixel points of the image frame and a timestamp corresponding to the dynamic pixel points; the dynamic pixel points are input into a preset dynamic neural network for implicit modeling to obtain a dynamic modeling result, including: inputting the second camera light information and the timestamp into the dynamic neural network for implicit modeling to obtain the rendered image color values ​​of the dynamic scene at different timestamps output by the dynamic neural network.

[0061] For dynamic scenes, a single-time, single-viewpoint image is not sufficient for the deep learning network to converge. In the embodiment of the present application, when modeling dynamic scenes, multi-time constraints are adopted. The camera light information at different times can be input into the dynamic neural network for implicit modeling. The rendered image color values ​​of the dynamic scene at different times can be obtained for model training of dynamic scenes.

[0062] In an exemplary embodiment, before inputting the second camera light information and the timestamp into the dynamic neural network, it also includes: inputting the rendered image color value of the static scene of the current moment image frame into the dynamic neural network to obtain the previous moment scene stream and the next moment scene stream of the current moment image frame output by the dynamic neural network; sampling the previous moment scene stream and the next moment scene stream respectively to obtain the sampled previous moment scene stream and the sampled next moment scene stream.

[0063] In an embodiment of the present application, before the second camera light information and the timestamp corresponding to the dynamic pixel point are input into the dynamic neural network, the output of the static neural network can be used as part of the dynamic neural network input, thereby constraining the convergence of the dynamic neural network.

[0064] As an example, the color value of the rendered image of the static scene of the current moment image frame can be input into the dynamic neural network, and the motion feature map of the previous moment and the motion feature map of the next moment of the current moment image frame are predicted, so that the dynamic neural network outputs the scene flow of the previous moment and the scene flow of the next moment of the current moment image frame. Among them, the scene flow of the previous moment can represent the motion state of the image frame from the previous moment t-1 to the current moment t, that is, the motion feature map of the previous moment; the scene flow of the next moment can represent the motion state of the image frame from the current moment t to the next moment t+1, that is, the motion feature map of the next moment.

[0065] As an example, the modeling of a dynamic scene can be defined as follows:

[0066]

[0067] Among them, F i-1 Indicates the scene flow at the previous moment; F i+1 Indicates the scene flow at the next moment; Indicates the dynamic pixel density at different times and different light directions; It represents the color value of dynamic pixels at different times and different light directions, θ represents the light angle; r(t) represents the light at different times.

[0068] Due to sparse samples, dynamic neural networks may have difficulty converging or have ambiguous results during training. The learning constraints of the network can be increased by locally adding sample features. When the dynamic domain occupies a small proportion, the neural network learning can still converge.

[0069] As an example, the scene stream at the previous moment and the scene stream at the next moment are sampled respectively, so as to obtain the scene stream at the previous moment after sampling and the scene stream at the next moment after sampling.

[0070] For example, we can analyze F i-1 ,F i+1 Perform uniform sampling:

[0071]

[0072] In an exemplary embodiment, the step of inputting the second camera light information and the timestamp into the dynamic neural network to perform implicit modeling to obtain the rendered image color values ​​of the dynamic scene at different timestamps output by the dynamic neural network includes:

[0073] The second camera light information at the current moment, the scene stream at the previous moment after the sampling, and the scene stream at the next moment after the sampling are input into the dynamic neural network for implicit modeling, and the rendered image color value of the dynamic scene at the current moment, the rendered image color value of the dynamic scene at the previous moment, and the rendered image color value of the dynamic scene at the next moment output by the dynamic neural network are obtained.

[0074] For example, the scene flow at the previous moment after sampling and the scene flow at the next moment after sampling are input into the dynamic neural network for modeling. The dynamic neural network can be expressed as:

[0075]

[0076]

[0077] In the process of dynamic scene reconstruction, the embodiment of the present application can use MLP to predict feature maps of three consecutive time dimensions and add light field constraints at intermediate moments, so that when the dynamic scene accounts for a small proportion of the global total, network learning can obtain more features so that the convergence direction is clearer and the pixel representation information is more accurate.

[0078] In an exemplary embodiment, the step of inputting the second camera light information at the current moment, the scene stream at the previous moment after sampling, and the scene stream at the next moment after sampling into the dynamic neural network includes: performing cycle consistency regularization processing on the scene stream at the previous moment after sampling and the scene stream at the next moment after sampling; and inputting the second camera light information at the current moment, the scene stream at the previous moment after sampling, and the scene stream at the next moment after sampling after cycle consistency regularization processing into the preset dynamic neural network.

[0079] Although the above-mentioned use of multi-view constrained dynamic neural networks makes the network convergence direction clearer, if there is no proper regularization, the dynamic neural network is prone to overfitting during the training process, thereby failing to achieve the rendering effect of dynamic scenes. The embodiment of the present application can use a regularized supervision mechanism based on the time domain to enhance the modeling accuracy and rendering effect, thereby preventing overfitting while enhancing the diversity of training set samples.

[0080] As an example, the difference between adjacent image frames can be minimized in a regularized manner, such as:

[0081]

[0082] Thereby, the motion feature map can be further normalized so that the motion feature map is smooth in the time dimension.

[0083] As an example, in order to further regularize the consistency of the feature map, the scene flow at the previous moment after sampling and the scene flow at the next moment after sampling can be processed by cycle consistency regularization, for example:

[0084]

[0085] Among them, l2 is the scene flow after regularization, r is the camera light, and F i-1 is the scene flow at the previous moment, F i+1 The scene flow for the next moment.

[0086] As an example, the scene flow at the previous moment after sampling and the scene flow at the next moment after sampling after cycle consistency regularization can be input into a preset dynamic neural network for implicit modeling, which further standardizes the motion feature map, reduces the ambiguity of the results, and makes the motion feature map smooth in the time dimension to increase modeling accuracy and rendering effect.

[0087] In an exemplary embodiment, after obtaining the color value of the rendered image of the dynamic scene at the current moment, the color value of the rendered image of the dynamic scene at the previous moment, and the color value of the rendered image of the dynamic scene at the next moment output by the dynamic neural network, it also includes:

[0088] Obtain the true color value of the image frame of the dynamic scene at the current moment, the true color value of the image frame of the dynamic scene at the previous moment, and the true color value of the image frame of the dynamic scene at the next moment;

[0089] The loss function of the dynamic scene is determined based on the rendered image color value of the dynamic scene at the current moment, the rendered image color value of the dynamic scene at the previous moment, the rendered image color value of the dynamic scene at the next moment, and the image frame color true value of the dynamic scene at the current moment, the image frame color true value of the dynamic scene at the previous moment, and the image frame color true value of the dynamic scene at the next moment.

[0090] As an example, the color difference of the image frame of the dynamic scene at the current moment can be determined based on the true color value of the image frame of the dynamic scene at the current moment and the color value of the rendered image of the dynamic scene at the current moment; the color difference of the image frame of the dynamic scene at the previous moment can be determined based on the true color value of the image frame of the dynamic scene at the previous moment and the color value of the rendered image of the dynamic scene at the previous moment; and the color difference of the image frame of the dynamic scene at the next moment can be determined based on the true color value of the image frame of the dynamic scene at the next moment and the color value of the rendered image of the dynamic scene at the next moment.

[0091] The loss function of the dynamic scene can be determined based on the color difference of the image frame of the dynamic scene at the current moment, the color difference of the image frame of the dynamic scene at the previous moment, and the color difference of the image frame of the dynamic scene at the next moment.

[0092] For example, the loss function of the dynamic MLP network can be designed as:

[0093]

[0094] Among them, x represents the row of the image, y represents the column of the image, and r xy Represents the starting point o, the light in the direction of x row and y column, Indicates the rendering color value at different times and different light directions, C gt Represents the true color value at different times and different light directions.

[0095] In an exemplary embodiment, a target loss function is determined based on the loss function of the static scene and the loss function of the dynamic scene; and the preset neural network model is trained based on the target loss function.

[0096] As an example, the objective loss function can be expressed as:

[0097]

[0098] The preset neural network model can be trained based on the target loss function l to optimize the preset neural network model.

[0099] Step S204: generating multi-viewpoint video data based on the modeling result.

[0100] Exemplarily, texture fusion may be performed based on the rendering color values ​​of static pixels output by a static neural network and the rendering color values ​​of dynamic pixels output by a dynamic neural network, thereby generating multi-viewpoint video data.

[0101] As an example, the dynamic pixel point data and static pixel point data obtained through training can be voxel rendered as a whole to generate multi-viewpoint video data based on neural radiation field rendering and model reconstruction under time.

[0102] The video generation method of the present application is further illustrated below through an example.

[0103] Example 1:

[0104] Figure 3 is a schematic diagram of a process for completing 3D reconstruction of a dynamic scene based on an input video according to an embodiment of the present application, referring to Figure 3 , where the input parameter is the video stream and the output parameter is the new perspective dynamic video. The specific steps can be as follows:

[0105] S301: Video can be captured on a mobile device, and the input video can be split into continuous video frames using Fast Forward Moving Picture Expert Group (FFmpeg). The video frame red, green and blue color mode (RedGreenBlue, RGB) images are segmented by a Mask-Recurrent Neural Network (MaskRNN) to generate static masks and dynamic masks.

[0106] S302: Since dynamic scene samples are sparse, all video frames can be used as key frames for training. COLMAP and the Augmented Reality Kit (ARkit) are used to calculate the camera parameters of each frame, and a three-dimensional coordinate system is established with the camera as the origin. The screen pixel coordinates are mapped to three-dimensional coordinates to obtain the scene space position information.

[0107] S303: In the static neural network, only the position information is input to obtain the static color features and density features. Since the same sampling point will produce two different properties, static and dynamic, in dynamic scenes at different viewing angles and at different times, the static output is used as part of the dynamic neural network input and trained jointly with the spatiotemporal information to constrain the network convergence. The loss function is set as the rendering difference of continuous timestamps, and time consistency is introduced to strengthen the constraints. Finally, the dynamic data and static data obtained through training are voxel rendered as a whole to complete the model reconstruction based on the neural radiation field rendering to generate new viewpoints and times.

[0108] In the embodiment of the present application, at least one image frame in the single-view video data is obtained, and the pixels of the image frame are divided into static pixels and dynamic pixels; static pixel information and dynamic pixel information are obtained; the static pixel information and dynamic pixel information are respectively input into the preset neural network model for implicit modeling to obtain the modeling result; based on the modeling result, multi-view video data is generated. By segmenting the image into dynamic and static points, and then using them as the input of the preset neural network model for implicit modeling, the computational overhead can be reduced, and a free viewpoint video of any viewpoint can be rendered, thereby solving the problem in the related art of using multiple viewpoints as input for global modeling, increasing data acquisition costs, limiting application scenarios, increasing computational overhead, and insufficient monocular input information leading to pathological rendering results.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0110] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0111] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0112] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0113] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0114] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0115] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0116] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video generation method, characterized in that: include: Acquire at least one image frame in the single-viewpoint video data, and divide the pixels of the image frame into static pixels and dynamic pixels; Obtain static pixel information and dynamic pixel information; Inputting the static pixel point information and the dynamic pixel point information into a preset neural network model for implicit modeling to obtain a modeling result; Based on the modeling result, multi-viewpoint video data is generated.

2. The method according to claim 1, characterized in that The preset neural network includes a static neural network and a dynamic neural network, and the static pixel point information and the dynamic pixel point information are respectively input into the preset neural network model for implicit modeling to obtain a modeling result, including: The static pixel point information is input into the static neural network, and the dynamic pixel point information is input into the dynamic neural network, and implicit modeling is performed respectively to obtain static modeling results and dynamic modeling results.

3. The method according to claim 2, characterized in that The dividing the pixels of the image frame into static pixels and dynamic pixels comprises: Obtaining the color value of a pixel of the current image frame, as well as the color value of a pixel of the image frame at the previous moment and the color value of a pixel of the image frame at the next moment corresponding to the pixel of the current image frame; Determine the color value variance of the pixel point according to the color value of the pixel point of the current image frame, and the color value of the pixel point of the image frame at the previous moment and the color value of the pixel point of the image frame at the next moment corresponding to the pixel point of the current image frame; When the color value variance of the pixel point is less than the preset variance threshold, determining that the pixel point is a static pixel point; When the color value variance of the pixel is greater than or equal to the preset variance threshold, the pixel is determined to be a dynamic pixel.

4. The method according to claim 2, characterized in that: The static pixels of the image frame constitute a static scene of the image frame, and the static pixel information includes light information of a first camera corresponding to the static pixel of the image frame; The step of inputting the static pixel points into a preset static neural network to perform implicit modeling to obtain a static modeling result includes: The first camera light information is input into the static neural network for implicit modeling to obtain a rendered image color value of a static scene output by the static neural network.

5. The method according to claim 4, characterized in that Also includes: Obtaining the true color value of the static scene of the image frame; A loss function of the static scene is determined based on the color value of the rendered image of the static scene and the true color value of the static scene.

6. The method according to claim 5, characterized in that The dynamic pixels of the image frame constitute a dynamic scene of the image frame, and the dynamic pixel information includes light information of a second camera corresponding to the dynamic pixel of the image frame and a timestamp corresponding to the dynamic pixel; The step of inputting the dynamic pixel points into a preset dynamic neural network to perform implicit modeling to obtain a dynamic modeling result includes: The second camera light information and the timestamp are input into the dynamic neural network for implicit modeling to obtain the rendered image color values ​​of the dynamic scene at different timestamps output by the dynamic neural network.

7. The method according to claim 6, characterized in that Before inputting the second camera light information and the timestamp into the dynamic neural network, the method further includes: Inputting the color value of the rendered image of the static scene of the image frame at the current moment into the dynamic neural network, obtaining the scene stream at the previous moment and the scene stream at the next moment of the image frame at the current moment output by the dynamic neural network; The scene stream at the previous moment and the scene stream at the next moment are sampled respectively to obtain the scene stream at the previous moment after sampling and the scene stream at the next moment after sampling.

8. The method according to claim 7, characterized in that The step of inputting the second camera light information and the timestamp into the dynamic neural network to perform implicit modeling to obtain the rendered image color values ​​of the dynamic scene at different timestamps output by the dynamic neural network includes: The second camera light information at the current moment, the scene stream at the previous moment after the sampling, and the scene stream at the next moment after the sampling are input into the dynamic neural network for implicit modeling, and the rendered image color value of the dynamic scene at the current moment, the rendered image color value of the dynamic scene at the previous moment, and the rendered image color value of the dynamic scene at the next moment output by the dynamic neural network are obtained.

9. The method according to claim 8, characterized in that The step of inputting the second camera light information at the current moment, the scene stream at the previous moment after sampling, and the scene stream at the next moment after sampling into the dynamic neural network comprises: Performing cycle consistency regularization processing on the scene flow at the previous moment after the sampling and the scene flow at the next moment after the sampling; The second camera light information at the current moment, the scene flow at the previous moment after sampling after cycle consistency regularization processing, and the scene flow at the next moment after sampling are input into the preset dynamic neural network.

10. The method according to claim 8, characterized in that After obtaining the color value of the rendered image of the dynamic scene at the current moment, the color value of the rendered image of the dynamic scene at the previous moment, and the color value of the rendered image of the dynamic scene at the next moment output by the dynamic neural network, the method further includes: Obtain the true color value of the image frame of the dynamic scene at the current moment, the true color value of the image frame of the dynamic scene at the previous moment, and the true color value of the image frame of the dynamic scene at the next moment; The loss function of the dynamic scene is determined based on the rendered image color value of the dynamic scene at the current moment, the rendered image color value of the dynamic scene at the previous moment, the rendered image color value of the dynamic scene at the next moment, and the image frame color true value of the dynamic scene at the current moment, the image frame color true value of the dynamic scene at the previous moment, and the image frame color true value of the dynamic scene at the next moment.

11. The method according to claim 10, characterized in that Also includes: Determining a target loss function based on the loss function of the static scene and the loss function of the dynamic scene; Based on the target loss function, the preset neural network model is trained.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 1 to 11 when executed by a processor.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in any one of claims 1 to 11 are implemented.

Images