Virtual viewpoint video sequence generation method and device, equipment and medium

Abstract

The invention discloses a virtual viewpoint video sequence generation method and device, equipment and a medium, and relates to the technical field of three-dimensional display, and the method comprises the steps: obtaining a single viewpoint video sequence, dividing adjacent frame images in the single viewpoint video sequence, and obtaining a current key frame image and a current associated frame image; determining a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image respectively, performing optimization processing on the current key frame depth map and the current associated frame depth map, and determining the current associated frame depth map based on the optimized current key frame depth map and the optimized current associated frame depth map. And performing three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, and generating a virtual viewpoint video sequence. The quality of the generated virtual viewpoint video sequence can be improved, and high-efficiency, high-quality and low-cost conversion from a single-viewpoint video to a multi-viewpoint video is realized.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional display technology, and in particular to a method, device, equipment and medium for generating a virtual viewpoint video sequence. Background Art

[0002] The videos commonly used at present are all shot by a single camera. The binocular videos used in stereoscopic cinemas are already very rare, and the multi-viewpoint videos used in naked-eye 3D display systems are even rarer. The main reason for the lack of display content is that only high-level film and television companies can produce multi-viewpoint videos. Moreover, compared with ordinary videos, the shooting cost of multi-viewpoint videos is increasing exponentially. Under normal circumstances, the shooting cost of binocular videos is twice that of ordinary monocular videos. If the naked-eye 3D display system uses eight-viewpoint videos, the shooting cost will increase to 8 times that of ordinary monocular videos. At present, the production cost of multi-viewpoint video animation is about 6,000 yuan / second. The high production cost has seriously restricted the promotion and popularization of 3D display technology.

[0003] As can be seen from the above, how to improve the quality of generating virtual viewpoint video sequences and achieve efficient, high-quality and low-cost conversion of single-viewpoint videos to multi-viewpoint videos is a problem to be solved in the field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and medium for generating a virtual viewpoint video sequence, which can improve the quality of generating a virtual viewpoint video sequence and realize efficient, high-quality and low-cost conversion from a single-viewpoint video to a multi-viewpoint video. The specific scheme is as follows:

[0005] In a first aspect, the present application discloses a method for generating a virtual viewpoint video sequence, comprising:

[0006] Acquire a single-view video sequence, and divide adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image;

[0007] Determining a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image respectively, and optimizing the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map;

[0008] Based on the optimized current key frame depth map and the optimized current associated frame depth map, performing three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image;

[0009] A virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

[0010] Optionally, dividing adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image includes:

[0011] Filtering a current frame image and a previous frame image adjacent to the current frame image from the single-view video sequence;

[0012] Determine whether the shooting depth of field between the current frame image and the previous frame image is consistent;

[0013] If the shooting depths of field between the current frame image and the previous frame image are inconsistent, taking the current frame image as the current key frame image;

[0014] If the shooting depth of field between the current frame image and the previous frame image is consistent, the current frame image is used as the current associated frame image.

[0015] Optionally, the determining whether the shooting depths of field between the current frame image and the previous frame image are consistent; if the shooting depths of field between the current frame image and the previous frame image are inconsistent, using the current frame image as the current key frame image; if the shooting depths of field between the current frame image and the previous frame image are consistent, using the current frame image as the current associated frame image includes:

[0016] Determine a first target near-view object and a first target far-view object from the current frame image, and calculate a first distance between the first target near-view object and the first target far-view object in a camera optical axis direction;

[0017] Determine a second target near-view object and a second target far-view object from the previous frame image adjacent to the current frame image, and calculate a second distance between the second target near-view object and the second target far-view object in the direction of the camera optical axis;

[0018] Determine whether the first distance is consistent with the second distance. If the first distance is inconsistent with the second distance, use the current frame image as the current key frame image. If the first distance is consistent with the second distance, use the current frame image as the current associated frame image.

[0019] Optionally, optimizing the current key frame depth map to obtain an optimized current key frame depth map includes:

[0020] Performing edge detection and connected domain identification on the current key frame image to obtain each segmented area;

[0021] Mapping each of the segmented areas to the current key frame depth map, and performing histogram statistics and low-frequency mutation depth value correction processing on the depth values ​​corresponding to each of the segmented areas in the mapped current key frame depth map, so as to obtain the processed current key frame depth map;

[0022] The depth values ​​corresponding to the segmented areas in the processed current key frame depth map are respectively subjected to median filtering to obtain an optimized current key frame depth map.

[0023] Optionally, optimizing the current associated frame depth map to obtain an optimized current associated frame depth map includes:

[0024] Mapping each segmented area after edge detection and connected domain identification of the current associated frame image to the current associated frame depth map, and performing histogram statistics, low-frequency mutation depth value correction processing, and median filtering on the depth values ​​of each segmented area in the mapped current associated frame depth map, so as to obtain the current associated frame depth map after median filtering;

[0025] Acquire a previous frame image adjacent to the current associated frame image; the previous frame image includes a previous key frame image or a previous associated frame image;

[0026] Determine each associated frame depth image region from the current associated frame depth map, and determine each image region corresponding to each associated frame depth image region from the current associated frame image, and match each image region with the previous frame image using a block search method to obtain corresponding previous frame image regions;

[0027] Determine each previous frame depth image area of ​​a previous frame depth map of the previous frame image according to each previous frame image area, and determine the optimized current associated frame depth map based on all the previous frame depth image areas and all the associated frame depth image areas.

[0028] Optionally, the step of determining the optimized current associated frame depth map based on all of the previous frame depth image regions and all of the associated frame depth image regions includes:

[0029] Determine whether the depth value of the previous frame depth image area is consistent with the corresponding depth value of the associated frame depth image area;

[0030] If the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area are inconsistent, determining whether a change between the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area meets a normal change condition;

[0031] If the change between the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area does not meet the normal change condition, then calculating the median of the depth values ​​of the previous frame depth image area, and replacing all the depth values ​​of the associated frame depth image area with the median of the depth values;

[0032] Continue to determine the depth values ​​of other depth image areas of the previous frame and corresponding depth image areas of the associated frame;

[0033] When the depth values ​​of all the previous frame depth image regions and all the associated frame depth image regions are determined, an optimized current associated frame depth map is generated.

[0034] Optionally, based on the optimized current key frame depth map and the optimized current associated frame depth map, performing three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, including:

[0035] Based on the depth value of each pixel in the optimized current key frame depth map and the optimized current associated frame depth map and the camera internal parameters of the camera shooting the single-view video sequence, each pixel in the current key frame image and the current associated frame image is projected from the two-dimensional image to the three-dimensional world coordinate system to obtain the key frame three-dimensional point cloud data and the associated frame three-dimensional point cloud data;

[0036] The key frame three-dimensional point cloud data and the associated frame three-dimensional point cloud data are projected onto a virtual viewpoint imaging plane, and two-dimensional image interpolation is used to fill holes to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image.

[0037] In a second aspect, the present application discloses a virtual viewpoint video sequence generation device, comprising:

[0038] An image division module, used for acquiring a single-view video sequence, and dividing adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image;

[0039] An optimization processing module, used to determine a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image, respectively, and optimize the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map;

[0040] A projection module, configured to perform three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image based on the optimized current key frame depth map and the optimized current associated frame depth map, so as to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image;

[0041] The virtual viewpoint video sequence generating module is used to generate a virtual viewpoint video sequence based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

[0042] In a third aspect, the present application discloses an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] The processor is used to execute the computer program to implement the aforementioned virtual viewpoint video sequence generation method.

[0045] In a fourth aspect, the present application discloses a computer storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed method for generating a virtual viewpoint video sequence are implemented.

[0046] It can be seen that the present application provides a method for generating a virtual viewpoint video sequence, including obtaining a single-viewpoint video sequence, dividing adjacent frame images in the single-viewpoint video sequence to obtain a current key frame image and a current associated frame image; determining a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image respectively, optimizing the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map; based on the optimized current key frame depth map and the optimized current associated frame depth map, performing three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image; generating a virtual viewpoint video sequence based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image. The present application divides each frame image in a single-view video sequence to obtain a current key frame image and a current associated frame image, determines the corresponding current key frame depth map and current associated frame depth map, and optimizes the current key frame depth map and the current associated frame depth map to ensure the rationality of the depth information in the single-frame image and the smooth transition of the depth information of the same scene between adjacent frames to obtain an optimized current key frame depth map and an optimized current associated frame depth map, and performs three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, which can effectively improve the generation quality and efficiency of virtual viewpoint images and bring greater visual impact and appeal to the audience. Finally, a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image, thereby realizing efficient, high-quality and low-cost conversion from single-viewpoint video to multi-viewpoint video. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flow chart of a method for generating a virtual viewpoint video sequence disclosed in the present application;

[0049] Figure 2 This is a flow chart of another method for generating a virtual viewpoint video sequence disclosed in this application;

[0050] Figure 3 (a) is the previous frame image adjacent to the current associated frame image disclosed in this application;

[0051] Figure 3 (b) is the current associated frame image disclosed in this application;

[0052] Figure 4 (a) is a depth map of the previous frame of the previous frame image disclosed in this application;

[0053] Figure 4 (b) is the depth map of the current associated frame disclosed in this application;

[0054] Figure 5 A flow chart of a depth information optimization algorithm based on motion estimation disclosed in this application;

[0055] Figure 6 A specific flow chart for generating a virtual viewpoint video sequence disclosed in this application;

[0056] Figure 7 A schematic diagram of the structure of a virtual viewpoint video sequence generating device disclosed in the present application;

[0057] Figure 8 A structural diagram of an electronic device provided for this application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] The videos commonly used at present are all shot by a single camera. The binocular videos used in stereoscopic cinemas are already very rare, and the multi-viewpoint videos used in naked-eye 3D display systems are even rarer. The main reason for the lack of display content is that only high-level film and television companies can produce multi-viewpoint videos. Moreover, compared with ordinary videos, the shooting cost of multi-viewpoint videos is increasing exponentially. Under normal circumstances, the shooting cost of binocular videos is twice that of ordinary monocular videos. If the naked-eye 3D display system uses eight-viewpoint videos, the shooting cost will increase to 8 times that of ordinary monocular videos. At present, the production cost of multi-viewpoint video animation is about 6,000 yuan / second. The high production cost seriously restricts the promotion and popularization of 3D display technology. As can be seen from the above, how to improve the quality of generating virtual viewpoint video sequences and realize the efficient, high-quality and low-cost conversion of single-viewpoint videos to multi-viewpoint videos is a problem to be solved in this field.

[0060] See also Figure 1As shown, the embodiment of the present invention discloses a method for generating a virtual viewpoint video sequence, which may specifically include:

[0061] Step S11: obtaining a single-view video sequence, and dividing adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image.

[0062] In this embodiment, a single-view video sequence is obtained, and a current frame image and a previous frame image adjacent to the current frame image are filtered from the single-view video sequence; it is determined whether the shooting depth of field between the current frame image and the previous frame image is consistent; if the shooting depth of field between the current frame image and the previous frame image is inconsistent, the current frame image is used as the current key frame image; if the shooting depth of field between the current frame image and the previous frame image is consistent, the current frame image is used as the current associated frame image.

[0063] Specifically, a single-view video sequence is obtained, a current frame image and a previous frame image adjacent to the current frame image are filtered from the single-view video sequence, a first target near-view object and a first target far-view object are determined from the current frame image, and a first distance between the first target near-view object and the first target far-view object in the direction of the camera optical axis is calculated, a second target near-view object and a second target far-view object are determined from the previous frame image adjacent to the current frame image, and a second distance between the second target near-view object and the second target far-view object in the direction of the camera optical axis is calculated, and whether the first distance is consistent with the second distance is determined; if the first distance is inconsistent with the second distance, the current frame image is used as the current key frame image; if the first distance is consistent with the second distance, the current frame image is used as the current associated frame image.

[0064] That is to say, a group of single-view video sequences are obtained, and the video sequences are divided into two categories, namely key frames and associated frames, according to whether the shooting depth of field of two adjacent frames of the single-view video sequence has changed, to obtain a current key frame image and a current associated frame image; wherein the shooting depth of field is the distance between the nearest scene and the farthest scene in the image in the direction of the camera optical axis, that is, for two adjacent frames in the video sequence, if the shooting depth of field of the current frame image has changed relative to the previous frame image, the current frame image is defined as the current key frame image, and if the shooting depth of field of the current frame image has not changed relative to the previous frame image, the current frame image is defined as the current associated frame image.

[0065] Step S12: determining a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image respectively, and optimizing the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map.

[0066] Step S13: Based on the optimized current key frame depth map and the optimized current associated frame depth map, perform three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image.

[0067] In this embodiment, based on the depth value of each pixel in the optimized current key frame depth map and the optimized current associated frame depth map and the camera internal parameters of the camera that shoots the single-view video sequence, each pixel in the current key frame image and the current associated frame image is projected from a two-dimensional image to a three-dimensional world coordinate system to obtain key frame three-dimensional point cloud data and associated frame three-dimensional point cloud data; the key frame three-dimensional point cloud data and the associated frame three-dimensional point cloud data are projected to a virtual viewpoint imaging plane, and a two-dimensional image interpolation method is used to fill holes to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image.

[0068] Specifically, the camera internal parameters of the camera shooting the single-view video sequence, the current key frame image and the corresponding optimized current key frame depth map, the current associated frame image and the corresponding optimized current associated frame depth map are used to project each pixel in the image from the two-dimensional image to the three-dimensional world coordinate system to generate three-dimensional point cloud data. The world coordinate system is freely defined by the user according to actual use needs. Without loss of generality, in this patent, the world coordinate system is defined to coincide with the camera coordinate system of the single-view video. The specific formula is:

[0069] ;

[0070] in, , , is the coordinate of the three-dimensional point in the world coordinate system, is the depth value of the pixel in the two-dimensional image, , is the equivalent focal length of the camera, , is the coordinate of the origin of the image coordinate system in the pixel coordinate system, and u and v are the coordinates of the pixel point in the image coordinate system.

[0071] Next, according to the transformation relationship between the virtual viewpoint and the real viewpoint, the 3D point cloud data is projected onto the imaging plane of the virtual viewpoint to generate a virtual viewpoint image. The specific formula is:

[0072] ;

[0073] in, , is the pixel coordinate in the virtual viewpoint image, , It is the transformation matrix from the virtual viewpoint camera coordinate system to the camera coordinate system of the single viewpoint video.

[0074] Finally, the holes in the image are filled using a two-dimensional image interpolation method to obtain the final virtual viewpoint image.

[0075] Step S14: generating a virtual viewpoint video sequence based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

[0076] In this embodiment, the key frame virtual viewpoint images and the associated frame virtual viewpoint images are merged together in time sequence to generate a virtual viewpoint video sequence.

[0077] In this embodiment, a single-view video sequence is obtained, and adjacent frame images in the single-view video sequence are divided to obtain a current key frame image and a current associated frame image; a current key frame depth map and a current associated frame depth map are determined from the current key frame image and the current associated frame image, respectively, and the current key frame depth map and the current associated frame depth map are optimized to obtain an optimized current key frame depth map and an optimized current associated frame depth map; based on the optimized current key frame depth map and the optimized current associated frame depth map, the current key frame image and the current associated frame image are projected with pixels in a three-dimensional coordinate system and projected with a virtual viewpoint to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image; a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image. The present application divides each frame image in a single-view video sequence to obtain a current key frame image and a current associated frame image, determines the corresponding current key frame depth map and current associated frame depth map, and optimizes the current key frame depth map and the current associated frame depth map to ensure the rationality of the depth information in the single-frame image and the smooth transition of the depth information of the same scene between adjacent frames to obtain an optimized current key frame depth map and an optimized current associated frame depth map, and performs three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, which can effectively improve the generation quality and efficiency of virtual viewpoint images and bring greater visual impact and appeal to the audience. Finally, a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image, thereby realizing efficient, high-quality and low-cost conversion from single-viewpoint video to multi-viewpoint video.

[0078] See also Figure 2 As shown, the embodiment of the present invention discloses a method for generating a virtual viewpoint video sequence, which may specifically include:

[0079] Step S21: obtaining a single-view video sequence, and dividing adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image.

[0080] Step S22: Determine the current key frame depth map and the current associated frame depth map from the current key frame image and the current associated frame image respectively, perform edge detection and connected domain identification on the current key frame image to obtain each segmented area, map each segmented area to the current key frame depth map, and perform histogram statistics and low-frequency mutation depth value correction processing on the depth values ​​corresponding to each segmented area in the mapped current key frame depth map to obtain the processed current key frame depth map, and perform median filtering on the depth values ​​corresponding to each segmented area in the processed current key frame depth map to obtain the optimized current key frame depth map.

[0081] In this embodiment, the optimized method for generating the depth map of the current key frame is: using deep learning technology to obtain the depth map of the single-view image, and optimizing the accuracy of the depth region division and depth value in the depth map based on the image content. The deep learning model can perform depth estimation on the single-view image to obtain a high-quality depth map. At present, there are many neural networks that can be used for depth estimation, such as VGG16, Resnet (Residual Network), etc. The Midas model used in this embodiment is a lightweight deep learning model launched by Intel Labs. The model focuses on real-time distance and scale estimation tasks. Using a single image, the Midas model can calculate the distance of the scene in the scene.

[0082] Specifically, edge detection and connected domain recognition are performed on the current key frame image to obtain each segmented area, so as to achieve effective segmentation of each scene in the image; then each segmented area is mapped to the current key frame depth map, and the depth value corresponding to each segmented area in the mapped current key frame depth map is histogram-statisticed, and the low-frequency mutation depth value that appears is corrected to the high-frequency depth value of the area to obtain the processed current key frame depth map; finally, the depth value corresponding to each segmented area in the processed current key frame depth map is median-filtered to achieve a smooth transition of depth information. For the segmented areas where the overall depth information has problems, manual intervention is used to correct them to obtain the optimized current key frame depth map.

[0083] Step S23: mapping each segmented area after edge detection and connected domain identification of the current associated frame image to the current associated frame depth map, and performing histogram statistics, low-frequency mutation depth value correction processing, and median filtering on the depth values ​​of each segmented area in the mapped current associated frame depth map to obtain the current associated frame depth map after median filtering, and obtaining a previous frame image adjacent to the current associated frame image; the previous frame image includes a previous key frame image or a previous associated frame image, determining each associated frame depth image area from the current associated frame depth map, and determining each image area corresponding to each associated frame depth image area from the current associated frame image, using a block search method to match each image area with the previous frame image to obtain corresponding each previous frame image area, determining each previous frame depth image area of ​​the previous frame depth map of the previous frame image according to each previous frame image area, and determining the optimized current associated frame depth map based on all the previous frame depth image areas and all the associated frame depth image areas.

[0084] In this embodiment, the method for optimizing the depth map of the current key frame is similar to the method for optimizing the depth map of the current associated frame, and the similarities are: edge detection and connected domain identification are also required, and then each segmented area after edge detection and connected domain identification of the current associated frame image is mapped to the depth map of the current associated frame, and the depth values ​​of each segmented area in the mapped current associated frame depth map are respectively subjected to histogram statistics and low-frequency mutation depth value correction processing and median filtering to obtain the current associated frame depth map after median filtering. The difference is: in the process of obtaining the optimized current associated frame depth map, the depth map optimization link not only optimizes the accuracy of the depth area division and its depth value in the depth map based on the image content, but also forms a smooth transition with the depth information of the previous frame.

[0085] Specifically, the current associated frame image is as follows Figure 3 As shown in (b), the previous frame image adjacent to the current associated frame image is as follows Figure 3 As shown in (a), the previous frame image includes the previous key frame image or the previous associated frame image; the previous frame depth map of the previous frame image is as follows: Figure 4 As shown in (a), the depth map of the current associated frame image is as follows: Figure 4 (b). Figure 4 (b) Determine the depth image area of ​​each associated frame, and then Figure 3 (b) Determine the Figure 4 (b) The image regions corresponding to the depth image regions of each associated frame are searched using a block search method. Figure 3 (b) The image regions and Figure 3(a) Matching is performed to obtain the corresponding image regions of each previous frame, and then Figure 3 (a) The image regions of each previous frame are determined Figure 3 (a) The depth image area of ​​each previous frame of the previous frame depth map, that is, according to Figure 3 (a) The image regions of each previous frame are determined Figure 4 (a) Determine the depth image area of ​​each previous frame Figure 4 (a) The depth value of the depth image area of ​​the previous frame and the corresponding Figure 4 (b) Whether the depth values ​​of the depth image area of ​​the associated frame are consistent, if not, then determine Figure 4 (a) The depth value of the depth image area of ​​the previous frame and the corresponding Figure 4 (b) Whether the change between the depth values ​​of the depth image area of ​​the associated frame meets the normal change condition. If not, calculate Figure 4 (a) is the median depth value of the depth image area of ​​the previous frame, and Figure 4 All depth values ​​of the depth image region of the associated frame in (b) are replaced by Figure 4 (a) The median depth value of the depth image area of ​​the previous frame, and then continue to Figure 4 Other previous frame depth image regions in (a) and their corresponding Figure 4 (b) The depth value of the associated frame depth image area is judged. Figure 4 All previous frame depth image regions in (a) and Figure 4 After all the depth values ​​of all the depth image areas of the associated frame in (b) are determined, an optimized current associated frame depth map is generated.

[0086] This application generates an optimized current associated frame depth map based on a depth map optimization algorithm based on motion estimation. The process of the depth map optimization algorithm based on motion estimation is as follows: Figure 5As shown: the depth map of the current associated frame is processed layer by layer in the order of depth values ​​from small to large. First, the coordinate set P of all pixel points with a depth value of i in the depth map of the current associated frame is extracted (that is, the depth image areas of each associated frame with a depth value of i are determined from the depth map of the current associated frame); then, in the current associated frame image, with the coordinate point in the set P as the center (the image areas corresponding to the depth image areas of each associated frame are determined from the current associated frame image), a block search method is used to intercept image blocks within a certain width as feature data, and the feature data (image blocks) are used to match the corresponding area of ​​the previous frame image to obtain the pixel points in the previous frame image when the cost function takes the minimum value. Coordinates, save the coordinate value in the pixel coordinate set Q (that is, obtain the corresponding previous frame image area); finally, in the previous frame depth map of the previous frame image, extract all depth values ​​of the coordinate set Q position, and save the median value, which is the depth value of the scene with depth value i in the current frame image in the previous frame image. If the shooting depth of the scene in the previous and next frames of images has not changed, the two values ​​are consistent and no processing is performed. If the two values ​​are inconsistent, it is necessary to determine whether the change between the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area meets the normal change condition, so there are the following two situations for processing:

[0087] (1) If the shooting depth of the scene in two adjacent frames does not change (i.e., it does not meet the normal change condition), then all the depth values ​​of the depth image area of ​​the associated frame are replaced with the median of the depth values ​​of the corresponding area of ​​the previous frame depth image;

[0088] (2) If the shooting depth of the scene in two adjacent frames changes (i.e. meets the normal change conditions), no processing is performed.

[0089] Finally, it is determined whether the current number of loops i is greater than a preset loop threshold, taking 255 as an example. If not, the extraction process of the coordinate set P is repeated. If greater, the optimization of the depth map of the current associated frame is completed.

[0090] Step S24: Based on the optimized current key frame depth map and the optimized current associated frame depth map, perform three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image.

[0091] Step S25: Generate a virtual viewpoint video sequence based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

[0092] The specific process of this application is as follows Figure 6As shown, a single-view video sequence is obtained, and adjacent frame images in the single-view video sequence are divided according to the shooting depth of field to obtain a current key frame image and a current associated frame image, a deep learning method is used to obtain a depth map of the current key frame image and a depth map of the current associated frame image, and the depth map of the current key frame image and the depth map of the current associated frame image are optimized to obtain an optimized current key frame depth map and an optimized current associated frame depth map, and based on the optimized current key frame depth map and the optimized current associated frame depth map, the current key frame image and the current associated frame image are projected with pixels in a three-dimensional coordinate system and projected with a virtual viewpoint to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, and a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image. This application is compatible with all single-viewpoint videos, enriching the display content of the three-dimensional display system to the greatest extent. The software operation is easy and ordinary technicians can operate it after simple training. Compared with the "3D modeling + rendering" video production method, although the video quality is reduced, the video conversion efficiency and cost advantages are very obvious. Compared with the high modeling and rendering costs, this method will become the current optimal stereoscopic video generation solution.

[0093] This patent proposes a virtual viewpoint video sequence generation system and method based on deep learning and motion estimation. The input is an ordinary single-viewpoint video sequence. Each frame of the single-viewpoint video sequence is segmented and depth information is set by artificial intelligence. In order to ensure the rationality of the depth information in the single-frame image and the smooth transition of the depth information of the same scene between adjacent frames, this patent proposes a depth map optimization method based on content and motion estimation, which effectively improves the generation quality of virtual viewpoint images and brings greater visual impact and appeal to the audience. At the same time, it can provide multiple parameter input interfaces for video producers, which not only makes the generated video source compatible with different display platforms, but also gives video producers a larger creative space. Producers can freely set the depth and position relationship of the scene in stereoscopic display according to their own ideas. This application can realize the efficient, high-quality and low-cost conversion of single-viewpoint video to multi-viewpoint video, and enrich the display content of the three-dimensional display system to the greatest extent.

[0094] In this embodiment, a single-view video sequence is obtained, and adjacent frame images in the single-view video sequence are divided to obtain a current key frame image and a current associated frame image; a current key frame depth map and a current associated frame depth map are determined from the current key frame image and the current associated frame image, respectively, and the current key frame depth map and the current associated frame depth map are optimized to obtain an optimized current key frame depth map and an optimized current associated frame depth map; based on the optimized current key frame depth map and the optimized current associated frame depth map, the current key frame image and the current associated frame image are projected with pixels in a three-dimensional coordinate system and projected with a virtual viewpoint to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image; a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image. The present application divides each frame image in a single-view video sequence to obtain a current key frame image and a current associated frame image, determines the corresponding current key frame depth map and current associated frame depth map, and optimizes the current key frame depth map and the current associated frame depth map to ensure the rationality of the depth information in the single-frame image and the smooth transition of the depth information of the same scene between adjacent frames to obtain an optimized current key frame depth map and an optimized current associated frame depth map, and performs three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, which can effectively improve the generation quality and efficiency of virtual viewpoint images and bring greater visual impact and appeal to the audience. Finally, a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image, thereby realizing efficient, high-quality and low-cost conversion from single-viewpoint video to multi-viewpoint video.

[0095] See also Figure 7 As shown, the embodiment of the present invention discloses a virtual viewpoint video sequence generation device, which may specifically include:

[0096] An image division module 11 is used to obtain a single-view video sequence and divide adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image;

[0097] The optimization processing module 12 is used to determine a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image, respectively, and optimize the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map;

[0098] The projection module 13 is used to perform three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image based on the optimized current key frame depth map and the optimized current associated frame depth map to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image;

[0099] The virtual viewpoint video sequence generating module 14 is used to generate a virtual viewpoint video sequence based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

[0100] In this embodiment, a single-view video sequence is obtained, and adjacent frame images in the single-view video sequence are divided to obtain a current key frame image and a current associated frame image; a current key frame depth map and a current associated frame depth map are determined from the current key frame image and the current associated frame image, respectively, and the current key frame depth map and the current associated frame depth map are optimized to obtain an optimized current key frame depth map and an optimized current associated frame depth map; based on the optimized current key frame depth map and the optimized current associated frame depth map, the current key frame image and the current associated frame image are projected with pixels in a three-dimensional coordinate system and projected with a virtual viewpoint to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image; a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image. The present application divides each frame image in a single-view video sequence to obtain a current key frame image and a current associated frame image, determines the corresponding current key frame depth map and current associated frame depth map, and optimizes the current key frame depth map and the current associated frame depth map to ensure the rationality of the depth information in the single-frame image and the smooth transition of the depth information of the same scene between adjacent frames to obtain an optimized current key frame depth map and an optimized current associated frame depth map, and performs three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image, which can effectively improve the generation quality and efficiency of virtual viewpoint images and bring greater visual impact and appeal to the audience. Finally, a virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image, thereby realizing efficient, high-quality and low-cost conversion from single-viewpoint video to multi-viewpoint video.

[0101] In some specific embodiments, the image segmentation module 11 may specifically include:

[0102] A screening module, used for screening a current frame image and a previous frame image adjacent to the current frame image from the single-view video sequence;

[0103] A shooting depth of field consistency judgment module is used to judge whether the shooting depth of field between the current frame image and the previous frame image is consistent;

[0104] A current key frame image determination module, configured to use the current frame image as the current key frame image if the shooting depth of field between the current frame image and the previous frame image is inconsistent;

[0105] The current associated frame image determining module is used to use the current frame image as the current associated frame image if the shooting depth of field between the current frame image and the previous frame image is consistent.

[0106] In some specific embodiments, the image segmentation module 11 may specifically include:

[0107] A first distance calculation module, used for determining a first target near-view object and a first target far-view object from the current frame image, and calculating a first distance between the first target near-view object and the first target far-view object in a camera optical axis direction;

[0108] A second distance calculation module is used to determine a second target near-view object and a second target far-view object from the previous frame image adjacent to the current frame image, and calculate a second distance between the second target near-view object and the second target far-view object in the direction of the camera optical axis;

[0109] An image determination module is used to determine whether the first distance is consistent with the second distance. If the first distance is inconsistent with the second distance, the current frame image is used as the current key frame image; if the first distance is consistent with the second distance, the current frame image is used as the current associated frame image.

[0110] In some specific embodiments, the optimization processing module 12 may specifically include:

[0111] A detection and recognition module, used for performing edge detection and connected domain recognition on the current key frame image to obtain each segmented area;

[0112] A statistics and processing module, used for mapping each of the segmented areas to the current key frame depth map, and performing histogram statistics and low-frequency mutation depth value correction processing on the depth values ​​corresponding to each of the segmented areas in the mapped current key frame depth map, so as to obtain the processed current key frame depth map;

[0113] The median filtering module is used to perform median filtering on the depth values ​​corresponding to each segmented area in the processed current key frame depth map to obtain the optimized current key frame depth map.

[0114] In some specific embodiments, the optimization processing module 12 may specifically include:

[0115] The statistics, processing and median filtering module is used to map each segmented area after edge detection and connected domain identification of the current associated frame image to the current associated frame depth map, and perform histogram statistics, low-frequency mutation depth value correction processing and median filtering on the depth values ​​of each segmented area in the mapped current associated frame depth map, so as to obtain the current associated frame depth map after median filtering;

[0116] A previous frame image acquisition module, used to acquire a previous frame image adjacent to the current associated frame image; the previous frame image includes a previous key frame image or a previous associated frame image;

[0117] A matching module, used to determine each associated frame depth image area from the current associated frame depth map, and determine each image area corresponding to each associated frame depth image area from the current associated frame image, and use a block search method to match each image area with the previous frame image to obtain corresponding previous frame image areas;

[0118] The optimized current associated frame depth map generation module is used to determine each previous frame depth image area of ​​the previous frame depth map of the previous frame image according to each previous frame image area, and determine the optimized current associated frame depth map based on all the previous frame depth image areas and all the associated frame depth image areas.

[0119] In some specific embodiments, the optimization processing module 12 may specifically include:

[0120] A depth value determination module, used to determine whether the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area are consistent;

[0121] A normal change judgment module, used for judging whether the change between the depth value of the previous frame depth image area and the depth value of the corresponding associated frame depth image area meets the normal change condition if the depth value of the previous frame depth image area and the depth value of the corresponding associated frame depth image area are inconsistent;

[0122] A depth value median replacement module is used to calculate the depth value median of the depth image area of ​​the previous frame if the change between the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area does not meet the normal change condition, and replace all the depth values ​​of the associated frame depth image area with the depth value median;

[0123] A depth value determination repeating execution module, used to continue to perform depth value determination on other depth image areas of the previous frame and corresponding depth image areas of the associated frame;

[0124] The optimized current associated frame depth map generation module is used to generate the optimized current associated frame depth map after all the depth values ​​of the previous frame depth image areas and all the associated frame depth image areas are judged.

[0125] In some specific embodiments, the projection module 13 may specifically include:

[0126] A three-dimensional point cloud projection module is used to project each pixel in the current key frame image and the current associated frame image from a two-dimensional image to a three-dimensional world coordinate system based on the depth value of each pixel in the optimized current key frame depth map and the optimized current associated frame depth map and the camera internal parameters of the camera shooting the single-view video sequence, so as to obtain the key frame three-dimensional point cloud data and the associated frame three-dimensional point cloud data;

[0127] The virtual plane projection module is used to project the key frame 3D point cloud data and the associated frame 3D point cloud data onto a virtual viewpoint imaging plane, and use a 2D image interpolation method to fill holes to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image.

[0128] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the virtual viewpoint video sequence generation method performed by the electronic device disclosed in any of the aforementioned embodiments.

[0129] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0130] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 221, a computer program 222 and data 223, etc. The storage method can be temporary storage or permanent storage.

[0131] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, so as to realize the operation and processing of the data 223 in the memory 22 by the processor 21, which can be Windows, Unix, Linux, etc. In addition to including a computer program that can be used to complete the virtual viewpoint video sequence generation method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks. In addition to including data transmitted from an external device received by the virtual viewpoint video sequence generation device, the data 223 can also include data collected by its own input and output interface 25, etc.

[0132] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0133] Furthermore, an embodiment of the present application also discloses a computer-readable storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the virtual viewpoint video sequence generation method disclosed in any of the aforementioned embodiments are implemented.

[0134] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0135] The above is a detailed introduction to a virtual viewpoint video sequence generation method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating a virtual viewpoint video sequence, characterized in that: include: Acquire a single-view video sequence, and divide adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image; Determining a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image respectively, and optimizing the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map; Based on the optimized current key frame depth map and the optimized current associated frame depth map, performing three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image; A virtual viewpoint video sequence is generated based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

2. The method for generating a virtual viewpoint video sequence according to claim 1, characterized in that: The step of dividing adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image includes: Filtering a current frame image and a previous frame image adjacent to the current frame image from the single-view video sequence; Determine whether the shooting depth of field between the current frame image and the previous frame image is consistent; If the shooting depths of field between the current frame image and the previous frame image are inconsistent, taking the current frame image as the current key frame image; If the shooting depth of field between the current frame image and the previous frame image is consistent, the current frame image is used as the current associated frame image.

3. The method for generating a virtual viewpoint video sequence according to claim 2, characterized in that: Determining whether the depth of field of the current frame image is consistent with that of the previous frame image; if the depth of field of the current frame image is inconsistent with that of the previous frame image, taking the current frame image as the current key frame image; If the shooting depth of field between the current frame image and the previous frame image is consistent, taking the current frame image as the current associated frame image includes: Determine a first target near-view object and a first target far-view object from the current frame image, and calculate a first distance between the first target near-view object and the first target far-view object in a camera optical axis direction; Determine a second target near-view object and a second target far-view object from the previous frame image adjacent to the current frame image, and calculate a second distance between the second target near-view object and the second target far-view object in the direction of the camera optical axis; Determine whether the first distance is consistent with the second distance. If the first distance is inconsistent with the second distance, use the current frame image as the current key frame image. If the first distance is consistent with the second distance, use the current frame image as the current associated frame image.

4. The method for generating a virtual viewpoint video sequence according to claim 1, characterized in that: Optimizing the current key frame depth map to obtain an optimized current key frame depth map includes: Performing edge detection and connected domain identification on the current key frame image to obtain each segmented area; Mapping each of the segmented areas to the current key frame depth map, and performing histogram statistics and low-frequency mutation depth value correction processing on the depth values ​​corresponding to each of the segmented areas in the mapped current key frame depth map, so as to obtain the processed current key frame depth map; The depth values ​​corresponding to the segmented areas in the processed current key frame depth map are respectively subjected to median filtering to obtain an optimized current key frame depth map.

5. The method for generating a virtual viewpoint video sequence according to claim 1, characterized in that: Optimizing the current associated frame depth map to obtain an optimized current associated frame depth map includes: Mapping each segmented area after edge detection and connected domain identification of the current associated frame image to the current associated frame depth map, and performing histogram statistics, low-frequency mutation depth value correction processing, and median filtering on the depth values ​​of each segmented area in the mapped current associated frame depth map, so as to obtain the current associated frame depth map after median filtering; Acquire a previous frame image adjacent to the current associated frame image; the previous frame image includes a previous key frame image or a previous associated frame image; Determine each associated frame depth image region from the current associated frame depth map, and determine each image region corresponding to each associated frame depth image region from the current associated frame image, and match each image region with the previous frame image using a block search method to obtain corresponding previous frame image regions; Determine each previous frame depth image area of ​​a previous frame depth map of the previous frame image according to each previous frame image area, and determine the optimized current associated frame depth map based on all the previous frame depth image areas and all the associated frame depth image areas.

6. The method for generating a virtual viewpoint video sequence according to claim 5, characterized in that: The step of determining the optimized current associated frame depth map based on all the previous frame depth image regions and all the associated frame depth image regions comprises: Determine whether the depth value of the previous frame depth image area is consistent with the corresponding depth value of the associated frame depth image area; If the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area are inconsistent, determining whether a change between the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area meets a normal change condition; If the change between the depth value of the previous frame depth image area and the corresponding depth value of the associated frame depth image area does not meet the normal change condition, then calculating the median of the depth values ​​of the previous frame depth image area, and replacing all the depth values ​​of the associated frame depth image area with the median of the depth values; Continue to determine the depth values ​​of other depth image areas of the previous frame and corresponding depth image areas of the associated frame; When the depth values ​​of all the previous frame depth image regions and all the associated frame depth image regions are determined, an optimized current associated frame depth map is generated.

7. The method for generating a virtual viewpoint video sequence according to any one of claims 1 to 6, characterized in that: The method of performing three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image based on the optimized current key frame depth map and the optimized current associated frame depth map to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image includes: Based on the depth value of each pixel in the optimized current key frame depth map and the optimized current associated frame depth map and the camera internal parameters of the camera shooting the single-view video sequence, each pixel in the current key frame image and the current associated frame image is projected from the two-dimensional image to the three-dimensional world coordinate system to obtain the key frame three-dimensional point cloud data and the associated frame three-dimensional point cloud data; The key frame three-dimensional point cloud data and the associated frame three-dimensional point cloud data are projected onto a virtual viewpoint imaging plane, and two-dimensional image interpolation is used to fill holes to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image.

8. A virtual viewpoint video sequence generation device, characterized in that: include: An image division module, used for acquiring a single-view video sequence, and dividing adjacent frame images in the single-view video sequence to obtain a current key frame image and a current associated frame image; An optimization processing module, used to determine a current key frame depth map and a current associated frame depth map from the current key frame image and the current associated frame image, respectively, and optimize the current key frame depth map and the current associated frame depth map to obtain an optimized current key frame depth map and an optimized current associated frame depth map; A projection module, configured to perform three-dimensional coordinate system pixel point projection and virtual viewpoint projection on the current key frame image and the current associated frame image based on the optimized current key frame depth map and the optimized current associated frame depth map, so as to obtain a key frame virtual viewpoint image and an associated frame virtual viewpoint image; The virtual viewpoint video sequence generating module is used to generate a virtual viewpoint video sequence based on the key frame virtual viewpoint image and the associated frame virtual viewpoint image.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the method for generating a virtual viewpoint video sequence according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program; wherein, when the computer program is executed by a processor, the method for generating a virtual viewpoint video sequence according to any one of claims 1 to 7 is implemented.

Images