Real-time any-view-angle and free-view-angle video generation method and system

By calculating bidirectional optical flow and embedded optical flow for forward distortion processing, a user-defined virtual view is generated, which solves the problems of response delay and transmission bandwidth in medium and high concurrency scenarios in the prior art, and realizes an efficient and lightweight free viewing angle Video generation supports multi-user personalized services.

CN119011871AActive Publication Date: 2024-11-22SHANGHAI JIAOTONG UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411010941.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-26
Publication Date
2024-11-22
Estimated Expiration
2044-07-26

AI Technical Summary

Technical Problem

Existing free-view systems are difficult to cope with the problems of high response latency and high transmission bandwidth in high concurrency scenarios, especially inadequate processing capabilities on low-end user terminals, resulting in poor user experience.

Method used

By obtaining multi-frame color images of different view angles of the same calibrator, using dense optical flow field estimation network and optical flow embedder to calculate bidirectional optical flow and embedded optical flow, perform forward distortion processing and fusion processing, and generate user-defined virtual view , reduce computing resource requirements and optimize transmission bandwidth.

Benefits of technology

It realizes the generation of high-quality virtual view of any angle with fewer computing resources, decouples the number of access users and edge server load, supports multi-user personalized free viewing video services, and improves the efficiency and user experience of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011871A_ABST
    Figure CN119011871A_ABST
Patent Text Reader

Abstract

The invention provides a real-time arbitrary view angle and free view angle video generation method and system, and the method comprises the steps: obtaining multiple frames of color images of different view angles of a same calibration object; inputting the multiple frames of color images into a preset dense optical flow field estimation network, and determining a bidirectional optical flow; inputting the multi-frame color image and the bidirectional optical flow into an optical flow embedder, and determining an embedded optical flow and a mask matrix; determining a first optical flow and a second optical flow according to a third visual angle input by the user and the embedded optical flow; and performing forward distortion processing on the multi-frame color image according to the first optical flow and the second optical flow, and determining a complete virtual view at the third visual angle. According to the technical scheme, real-time arbitrary view angle generation is achieved, the lightweight class is high, efficiency is high, computing resource consumption is reduced, and user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of immersive self-media technology, and in particular to a real-time arbitrary-viewing-angle and free-viewing-angle video generation method and system. Background Art

[0002] As a representative of interactive immersive media content, free viewpoint technology allows viewers to choose any direction and viewpoint to watch the visual scene through interaction according to their own needs, without being restricted by the position of the camera used for shooting. Compared with the "inside-out" VR technology, the "outside-in" interactive form of free viewpoint technology gives users a stronger three-dimensional experience and more direct interaction. Free viewpoint video is a new type of immersive media with strong interactive characteristics. Its appeal goes beyond typical application scenarios such as cloud games and remote virtual reality, and is expected to change the way we consume visual content.

[0003] Generally speaking, a free view system consists of four parts: a multi-view acquisition system, free view content production, encoding transmission, and client. The multi-view acquisition system aims to provide multi-angle and multi-directional video source information for the free view system. However, due to the limitations of hardware cost and data volume, the acquisition system can only be limited to a sparse and limited number of camera shots. Virtual view synthesis technology aims to obtain other uncollected viewpoint information from limited view information. The DIBR (Depth Image-Based Rendering) method is the most commonly used view synthesis method in the free view system. However, due to the introduction of occlusion and black holes in the three-dimensional image distortion, the synthesis results are often unsatisfactory. In addition, the acquisition of accurate depth maps also faces great challenges.

[0004] Free viewpoint systems can generally be divided into two models: centralized and distributed.

[0005] In the centralized model, the viewpoints required by different users are synthesized on the server side. Some existing real-time viewpoint synthesis methods require sufficient computing resources, so one server can only serve a limited number of user terminals. As the number of connected users increases, the number of servers also needs to increase accordingly. This model has difficulty coping with high concurrency scenarios and will cause additional response delays during the interaction process.

[0006] The distributed model is able to serve multiple users simultaneously because it performs the view synthesis process on the client side. However, the Multiview-VideoPlus-Depth (MVD) representation required for view synthesis needs to be transmitted to the user, which may result in high transmission bandwidth. In addition, the view synthesis method requires a lot of processing power, which is not friendly to some low-end user terminals. Summary of the invention

[0007] In view of the defects in the prior art, the purpose of the present invention is to provide a real-time arbitrary perspective generation method and system, and a free perspective video method and system.

[0008] In order to achieve the above object, according to a first aspect of the present disclosure, a real-time arbitrary perspective generation method is provided, comprising:

[0009] Acquire multiple frames of color images from different perspectives of the same calibration object, wherein the multiple frames of color images from different perspectives of the same calibration object include a first color texture image from a first perspective and a second color texture image from a second perspective;

[0010] Inputting the first color texture image and the second color texture image into a preset dense optical flow field estimation network to determine a bidirectional optical flow;

[0011] Inputting the first color texture image, the second color texture image and the bidirectional optical flow into an optical flow embedder to determine a preset logarithmic embedded optical flow and a mask matrix;

[0012] Determine, according to the third perspective input by the user and the preset logarithm of embedded optical flows, a first preset number of first optical flows from the first color texture image to a virtual view of the third perspective, and a first preset number of second optical flows from the second color texture image to a virtual view of the third perspective;

[0013] The first color texture image and the second color texture image are forward-warped according to the first preset number of first optical flows and the first preset number of second optical flows, respectively, and the images after the forward warping are fused according to the mask matrix to determine a complete virtual view at the third perspective.

[0014] Optionally, inputting the first color texture image, the second color texture image, and the bidirectional optical flow into an optical flow embedder to determine a preset logarithmic embedded optical flow and a mask matrix includes:

[0015] Performing feature extraction processing on the first color texture image and the second color texture image to determine image features of the color images of the first viewing angle and the second viewing angle at different resolution scales;

[0016] Performing a first distortion process on the image features of the color images of the first viewing angle and the second viewing angle according to the initial optical flow, and determining the image features after the first distortion process;

[0017] Aligning and splicing the image features after the first distortion processing and the original image features at the distortion end point in sequence to determine spliced ​​image features;

[0018] Performing encoding and decoding processing on the stitched image features in sequence to determine a second preset number of optical flow residuals, wherein the value of the second preset number is the same as the value of the preset logarithm;

[0019] The initial optical flow is optimized according to the second preset number of optical flow residuals to determine a preset number of embedded optical flows and a mask matrix.

[0020] Optionally, the determining, according to the third perspective input by the user and the preset logarithm of embedded optical flows, a first preset number of first optical flows from the first color texture image to the virtual view of the third perspective, and a first preset number of second optical flows from the second color texture image to the virtual view of the third perspective, comprises:

[0021]

[0022] in, represents the first optical flow, p represents the third viewing angle, represents the embedded optical flow from the first view to the second view, represents the second optical flow, represents the embedded optical flow from the second view to the first view.

[0023] Optionally, the fusion process includes a first fusion process and a second fusion process;

[0024] The method of using the first preset number of first optical flows and the first preset number of second optical flows to perform forward warping on the first color texture image and the second color texture image, and using the mask matrix to fuse the images after the forward warping process to determine the complete virtual view at the third perspective includes:

[0025] Using a forward warping operator to forward warp the first color texture image to the third viewing angle according to the first preset number of first optical flows, and determining a first preset number of first-side virtual third color texture images;

[0026] Using the forward warping operator to forward warp the second color texture image to the third viewing angle according to the first preset number of second optical flows, to determine a first preset number of second-side virtual third color texture images;

[0027] Perform the first fusion process on the first preset number of first-side virtual third color texture images and the first preset number of second-side virtual third color texture images respectively, and determine a first-side fusion image and a second-side fusion image after the first fusion process;

[0028] The first side fusion image and the second side fusion image are subjected to the second fusion process according to the mask matrix to determine a complete virtual view at the third viewing angle.

[0029] Optionally, performing the second fusion process on the first side fusion image and the second side fusion image according to the mask matrix to determine the complete virtual view at the third viewing angle includes:

[0030] I p =M⊙I left→p +(1-M)⊙I right→p

[0031]

[0032]

[0033] Among them, I p represents the complete virtual image at the third viewing angle, M represents the mask matrix, and I left→p represents the first side fusion image, I right→p represents the second side fused image, represents the first side virtual third color texture image, represents the second side virtual third color texture image, and k represents the first preset number.

[0034] According to a second aspect of the present disclosure, a real-time arbitrary perspective generation system is provided, comprising:

[0035] An acquisition module is used to acquire multiple frames of color images of the same calibration object from different perspectives, wherein the multiple frames of color images of the same calibration object from different perspectives include a first color texture image from a first perspective and a second color texture image from a second perspective;

[0036] A first determination module, configured to input the first color texture image and the second color texture image into a preset dense optical flow field estimation network to determine a bidirectional optical flow;

[0037] A second determination module is used to input the first color texture image, the second color texture image and the bidirectional optical flow into an optical flow embedder to determine an embedded optical flow and a mask matrix of a preset logarithm;

[0038] A third determining module is used to determine a first preset number of first optical flows from the first color texture image to the virtual view of the third perspective and a first preset number of second optical flows from the second color texture image to the virtual view of the third perspective according to a third perspective input by a user and the preset logarithm of embedded optical flows;

[0039] a fourth determination module, configured to perform forward warping on the first color texture image and the second color texture image according to the first preset number of first optical flows and the first preset number of second optical flows, and to fuse the images after the forward warping according to the mask matrix to determine a complete virtual view at the third perspective.

[0040] According to a third aspect of the present disclosure, a free-viewpoint video generation method is provided, comprising:

[0041] Acquire multiple color images of the same calibration object from different perspectives;

[0042] Combining the multiple color image frames in pairs;

[0043] Obtaining in real time a preset logarithmic embedded optical flow and a mask matrix for each of the two-by-two combined multi-frame color images according to the method provided by the first aspect of the present disclosure;

[0044] Adaptively splicing the multiple color image frames combined in pairs and the preset logarithmic embedded optical flows and mask matrices thereof to form multiple synthetic images in the spatial domain;

[0045] Encoding the multiple composite images, dividing them into video segments in the time domain, and transmitting them through the HLS protocol;

[0046] Different clients interactively select target synthetic images and download them, and synthesize a complete virtual view at the perspective viewed by the user in real time according to the method provided in the first aspect of the present disclosure.

[0047] Optionally, the step of adaptively splicing the multiple color image frames combined in pairs and the preset logarithmic embedded optical flows and mask matrices to form multiple synthetic images in the spatial domain includes:

[0048] Perform channel dimension synthesis processing on the preset logarithmic embedded optical flow and mask matrix corresponding to each of the two-by-two multi-frame color images to determine a three-channel matrix corresponding to each of the two-by-two multi-frame color images;

[0049] Performing spatial domain synthesis processing on each of the two-by-two combination of the multi-frame color images and the three-channel matrix corresponding to each of the two-by-two combination of the multi-frame color images to determine a synthesized image;

[0050] The multiple frames of color images from different viewing angles and all pairs of embedded optical flows and mask matrices are subjected to resolution reduction processing to form a total composite image.

[0051] Optionally, the different clients interactively select and download target synthetic images, and synthesize a complete virtual view at the viewing angle of the user in real time according to the method provided by the first aspect of the present disclosure, including:

[0052] Using a global viewing angle index, download the synthetic image and the total synthetic image corresponding to the viewing angle viewed by the user;

[0053] Within a video slice time, the viewing angle of the user falls within the downloaded synthetic image, and the synthetic image is used to synthesize a complete virtual view at the viewing angle of the user in real time according to the method provided in the first aspect of the present disclosure;

[0054] Within a video slice time, the viewing angle of the user falls outside the downloaded synthetic image, and the total synthetic image is used to synthesize a complete virtual view at the viewing angle of the user in real time according to the method provided in the first aspect of the present disclosure;

[0055] After a video slicing time, the user downloads the complete virtual video and the total synthesized image at the viewing angle.

[0056] According to a fourth aspect of the present disclosure, a free-viewpoint video generation system is provided, comprising:

[0057] The acquisition server module is used to obtain multiple frames of color images of the same calibration object from different perspectives;

[0058] A cloud processing and content distribution network module, used for combining the multiple frames of color images in pairs and distributing them to the edge server module through a content distribution network;

[0059] The edge server module is used to obtain the preset logarithmic embedded optical flow and mask matrix of each of the two-by-two combined multi-frame color images in real time; adaptively splice each of the two-by-two combined multi-frame color images and the preset logarithmic embedded optical flow and mask matrix to form a plurality of composite images in the spatial domain; encode the plurality of composite images, divide them into video segments in the time domain, and transmit them through the HLS protocol;

[0060] The client module is used for different clients to interactively select target synthetic images and download them, so as to synthesize a complete virtual view at the user's viewing angle in real time.

[0061] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:

[0062] Through the above technical solution, after obtaining color images of different perspectives facing the same calibration object, the collected multiple frames of images are combined in pairs, and virtual intermediate perspective views of any perspective between the two combined images are interpolated in real time. It is lightweight and efficient, and can use fewer computing resources to interpolate high-quality virtual views of any perspective in real time. It is also easy to deploy on the edge server or client, which is conducive to the development of free perspective video generation systems.

[0063] The free-viewpoint video generation method provided by the present invention is a free-viewpoint video generation method for multiple users. The free-viewpoint video generation system built based on this method decouples the number of access users and the edge server load, so that a single edge server can provide personalized free-viewpoint video services for multiple users. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0065] Figure 1 The figure is a flow chart of a method for generating arbitrary viewing angles in real time according to an exemplary embodiment.

[0066] Figure 2 The invention is a flowchart of a method for determining a preset logarithmic embedded optical flow and a mask matrix according to an exemplary embodiment.

[0067] Figure 3 The figure is a schematic diagram showing a process of real-time arbitrary perspective generation according to an exemplary embodiment.

[0068] Figure 4 The present invention is a flow chart of a method for determining a complete virtual view at a third perspective according to an exemplary embodiment.

[0069] Figure 5 The figure is a flow chart of a method for generating a free-viewing-angle video according to an exemplary embodiment.

[0070] Figure 6 The figure is a schematic diagram showing the arrangement of an image collector according to an exemplary embodiment.

[0071] Figure 7 The invention is a block diagram of a real-time arbitrary perspective generation system according to an exemplary embodiment.

[0072] Figure 8 The diagram is a schematic diagram of the architecture of a free-viewpoint video generation system according to an exemplary embodiment. DETAILED DESCRIPTION

[0073] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0074] Figure 1 The figure is a flow chart of a method for generating arbitrary viewing angles in real time according to an exemplary embodiment.

[0075] like Figure 1 As shown, a real-time arbitrary viewing angle generation method includes S11 to S15.

[0076] S11, obtaining multiple frames of color images of the same calibration object from different viewing angles.

[0077] The multiple frames of color images at different angles of the same calibration object include a first color texture image at a first viewing angle and a second color texture image at a second viewing angle.

[0078] Image collectors are arranged around the same calibration object at preset angles and preset radius. The image collectors can be devices such as cameras or video cameras. The baseline between every two image collectors, that is, the distance between the two image collectors, is set to an appropriate length, and the length of the baseline should not be too long.

[0079] The two spatially adjacent image collectors are respectively referred to as the first image collector and the second image collector. In the present disclosure, the image collected by the first image collector is a first color texture image of a first perspective, and the image collected by the second image collector is a second color texture image of a second perspective, and the first color texture image and the second color texture image are collected synchronously.

[0080] In the present disclosure, the first perspective may also be regarded as a left perspective, and the second perspective may also be regarded as a right perspective.

[0081] In a possible embodiment, the image collector may be an industrial camera, the baseline between every two image collectors is set to 40 cm, and the first color texture image and the second color texture image are both in RGB format.

[0082] S12, inputting the first color texture image and the second color texture image into a preset dense optical flow field estimation network to determine the bidirectional optical flow.

[0083] The preset dense optical flow length estimation network is an open source network. The first color texture image and the second color texture image are input into the preset dense optical flow length estimation network. The preset dense optical flow length estimation network, such as the RAFT network, is used to simultaneously calculate the bidirectional pixel-by-pixel offset from the first color texture image to the second color texture image and from the second color texture image to the first color texture image, that is, the bidirectional optical flow, including optical flow F1 and optical flow F2, wherein optical flow F1 and optical flow F2 are both two-dimensional vector matrix images.

[0084] Among them, whether to perform downsampling processing before the first color texture image and the second color texture image are input and whether to perform upsampling processing after the output of the preset dense optical flow length estimation network can be determined according to factors such as computing resources.

[0085] In the present disclosure, the first color texture image and the second color texture image are first downsampled to 1 / 2 of the size of the original image, and then the low-resolution optical flow is calculated through the RAFT network (Recurrent All-Pairs Field Transforms for Optical Flow). Finally, the low-resolution optical flow and the visibility mask are upsampled to the size of the original image to obtain the optical flow F1 and the optical flow F2.

[0086] Through the above technical solution, a preset dense optical flow field estimation network is adopted. On the one hand, its effect is better, and on the other hand, it can make a compromise between performance and speed by adjusting parameters to ensure the real-time performance of the network.

[0087] S13, inputting the first color texture image, the second color texture image and the bidirectional optical flow into an optical flow embedder to determine a preset logarithmic embedded optical flow and a mask matrix.

[0088] The value of the preset logarithm is k, and each pair of embedded optical flows includes an embedded optical flow from the first perspective to the second perspective and an embedded optical flow from the second perspective to the first perspective. The selection of the value k is determined according to the deployment of the application. Specifically, the embedded optical flow and mask matrix of the preset logarithm obtained need to be encoded in the free perspective video system and transmitted over the network. The larger the value of k, the better the quality of the generated virtual perspective image, but at the same time, the amount of encoded data is large, and the occupied network transmission bandwidth will also be higher.

[0089] The mask matrix is ​​used to describe the visibility of the first and second perspectives.

[0090] S14, determining a first preset number of first optical flows from the first color texture image to the virtual view of the third perspective and a first preset number of second optical flows from the second color texture image to the virtual view of the third perspective according to the third perspective input by the user and the preset logarithm of embedded optical flows.

[0091] The value of the preset logarithm of the embedded optical flow is the same as the value of the first preset number.

[0092] Optical flow represents the pixel offset from one image to another, and is represented by a two-dimensional vector, which can represent the horizontal offset or the vertical offset in the image matrix.

[0093] S15, forward warping the first color texture image and the second color texture image according to a first preset number of first optical flows and a first preset number of second optical flows, respectively, and fusing the images after the forward warping according to a mask matrix to determine a complete virtual view at a third perspective.

[0094] The distortion process is the process of adding the image and the pixel offset.

[0095] As an example, there is an overall rightward pixel offset from the first color texture image of the first perspective to the second color texture image of the second perspective, and the distortion process is equivalent to shifting the first color texture image of the first perspective rightward by an offset.

[0096]

[0097] Among them, x1 represents the horizontal coordinate of the pixel of the first color texture image, y1 represents the vertical coordinate of the pixel of the first color texture image, u1 represents the horizontal coordinate of the pixel of the second color texture image, v1 represents the vertical coordinate of the pixel of the second color texture image, and F1 represents the optical flow from the first color texture image to the second color texture image.

[0098] As another example, there is an overall leftward pixel offset from the second color texture image of the second perspective to the first color texture image of the first perspective, and the distortion process is equivalent to shifting the first color texture image of the second perspective to the left by an offset.

[0099]

[0100] Among them, u2 represents the horizontal coordinate of the pixel of the first color texture image, v2 represents the vertical coordinate of the pixel of the first color texture image, x2 represents the horizontal coordinate of the pixel of the second color texture image, y2 represents the vertical coordinate of the pixel of the second color texture image, and F2 represents the optical flow from the second color texture image to the first color texture image.

[0101] In the present disclosure, when performing pixel mapping according to the first optical flow and the second optical flow, that is, performing a warping process, a forward warping process is performed using a forward warping operator σ instead of a reverse warping process.

[0102] By adopting the forward warping operator to perform forward warping processing, optical flow inversion can be avoided, and the image at the third viewing angle is directly acquired by adopting the first color texture image and the first optical flow, thereby improving the efficiency of the virtual view and reducing artifacts.

[0103] As an example, the first color texture image of the first perspective is forward-warped to the target position p, that is, the third perspective: Among them, I left represents the first color texture image, F left→p represents a first optical flow, and σ represents the forward warping operator.

[0104] The process of forward warping the first color texture image of the first perspective to the target position p includes:

[0105] The first color texture image pixel q is mapped to the third viewing angle according to the first optical flow, and the pixel p is bilinearly sampled, that is,

[0106] p=bilinear(q+F left→p [q])

[0107] Wherein, bilinear represents the bilinear sampling operator.

[0108] Bilinear weights:

[0109] u=p-(q+F left→p [q])

[0110] Where u represents the bilinear weight.

[0111] Bilinear Kernel:

[0112] b(u)=max(0,1-|u x |)·max(0,1-|u y |)

[0113] Where b(u) represents the bilinear kernel, u x represents the x-component of the bilinear weight u, u y Represents the y-component of the bilinear weight u.

[0114] Use bilinear weighting to determine the current pixel value:

[0115]

[0116] Among them, I p [p] represents the current pixel value.

[0117] As another example, the second color texture image of the second perspective is forward-warped to the target position p, that is, at the third perspective: Among them, I right represents the second color texture image, F right→p Represents a second optical flow.

[0118] The process of forward-warping the second color texture image of the second perspective to the target position p is the same as the method of forward-warping the first color texture image of the first perspective to the target position p, which will not be described in detail here.

[0119] Thus, the first viewing angle can obtain k groups of images mapped to the third viewing angle, and the second viewing angle can obtain k groups of images mapped to the third viewing angle.

[0120] The mask matrix is ​​then used to fuse the forward-warped images to determine a complete virtual view from a third-person perspective.

[0121] Through the above technical solution, after obtaining color images of different perspectives facing the same calibration object, the collected multiple frames of images are combined in pairs, and virtual intermediate perspective views of any perspective between the two combined images are interpolated in real time. It is lightweight and efficient, and can use fewer computing resources to interpolate high-quality virtual views of any perspective in real time. It is also easy to deploy on the edge server or client, which is conducive to the development of free perspective video generation systems.

[0122] Figure 2 The invention is a flowchart of a method for determining a preset logarithmic embedded optical flow and a mask matrix according to an exemplary embodiment. Figure 3 The figure is a schematic diagram showing a process of real-time arbitrary perspective generation according to an exemplary embodiment.

[0123] like Figure 2 , Figure 3 As shown, in some possible embodiments, the first color texture image, the second color texture image and the bidirectional optical flow are input into an optical flow embedder to determine a preset logarithmic embedded optical flow and a mask matrix, including S21 to S25.

[0124] S21, performing feature extraction processing on the first color texture image and the second color texture image to determine image features of the color images of the first viewing angle and the second viewing angle at different resolution scales.

[0125] Among them, a feature extractor may be used to extract image features of the first color texture image and the second color texture image at different resolution scales.

[0126] S22, performing a first distortion process on the image features of the color images of the first viewing angle and the second viewing angle according to the initial optical flow, and determining image features after the first distortion process.

[0127] S23, aligning and splicing the features of the image after the distortion process and the features of the original image at the distortion end point in sequence to determine the spliced ​​image features.

[0128] Among them, the first distortion process is the distortion process mentioned above, which distorts the image features of one perspective to another perspective through the distortion process and aligns them with the image features of another perspective, splices the aligned image features, and determines the image features after splicing.

[0129] S24, encoding and decoding the spliced ​​image features in sequence to determine a second preset number of optical flow residuals.

[0130] The value of the second preset number is the same as the value of the preset logarithm.

[0131] like Figure 3 As shown, in a possible embodiment, the concatenated image features are input into the Encoder layer for optical flow encoding, and then passed through the Decoder layer for optical flow decoding, wherein the Decoder layer and the Encoder layer share features through jump connections, and output a set of residuals for optimizing the initial optical flow, and the number of the residuals in the set is k, which has the same meaning and value as k in step S13 of the present disclosure.

[0132] S25, optimizing the initial optical flow according to a second preset number of optical flow residuals to determine a preset number of embedded optical flows and a mask matrix.

[0133] Continuing with the above example, k optical flow residuals can be used to optimize the initial optical flow to determine a set of k pairs of embedded optical flows and mask matrices for describing the first perspective and the second perspective, where the k pairs of embedded optical flows are:

[0134]

[0135] in, represents the embedded optical flow from the first view to the second view, Represents the embedded optical flow from the second view to the first view.

[0136] In some possible embodiments, determining a first preset number of first optical flows from the first color texture image to a virtual view of the third perspective and a first preset number of second optical flows from the second color texture image to a virtual view of the third perspective according to a third perspective input by a user and a preset logarithm of embedded optical flows includes:

[0137]

[0138] in, represents the first optical flow, p represents the third viewing angle, represents the embedded optical flow from the first view to the second view, represents the second optical flow, Represents the embedded optical flow from the second view to the first view.

[0139] Figure 4 The present invention is a flow chart of a method for determining a complete virtual view at a third perspective according to an exemplary embodiment.

[0140] like Figure 3 , Figure 4 As shown, in some possible embodiments, a first preset number of first optical flows and a first preset number of second optical flows are used to perform forward warping on the first color texture image and the second color texture image, and a mask matrix is ​​used to fuse the images after the forward warping process to determine that the complete virtual view at the third perspective includes S31 to S34.

[0141] In the present disclosure, the fusion process includes a first fusion process and a second fusion process.

[0142] S31, forward warping the first color texture image to a third viewing angle according to a first preset number of first optical flows using a forward warping operator, and determining a first preset number of first-side virtual third color texture images.

[0143] S32, forward warping the second color texture image to a third viewing angle according to a first preset number of second optical flows using a forward warping operator, and determining a first preset number of second-side virtual third color texture images.

[0144] Continuing with the above example, in S31 to S32, according to k pairs of optical flow pairs consisting of the first optical flow and the second optical flow, the first perspective and the second perspective can respectively obtain k groups of images mapped to the target position, that is, k groups of first-side virtual third color texture images and k groups of second-side virtual third color texture images, wherein the first-side virtual third color texture image and the second-side virtual third color texture image are both images of the third perspective:

[0145]

[0146] in, represents the first side virtual third color texture image, represents the first side, the second side, and the third virtual color texture image, I left Represents the first color texture image, I right represents the second color texture image, represents the first optical flow, represents the second optical flow.

[0147] S33, performing a first fusion process on a first preset number of first-side virtual third color texture images and a first preset number of second-side virtual third color texture images respectively, to determine a first-side fusion image and a second-side fusion image after the first fusion process.

[0148] Among them, the first side fusion image:

[0149]

[0150] Among them, I left→p represents the first side fused image, represents the first side virtual third color texture image, and k represents the first preset number.

[0151] Second side fused image:

[0152]

[0153] Among them, I right→p represents the second side fused image, Represents the second side virtual third color texture image.

[0154] S34, performing a second fusion process on the first side fusion image and the second side fusion image according to the mask matrix to determine a complete virtual view at a third viewing angle.

[0155] Continuing with the above example,

[0156] I p =M⊙I left→p +(1-M)⊙I right→p

[0157] Among them, I p Represents the complete virtual image from the third perspective.

[0158] In the present disclosure, a mask matrix is ​​used as a weight to perform weighted summation processing.

[0159] In a possible embodiment, the leftmost area at the first perspective is invisible in the third perspective. After the distortion process, the leftmost area of ​​the virtual third color texture image on the first side is an invalid area. At this time, M=0, and the mask matrix M is learned through the convolutional neural network.

[0160] Based on the same inventive concept, the present disclosure also provides a free-viewpoint video generation method.

[0161] Figure 5 The figure is a flow chart of a method for generating a free-viewing-angle video according to an exemplary embodiment. Figure 6 The figure is a schematic diagram showing the arrangement of an image collector according to an exemplary embodiment.

[0162] like Figure 5 As shown, the present disclosure also provides a free-viewpoint video generation method, including S101 to S106.

[0163] S101, acquiring multiple frames of color images of the same calibration object from different viewing angles.

[0164] like Figure 6 As shown, as an example, an image collector is used to obtain multiple frames of color images of the same calibration object from different perspectives. The image collector can be an industrial camera that supports shooting scenes at 4K / 120FPS.

[0165] In the present disclosure, the number of industrial cameras can be 12, and they are arranged in a fixed arc with a field of view of 70 degrees, and the 12 industrial cameras are used to shoot the scene. The image information collected by the 12 industrial cameras quasi-synchronously is transmitted to the cloud for processing.

[0166] S102, combining multiple frames of color images in pairs.

[0167] S103, obtaining preset logarithmic embedded optical flows and mask matrices in real time from each of the multiple color image frames combined in pairs according to a real-time arbitrary viewing angle generation method.

[0168] Continuing with the above example, 12 industrial cameras are used to collect multiple frames of color images, i.e., videos, which are processed in the cloud, where the cloud processing may include encoding and compression processing, and then distributed to the edge media server. A real-time arbitrary perspective generation network is deployed in the edge media server of the present disclosure to perform real-time virtual perspective interpolation.

[0169] The edge media server combines the adjacent two channels of the 12 received videos into 11 pairs of left and right perspectives, and then obtains the corresponding preset logarithmic embedded optical flow and mask matrix for each pairwise combination.

[0170] S104, adaptively splicing each of the two-by-two combinations of multiple frames of color images and their preset logarithmic embedded optical flows and mask matrices to form multiple synthetic images in the spatial domain.

[0171] In some possible embodiments, S104 specifically includes S201 to S203.

[0172] S201, performing channel dimension synthesis processing on the preset logarithmic embedded optical flow and mask matrix corresponding to each of the multiple color frames combined in pairs, and determining a three-channel matrix corresponding to each of the multiple color frames combined in pairs.

[0173] Among them, the embedded optical flow is 2 channels, and the mask matrix is ​​1 channel. After channel dimension synthesis processing, a three-channel matrix is ​​formed, and its resolution is consistent with the resolution of multi-frame color images combined in pairs.

[0174] S202 , performing spatial domain synthesis processing on each of the two-by-two combination of multi-frame color images and the three-channel matrix corresponding to each of the two-by-two combination of multi-frame color images to determine a synthesized image.

[0175] The two-by-two combination of multi-frame color images are placed in the upper left and upper right, respectively, and the three-channel matrix is ​​placed in the lower left. The lower right part of the composite image can be added with texture images of other perspectives other than the two-by-two combination and embedded with optical flow and mask matrix depending on the situation.

[0176] S203, performing resolution reduction processing on the multiple color images of different viewing angles and all the embedded optical flows and mask matrices in pairs to form a total composite image.

[0177] S105 , encoding the multiple composite images, dividing them into video segments in the time domain and transmitting them through the HLS protocol.

[0178] In some possible embodiments, S105 specifically includes S301 to S303.

[0179] S301, encoding each synthesized image and performing time domain slicing in time sequence;

[0180] S302, dividing each slice into fixed time sizes to form multiple video segments jointly distributed in time domain and space domain;

[0181] S303: Use a standard streaming media transmission protocol (such as HLS, etc.) to transmit the video clip.

[0182] As an example, a time-space segmentation method is used to cope with multi-user scenarios. Multiple synthetic image phenomenon streams are divided into a series of video slices in the time domain and the space domain. Specifically, 12 adaptively organized synthetic images form segments in the space domain, and then each segment is segmented in chronological order. In this embodiment, an edge media server is used as a media resource repository, where the server load is independent of the number of clients, and the client operation will be explained separately. As for content transmission, the HTTPAdaptive Streaming transmission protocol divides the video content into video slices of the same length for transmission. The number of frames of each video segment is an integer multiple of the encoded GOP Size, and the first frame is a full-frame intra-reference frame (I frame), so that each video segment can be decoded independently. The HLS protocol is one of them, which is very suitable for the transmission of time-space video slices.

[0183] S106, different clients interactively select target synthetic images and download them, and synthesize a complete virtual view at the perspective viewed by the user in real time according to a real-time arbitrary perspective generation method.

[0184] In a possible embodiment, S106 specifically includes S401 to S404.

[0185] S401, using a global viewing angle index, downloading a synthetic image and a total synthetic image corresponding to the viewing angle viewed by the user;

[0186] S402, within a video slice time, the viewing angle of the user falls within the downloaded synthetic image, and the synthetic image is used to synthesize a complete virtual view at the viewing angle of the user in real time according to a real-time arbitrary viewing angle generation method;

[0187] S403, within a video slice time, the viewing angle of the user falls outside the downloaded synthetic image, and a complete virtual view at the viewing angle of the user is synthesized in real time using the total synthetic image according to a real-time arbitrary viewing angle generation method;

[0188] S404, after a video slice time, the user downloads the complete virtual video and the total synthesized image at the viewing angle.

[0189] As an example, users of different clients interactively select and download the viewpoint synthetic image to be viewed, and synthesize the complete virtual view at the viewed viewpoint in real time. In this embodiment, when users of different clients interactively request the corresponding viewpoint, that is, the complete virtual view at the viewpoint, the operation of searching the corresponding view index from the global lookup table (corresponding to the previous global viewpoint index, which contains the position information of different viewpoints in the synthetic picture, and the corresponding viewpoint tile can be found through this lookup table) will be performed. If the required viewpoint is in the current synthetic picture, the viewpoint extraction unit extracts the corresponding viewpoint tile bitstream and decodes it through the video slice decoding unit. If it is not in the current synthetic picture, the corresponding viewpoint tile bitstream will be extracted from the total synthetic picture. The user's interaction directly determines the next synthetic picture video segment to be downloaded, that is, the synthetic picture closest to the required viewing angle. The video segment decoding unit is responsible for decoding the extracted bitstream into YUV format, which is then played by the OpenGL-based video player.

[0190] Based on the same concept, the present disclosure also provides a real-time arbitrary perspective generation system. Figure 7 is a block diagram of a real-time arbitrary perspective generation system according to an exemplary embodiment. Figure 7The real-time arbitrary perspective generation system 100 includes: an acquisition module 110, a first determination module 120, a second determination module 130, a third determination module 140, and a fourth determination module 150.

[0191] An acquisition module 110 is used to acquire multiple color images of the same calibration object from different perspectives, wherein the multiple color images of the same calibration object from different perspectives include a first color texture image from a first perspective and a second color texture image from a second perspective;

[0192] A first determination module 120, configured to input the first color texture image and the second color texture image into a preset dense optical flow field estimation network to determine a bidirectional optical flow;

[0193] A second determination module 130 is used to input the first color texture image, the second color texture image and the bidirectional optical flow into an optical flow embedder to determine an embedded optical flow and a mask matrix of a preset logarithm;

[0194] A third determining module 140 is used to determine a first preset number of first optical flows from the first color texture image to the virtual view of the third perspective and a first preset number of second optical flows from the second color texture image to the virtual view of the third perspective according to a third perspective input by a user and the preset logarithm of embedded optical flows;

[0195] a fourth determination module, configured to perform forward warping on the first color texture image and the second color texture image according to the first preset number of first optical flows and the first preset number of second optical flows, and to fuse the images after the forward warping according to the mask matrix to determine a complete virtual view at the third perspective.

[0196] Regarding the embodiment of the above-mentioned real-time arbitrary perspective generation system, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0197] Figure 8 The diagram is a schematic diagram of the architecture of a free-viewpoint video generation system according to an exemplary embodiment.

[0198] Based on the same concept, Figure 8 As shown, the present disclosure also provides a free viewpoint video generation system, which mainly includes a collection module, a cloud processing and content distribution network module, an edge server module and a client module.

[0199] The acquisition server module is used to obtain multiple frames of color images of the same calibration object from different perspectives;

[0200] A cloud processing and content distribution network module, used for combining the multiple frames of color images in pairs and distributing them to the edge server module through a content distribution network;

[0201] The edge server module is used to obtain the preset logarithmic embedded optical flow and mask matrix of each of the two-by-two combined multi-frame color images in real time; adaptively splice each of the two-by-two combined multi-frame color images and the preset logarithmic embedded optical flow and mask matrix to form a plurality of composite images in the spatial domain; encode the plurality of composite images, divide them into video segments in the time domain, and transmit them through the HLS protocol;

[0202] The client module is used for different clients to interactively select target synthetic images and download them, so as to synthesize a complete virtual view at the user's viewing angle in real time.

[0203] The function of each module can be found in the above-mentioned embodiment of the free viewpoint video method, which will not be described in detail here.

[0204] In actual applications, when building the free-viewpoint video generation system of this embodiment, a pipeline design pattern can be adopted, and a first-in-first-out queue data structure can be used to achieve data transmission and thread separation, so that each unit in the system can execute in highly parallel manner. Therefore, the overall delay bottleneck of the system is only limited by the module that takes the longest time, thereby achieving better real-time performance. In addition, heterogeneous computing, including CPU and GPU, can be used to greatly improve the performance of the entire system.

[0205] Table 1 Performance test results of free viewpoint video system

[0206]

[0207] Table 1 is a performance test result table of the free viewpoint video generation system pipeline of the embodiment of the present disclosure. The delay of each unit of the free viewpoint video generation system and the average delay of 1000 frames are tested. As shown in Table 1, the free viewpoint video generation system of the embodiment of the present disclosure achieves good real-time performance. It should be noted that the client of this embodiment only uses two clients, the PC and the mobile phone, for testing, but our system can support any number of clients for access in theory.

[0208] Through the above technical solution, the free-viewpoint video generation system built based on this method decouples the number of access users and the edge server load, so that a single edge server can provide personalized free-viewpoint video services for multiple users.

[0209] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0210] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0211] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0212] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0213] The above describes the specific embodiments of the present disclosure. It should be understood that the present disclosure is not limited to the above specific embodiments, and those skilled in the art can make various deformations or modifications within the scope of the claims, which does not affect the essence of the present disclosure. The above preferred features can be used in any combination without conflicting with each other.

Claims

1. A real-time arbitrary viewing angle generation method, characterized in that: include: Acquire multiple frames of color images from different perspectives of the same calibration object, wherein the multiple frames of color images from different perspectives of the same calibration object include a first color texture image from a first perspective and a second color texture image from a second perspective; Inputting the first color texture image and the second color texture image into a preset dense optical flow field estimation network to determine a bidirectional optical flow; Inputting the first color texture image, the second color texture image and the bidirectional optical flow into an optical flow embedder to determine a preset logarithmic embedded optical flow and a mask matrix; Determine, according to the third perspective input by the user and the preset logarithm of embedded optical flows, a first preset number of first optical flows from the first color texture image to a virtual view of the third perspective, and a first preset number of second optical flows from the second color texture image to a virtual view of the third perspective; The first color texture image and the second color texture image are forward-warped respectively using the first preset number of first optical flows and the first preset number of second optical flows, and the images after the forward warping are fused using the mask matrix to determine a complete virtual view at the third perspective.

2. The method according to claim 1, characterized in that The step of inputting the first color texture image, the second color texture image, and the bidirectional optical flow into an optical flow embedder to determine an embedded optical flow and a mask matrix of a preset logarithm includes: Performing feature extraction processing on the first color texture image and the second color texture image to determine image features of the color images of the first viewing angle and the second viewing angle at different resolution scales; Performing a first distortion process on the image features of the color images of the first viewing angle and the second viewing angle through an initial optical flow, and determining image features after the first distortion process; Aligning and splicing the image features after the first distortion processing and the original image features at the distortion end point in sequence to determine spliced ​​image features; Performing encoding and decoding processing on the stitched image features in sequence to determine a second preset number of optical flow residuals, wherein the value of the second preset number is the same as the value of the preset logarithm; The initial optical flow is optimized according to the second preset number of optical flow residuals to determine a preset number of embedded optical flows and a mask matrix.

3. The method according to claim 1, characterized in that The step of determining, according to the third perspective input by the user and the preset logarithm of embedded optical flows, a first preset number of first optical flows from the first color texture image to the virtual view of the third perspective, and a first preset number of second optical flows from the second color texture image to the virtual view of the third perspective, comprises: in, represents the first optical flow, p represents the third viewing angle, represents the embedded optical flow from the first view to the second view, represents the second optical flow, represents the embedded optical flow from the second view to the first view.

4. The method according to claim 1, characterized in that: The fusion process includes a first fusion process and a second fusion process; The method of using the first preset number of first optical flows and the first preset number of second optical flows to perform forward warping processing on the first color texture image and the second color texture image respectively, and using the mask matrix to fuse the images after the forward warping processing to determine the complete virtual view at the third perspective includes: Using a forward warping operator to forward warp the first color texture image to the third viewing angle according to the first preset number of first optical flows, and determining a first preset number of first-side virtual third color texture images; Using the forward warping operator to forward warp the second color texture image to the third viewing angle according to the first preset number of second optical flows, to determine a first preset number of second-side virtual third color texture images; Perform the first fusion process on the first preset number of first-side virtual third color texture images and the first preset number of second-side virtual third color texture images respectively, and determine a first-side fusion image and a second-side fusion image after the first fusion process; The first side fusion image and the second side fusion image are subjected to the second fusion process according to the mask matrix to determine a complete virtual view at the third viewing angle.

5. The method according to claim 4, characterized in that performing the second fusion process on the first side fusion image and the second side fusion image according to the mask matrix to determine a complete virtual view at the third viewing angle, include: I p =M⊙I left→p +(1-M)⊙I right→p Among them, I p represents the complete virtual image at the third viewing angle, M represents the mask matrix, and I left→p represents the first side fusion image, I right→p represents the second side fused image, represents the first side virtual third color texture image, represents the second side virtual third color texture image, and k represents the first preset number.

6. A real-time arbitrary perspective generation system, characterized in that: include: An acquisition module is used to acquire multiple frames of color images of the same calibration object from different perspectives, wherein the multiple frames of color images of the same calibration object from different perspectives include a first color texture image from a first perspective and a second color texture image from a second perspective; A first determination module, configured to input the first color texture image and the second color texture image into a preset dense optical flow field estimation network to determine a bidirectional optical flow; A second determination module is used to input the first color texture image, the second color texture image and the bidirectional optical flow into an optical flow embedder to determine an embedded optical flow and a mask matrix of a preset logarithm; A third determining module is used to determine a first preset number of first optical flows from the first color texture image to the virtual view of the third perspective and a first preset number of second optical flows from the second color texture image to the virtual view of the third perspective according to a third perspective input by a user and the preset logarithm of embedded optical flows; a fourth determination module, configured to perform forward warping on the first color texture image and the second color texture image according to the first preset number of first optical flows and the first preset number of second optical flows, and to fuse the images after the forward warping according to the mask matrix to determine a complete virtual view at the third perspective.

7. A method for generating a free-viewpoint video, characterized in that: include: Acquire multiple color images of the same calibration object from different perspectives; Combining the multiple color image frames in pairs; Obtaining a preset logarithmic embedded optical flow and mask matrix in real time from each of the two-by-two combined multi-frame color images according to the method described in any one of claims 1-2; Adaptively splicing the multiple color image frames combined in pairs and the preset logarithmic embedded optical flows and mask matrices thereof to form multiple synthetic images in the spatial domain; Encoding the multiple composite images, dividing them into video segments in the time domain, and transmitting them through the HLS protocol; Different clients interactively select target synthetic images and download them, and synthesize a complete virtual view at the user's viewing angle in real time according to the method described in any one of claims 3-4.

8. The method according to claim 7, characterized in that The step of adaptively splicing the multiple color image frames combined in pairs and the preset logarithmic embedded optical flows and mask matrices to form multiple synthetic images in the spatial domain includes: Perform channel dimension synthesis processing on the preset logarithmic embedded optical flow and mask matrix corresponding to each of the two-by-two multi-frame color images to determine a three-channel matrix corresponding to each of the two-by-two multi-frame color images; Performing spatial domain synthesis processing on each of the two-by-two combination of the multi-frame color images and the three-channel matrix corresponding to each of the two-by-two combination of the multi-frame color images to determine a synthesized image; The multiple frames of color images from different viewing angles and all pairs of embedded optical flows and mask matrices are subjected to resolution reduction processing to form a total composite image.

9. The method according to claim 8, characterized in that The different clients interactively select target synthetic images and download them, and synthesize a complete virtual view at the viewing angle of the user in real time according to the method described in any one of claims 3 to 4, including: Using a global viewing angle index, download the synthetic image and the total synthetic image corresponding to the viewing angle viewed by the user; Within a video slice time, the viewing angle of the user falls within the downloaded synthetic image, and the synthetic image is used to synthesize a complete virtual view at the viewing angle of the user in real time according to any one of claims 4-5; In a video slice time, the viewing angle of the user falls outside the downloaded synthetic image, and the total synthetic image is used to synthesize a complete virtual view at the viewing angle of the user in real time according to any one of claims 4-5; After a video slicing time, the user downloads the complete virtual video and the total synthesized image at the viewing angle.

10. A free viewpoint video generation system, characterized in that: include: The acquisition server module is used to obtain multiple frames of color images of the same calibration object from different perspectives; A cloud processing and content distribution network module, used for combining the multiple frames of color images in pairs and distributing them to the edge server module through a content distribution network; The edge server module is used to obtain the preset logarithmic embedded optical flow and mask matrix of each of the two-by-two combined multi-frame color images in real time; adaptively splice each of the two-by-two combined multi-frame color images and the preset logarithmic embedded optical flow and mask matrix to form a plurality of composite images in the spatial domain; encode the plurality of composite images, divide them into video segments in the time domain, and transmit them through the HLS protocol; The client module is used for different clients to interactively select target synthetic images and download them, so as to synthesize a complete virtual view at the user's viewing angle in real time.

Citation Information

Patent Citations

  • Multi-user free view angle video method and system based on real-time virtual view angle interpolation

    CN114897681A

  • Method and device for generating video intermediate frame

    CN115065796A

  • Intermediate frame generation method and device, equipment and medium

    CN115761065A

  • Multi-view synchronization method and free view system

    CN117221627A

  • Virtual viewpoint synthesis method and system, storage medium and equipment

    CN117915056A