Real-time multi-view video conversion method and system

By receiving 2D video streams and generating multi-view frames using multi-view parameters, the problem of real-time conversion of 2D video into multi-view video in existing technologies is solved, realizing real-time conversion and rendering of multi-view video, which is suitable for multi-view displays.

CN116601943BActive Publication Date: 2026-11-17LEIA INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202180082001.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-12-07
Filing Date
2021-02-28
Publication Date
2026-11-17
Estimated Expiration
2041-02-28

AI Technical Summary

Technical Problem

Existing video conversion technologies struggle to convert 2D video into multi-view video in real time, especially due to a lack of dynamic adjustment and efficient processing during generation and rendering.

Method used

By receiving 2D video streams and loading them into a buffer, multi-view frames are generated in real time using multi-view parameters such as camera baseline and center viewpoint. Views are generated by mixing subsets of 2D frames, and user-specified parameters are dynamically adjusted to achieve real-time rendering of multi-view video.

Benefits of technology

It enables real-time conversion and rendering from 2D video streams to multi-view videos, supports dynamic adjustment and efficient processing, and is suitable for various multi-view displays such as light field displays and glasses-free 3D displays.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116601943B_ABST
    Figure CN116601943B_ABST
Patent Text Reader

Abstract

The system and method involve real-time multi-view video conversion. This conversion may involve receiving a video stream comprising two-dimensional (2D) frames, where each 2D frame corresponds to a corresponding 2D video timestamp. Furthermore, a camera baseline and center viewpoint are identified. These parameters can be user-specified or predetermined. The target timestamp of the view of the multi-view frames can be determined based on the camera baseline and center viewpoint. The view is generated from a subset of 2D frames having 2D video timestamps adjacent to the target timestamp. The multi-view video is rendered for display, wherein the multi-view video comprises the views of the multi-view frames.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Patent Application No. 63 / 122,419, filed on December 7, 2020, the entire contents of which are incorporated herein by reference.

[0003] Statement regarding federally sponsored research or development

[0004] none Background Technology

[0005] A video stream consists of a series of frames, each of which can be a two-dimensional (2D) image. Video streams can be compressed according to video coding standards to reduce video file size, thereby reducing network bandwidth requirements. Video streams can be received by computing devices from various sources. Video streams can be decoded and rendered for display in a graphics pipeline. Rendering these frames at a specific frame rate produces a video display.

[0006] Emerging display technologies such as holographic displays, 3D displays, volumetric displays, light field displays, and other similar displays can provide a more immersive viewing experience than conventional 2D video. The techniques used to convert 2D video into different formats involve extensive preprocessing work before the video can be rendered. Attached Figure Description

[0007] The various features of the examples and embodiments based on the principles described herein can be more readily understood by referring to the following detailed description taken in conjunction with the accompanying drawings, wherein the same reference numerals denote the same structural elements, and wherein:

[0008] Figure 1 An overview of a process for converting a two-dimensional (2D) video stream into a multi-view video, based on embodiments consistent with the principles described herein, is provided.

[0009] Figure 2 An embodiment is shown that receives a 2D video stream and loads 2D frames into a buffer, according to principles consistent with those described herein.

[0010] Figure 3 An example is shown of generating a view of a multi-view video frame from a subset of 2D frames according to an embodiment consistent with the principles described herein.

[0011] Figure 4 An example of a user interface for receiving multi-view parameters is shown, based on an embodiment consistent with the principles described herein.

[0012] Figure 5 An example of rendering multi-view video for display is shown, based on an embodiment consistent with the principles described herein.

[0013] Figure 6 An example is shown of real-time conversion and rendering of multi-view video for display while receiving a 2D video stream as input, according to an embodiment consistent with the principles described herein.

[0014] Figure 7A An example is shown of an implementation consistent with the principles described herein, which performs a weighted blending of time-adjacent 2D frames proportional to their temporal differences to generate a view.

[0015] Figure 7B An example is shown of motion-based blending of 2D frames with execution times adjacent to each other to generate a view, according to an embodiment consistent with the principles described herein.

[0016] Figure 8 An example of an architecture for converting a 2D video stream into a multi-view video in real time, based on an embodiment consistent with the principles described herein, is shown.

[0017] Figure 9 A flowchart is shown of a system and method for performing real-time multi-view video conversion according to an embodiment consistent with the principles described herein.

[0018] Figure 10 This is a schematic block diagram illustrating an example of a computing device performing real-time multi-view video conversion according to an embodiment consistent with the principles described herein.

[0019] Some examples and embodiments have other features that complement or replace the features shown in the above figures. These and other features are described in detail below with reference to the above figures. Detailed Implementation

[0020] Examples and embodiments based on the principles described herein provide techniques for real-time conversion of two-dimensional (2D) video streams into multi-view video. Specifically, embodiments involve receiving a 2D video stream and dumping 2D frames of the 2D video stream into a buffer for conversion. The 2D video stream may represent a scene captured with the camera view moving relative to the scene along a linear or orbital direction. This may involve translating along a specific direction or hovering around an object. Embodiments involve computing different views of a multi-view frame from a sampled subset of the 2D frames. Various multi-view parameters (such as, for example, camera baseline or center viewpoint) are used to define a target timestamp for each view in the multi-view frame. Using the target timestamp, temporally adjacent 2D frames are identified (e.g., sampled) and blended to generate each view of the multi-view frame. Multi-view frames can be generated in real-time from an incoming 2D video stream to allow for real-time conversion and rendering of multi-view video.

[0021] According to an embodiment, multi-view frames can be dynamically generated in real time by sampling 2D frames based on timestamps. Furthermore, the user interface allows for on-the-fly changes to multi-view parameters, thereby dynamically adjusting the output of the real-time rendering.

[0022] Figure 1 An overview of a process for converting a two-dimensional (2D) video stream into multi-view video, based on embodiments consistent with the principles described herein, is provided. Video capture processing 103 involves capturing video of scene 105. Scene 105 may be a physical arrangement of objects of interest to a viewer. Video capture processing 103 may be performed in a studio or any other physical environment. The goal is to visually capture scene 105 using camera 107. Camera 107 may capture video, which consists of a series of time-series images (called frames). The camera may move relative to the scene along a specific direction 109 to capture different angles of scene 105. Direction 109 may be primarily linear or orbital to obtain different viewing angles of the scene.

[0023] Camera 107 can be attached to a vehicle that moves relative to scene 105. The vehicle can be a drone or other unmanned aerial vehicle (UAV). In some examples, scene 105 may include one or more objects set on a turntable that rotates relative to the fixed camera 107. Video capture processing 103 ultimately transforms the visual details of the scene occupying physical space into a 2D video stream 112. The 2D video stream may consist of a series of time-ordered 2D frames 115. Each 2D frame 115 represents a single view of the scene at a specific point in time. Time can be recorded as a timestamp specific to the corresponding 2D frame. The result of rendering the 2D frames sequentially at a target frame rate is the 2D video perceived by the viewer.

[0024] Although 2D video stream 112 is described as video of a physical scene, it should be understood that 2D video stream 112 can be generated by moving a virtual camera relative to a 3D model. For example, game engines and other 3D model generators can allow the creation of 2D video by applying a virtual camera (defined according to position and orientation) to a 3D model (e.g., a volumetric model of a scene defined according to point clouds, surface meshes, texture maps, etc.).

[0025] Camera 107 can transmit a 2D video stream via a network or otherwise upload it to a repository. The repository may include a storage device, a server, a cloud-based storage device, or other computing systems configured to receive and store video files. The network may include the Internet, an intranet, a wireless network, a wired network, or any other network facilitating data streaming. The 2D video stream 112 can be encoded according to a specific encoding format. Encoding involves applying lossy or lossless compression to reduce file size to accommodate network bandwidth. The video encoding format may be, for example, H.264, H.265, or any format specified by the Moving Picture Experts Group (MPEG). Additionally, the 2D video stream 112 can be packetized for transmission over an Internet Protocol (IP) network.

[0026] Multiview conversion processing 119 can be used to generate multiview video 118 from 2D video stream 112. Multiview conversion processing 119 is a computer-implemented process that converts the 2D video stream into multiview video 118 that can be rendered for display. Multiview conversion processing 119 can be implemented as a processor-executable software program, routine, or module. Multiview conversion processing 119 can receive 2D video stream 112 as a compressed, packetized data stream. For example, multiview conversion processing 119 can use an application programming interface (API) or other function calls to retrieve 2D video stream 112 from a storage library. The 2D video stream can be live video, such that video capture processing 103 sends the 2D video stream along with the video it is capturing of scene 105, and wherein the 2D video is streamed to a computing system implementing the multiview conversion processing. In some embodiments, 2D video stream 112 includes pre-recorded (non-live) video configured to be streamed to a computing device upon request.

[0027] Multiview video 118 can be stored in memory as one or more computer files or other files in other data formats. Multiview video 118 includes a series of multiview frames 121 depicted as multiview frames A to N. The multiview frames 121 are displayed sequentially at a specified frame rate (expressed as frames per second (FPS)). Therefore, multiview frames 121 are multiview images. Each multiview frame 121 consists of multiple views 124. For example, multiview frame A is depicted as having multiple views. Figure 1 The view n can be applied to each multiview frame 121 in the multiview video 118. In some embodiments, when rendered on a multiview display, each view 124 of the multiview frame 121 corresponds to a corresponding principal angular direction at a surface normal relative to the multiview display. As a result, the viewer is able to perceive multiple different views 124 simultaneously. This results in three-dimensional (3D) or light field image content.

[0028] The embodiment pertains to a multi-view conversion process 119, which can convert a 2D video stream into a multi-view video in real time. Furthermore, as the 2D video stream is received, the multi-view conversion process 119 allows for real-time rendering and display. In this document, a "two-dimensional display" or "2D display" is defined as a display configured to provide substantially the same image view regardless of the viewing direction (i.e., within a predefined viewing angle or range of the 2D display). Conventional liquid crystal displays (LCDs) found in many smartphones and computer monitors are examples of 2D displays. Conversely, a "multi-view display" is defined as an electronic display or display system configured to simultaneously provide different views of a multi-view image from different viewing directions or from different viewing directions from the user's perspective. Specifically, the different views can represent different perspectives of a multi-view frame.

[0029] Multi-view displays can be implemented using various techniques that adapt the presentation of different image views so that they can be perceived simultaneously. One example of a multi-view display is one that uses diffraction gratings to control the dominant angular direction of different views. According to some embodiments, a multi-view display can be a light field display, which presents multiple beams of light corresponding to different colors and directions for different views. In some examples, a light field display is a so-called "glasses-free" 3D display that can use diffraction gratings to provide an automatic stereoscopic representation of multi-view images without requiring special eyewear to perceive depth. In some embodiments, a multi-view display may require glasses or other eyepieces to control which views are perceived by each of the user's eyes.

[0030] As used in this article, "parallax" is defined as the difference between at least two views of a multiview image at corresponding locations. For example, in the context of stereoscopic vision, the left and right eyes may see the same object, but at slightly different locations, due to the difference in viewing angles between the eyes. This difference can be quantified as parallax. Variations in parallax across multiview images convey a sense of depth.

[0031] As used herein, the “baseline” or “camera baseline” is defined as the distance between two cameras (or a single camera at different times) capturing corresponding views of a scene. For example, in the context of stereoscopic vision, the baseline is the distance between the left and right eyes. A larger baseline results in increased parallax and can enhance the 3D effect of multi-view images.

[0032] As used in this article, "convergence offset" refers to the distance between the camera and a point along the convergence plane. Modifying the convergence offset will change the position of the convergence plane so that the multi-view image can be refocused on new objects at different depths.

[0033] Furthermore, as used herein, the article “a” is intended to have its common meaning in the patent field, namely “one or more”. For example, “a processor” means one or more processors, and therefore, “a memory” herein means “one or more memory components”.

[0034] The embodiments relate to a computer-implemented method for real-time multi-view video conversion. This is discussed in further detail below. Figure 10 An example of a computing device configured to implement a method for real-time multi-view video convergence is provided. (The above is about...) Figure 1 The multi-view transformation process 119 described herein can be configured to perform real-time multi-view transformation. Real-time multi-view transformation may include receiving a video stream comprising two-dimensional (2D) frames, where each 2D frame corresponds to a corresponding 2D video timestamp. Real-time multi-view transformation may include identifying a camera baseline and a center viewpoint. Real-time multi-view transformation may also include determining a target timestamp for the view of the multi-view frames based on the camera baseline and center viewpoint. Real-time multi-view transformation may also include generating views from a subset of 2D frames having 2D video timestamps adjacent to the target timestamp. Real-time multi-view transformation may also include rendering multi-view video for display, the multi-view video comprising the views of the multi-view frames.

[0035] In some embodiments, real-time multi-view conversion may include decoding a video stream to extract 2D frames in graphics processing unit (GPU) memory. In some embodiments, real-time multi-view conversion may include loading 2D frames from GPU memory into central processing unit (CPU) memory to generate views. In some embodiments, real-time multi-view conversion may include rendering multi-view video for display while receiving the video stream.

[0036] Some embodiments involve a user interface for obtaining user-specified parameters and applying those parameters in real-time during multi-view video conversion. For example, real-time multi-view conversion may include receiving user input via a user interface, the user input including at least one of a camera baseline and a center viewpoint. In some embodiments, real-time multi-view conversion may include dynamically updating the camera baseline and center viewpoint while rendering multi-view video.

[0037] In some embodiments, a view is generated by blending a set of 2D frames. In some embodiments, the real-time conversion process includes determining the time difference between a target timestamp and a 2D video time adjacent to the target timestamp; and may also include performing a weighted blend proportional to the time difference to generate the view. In some embodiments, motion-based blending can be used to perform the blending of 2D frames.

[0038] Figure 2An embodiment is shown that receives a 2D video stream and loads 2D frames into a buffer, according to principles consistent with those described herein. Figure 2 At least a portion of a real-time multi-view video conversion is shown. For example, Figure 2 The image depicts receiving a 2D video stream 112 comprising two-dimensional (2D) frames, each 2D frame (e.g., 2D frame 115) corresponding to a corresponding 2D video timestamp. The timestamp can be a value reflecting relative or absolute time in any time unit (e.g., seconds, date / time, etc.). Each 2D frame may include a timestamp to organize the frames in a time sequence.

[0039] 2D video stream 112 can be received by a computing device from video source 127. Video source 127 can be remotely located relative to the computing device and coupled to the computing device via a wireless or wired network connection. The network connection can be an internet connection, an intranet connection, a peer-to-peer connection, or any other network connection. In this respect, 2D video stream 112 can be downloaded (e.g., streamed in real-time) from a remote computing system via the network. Video source 127 can also be local relative to the computing device. For example, video source 127 can include a camera application that receives video captured by a local camera communicating with the computing device. The camera application can convert the video feed from the local camera into 2D video stream 112. In some cases, 2D video stream 112 can be formatted in a native format such that it is not encoded or otherwise compressed. In various embodiments, 2D video source represents live video acquired in real-time from video capture processing 103.

[0040] 2D frames 115, as part of a 2D video stream 112, can be extracted and loaded into a buffer 128. The buffer 128 can be defined as an address space that is part of a central processing unit (CPU) memory 130, which is utilized by a CPU 132. The CPU 132 can be a general-purpose processor that executes instructions, supports an operating system, and provides applications (e.g., user-level applications). The CPU memory 130 can be system memory accessible to an operating system or an application running on a computing device. The computing device may include a graphics pipeline 134 that receives the 2D video stream. As used herein, a “graphics pipeline” is defined as a computer-implemented environment that renders image data for display. The graphics pipeline 134 may include one or more graphics processing units (GPUs) 137 or other dedicated processing circuitry optimized for rendering image content to a screen. For example, the GPU 137 may include a vector processor that executes a set of instructions to operate on an array of data in parallel. The graphics pipeline 134 may include a graphics card, a graphics driver, or other hardware and software for rendering graphics. The graphics pipeline 134 can be configured to render images on a multi-view display. The graphics pipeline 134 maps pixels to corresponding locations on the display and controls the display to emit light to display the rendered image. The graphics pipeline 134 may also include GPU memory 140. GPU memory 140 may include one or more memory devices used by the graphics pipeline 134.

[0041] In some embodiments, the graphics pipeline 134 is a subsystem separate from the CPU 132. For example, the graphics pipeline 134 may include a GPU 137 separate from the CPU 132. In some embodiments, the graphics pipeline 134 is implemented purely as software by the CPU 132. For example, the CPU 132 may execute software modules that operate as the graphics pipeline 134 without dedicated graphics hardware. In some embodiments, portions of the graphics pipeline 134 are implemented in dedicated hardware, while other portions are implemented as software modules by the CPU 132.

[0042] The graphics pipeline 134 can be used to decode a video stream to extract 2D frames from GPU memory 140. For example, the graphics pipeline 134 can decode a 2D video stream 112 to identify 2D frames according to their encoding format. For example, in an MPEG encoding format, frames can be formatted as I-frames, P-frames, B-frames, etc. The graphics pipeline 134 can store the decoded 2D frames in GPU memory 140. These decoded 2D frames can then be loaded into buffer 128 using frame dump processing.

[0043] Although Figure 2The graphics pipeline 134 performing decoding and frame extraction is shown; however, it should be understood that after decoding the 2D video stream 112, 2D frames 115 can be loaded into buffer 128 using a dedicated hardware or software decoder. While extracting 2D frames 115 from the 2D video stream 112 and loading them into buffer 128, real-time multi-view video conversion then operates on the buffered 2D frames, as described in further detail below.

[0044] Figure 3 An example is shown of generating a view of a multi-view video frame from a subset of 2D frames according to an embodiment consistent with the principles described herein. Figure 3 As shown in the example, a 2D frame is part of a 2D video stream 112. Each 2D frame represents a snapshot of a scene, where the scene is composed of objects of varying depths (e.g., depicted as circles, triangles, pentagons). Furthermore, the 2D video stream 112 represents the scene as the camera pans horizontally relative to the viewer.

[0045] Figure 3 A real-time multi-view video conversion is depicted using 2D frames 115, which can be loaded into buffer 128, as input for the conversion process. This example shows a series of 2D frames 115, depicted as frames A through E. Each 2D frame 115 corresponds to a specific 2D video timestamp (shown as a generic time unit). For example, frame A has a timestamp of 0.0, frame B has a timestamp of 9.0, frame C has a timestamp of 18.0, frame D has a timestamp of 27.0, and frame E has a timestamp of 36.0. Therefore, the time interval between adjacent frames is 9.0 time units. The timestamp of each frame can be stored as metadata.

[0046] A computer-implemented method for real-time multi-view video conversion involves generating multi-view video 118 from 2D frames in a 2D video stream 112 using the timestamp of each 2D frame. Specifically, this involves generating a series of multi-view frames 121, where each multi-view frame includes a different set of views. Figure 3 The example uses four views 124 (views) Figure 1 ,See Figure 2 ,See Figure 3 Heshi Figure 4 Each multiview frame 121 is defined using 121. It should be understood that any number of different views can be used to define the multiview frame 121.

[0047] To generate each view (e.g., view ... Figure 1-4124) Real-time multi-view video conversion processing may involve identifying camera baseline 145 and center viewpoint 148. Camera baseline 145 and center viewpoint 148 are multi-view parameters that control where 2D frames are sampled to generate different views of the multi-view frame 121. Camera baseline 145 quantifies the distance between camera positions (e.g., virtual or physical cameras) associated with corresponding views. A large camera baseline 145 means that the two views of the scene are acquired from distant points. In this respect, as the camera baseline 145 increases, there is a greater overall parallax between the views. While camera baseline 145 can be characterized as a spatial distance between two viewpoints, for the purposes of multi-view video conversion, camera baseline 145 can also be expressed as a temporal difference. For example, camera baseline 145 expressed as spatial distance can also be expressed as a temporal distance based on the linear or orbital motion of the camera. In this example, camera baseline 145 is 8.0 time units.

[0048] Center viewpoint 148 specifies the midpoint between view ranges in multiview frame 121. For example, center viewpoint 148 is a position that can be equidistant from the leftmost and rightmost views. Center viewpoint 148 can be a position represented as the spatial distance between viewpoints in multiview frame 121 or a position based on the temporal distance at the time point at which multiview frame 121 is constructed. In some embodiments, camera baseline 145 and / or center viewpoint 148 can be predetermined parameters stored in memory or otherwise hard-coded. Camera baseline 145 and center viewpoint 148 can be identified by performing a lookup or other read command to determine the values ​​representing camera baseline 145 and center viewpoint 148. In other embodiments, camera baseline 145 and / or center viewpoint 148 can be user-specified. Furthermore, camera baseline 145 and / or center viewpoint 148 can change dynamically in response to user input during real-time multiview video transitions.

[0049] Real-time multi-view video conversion may also include determining the target timestamps of the views in the multi-view frame 121 based on camera baseline 145 and center viewpoint 148. Camera baseline 145 and center viewpoint 148 can be used to calculate the target timestamp for each view before the views are generated. For example, center viewpoint 148 can specify the center time point of the multi-view frame 121, and camera baseline 145 can define the time interval between each view because they are positioned around the center time point.

[0050] For example, Figure 3 The image depicts four views 124 (e.g., viewports) that generate a specific multi-view frame 121 by sampling 2D frames 115 (e.g., frames A through E). Figure 1 To see Figure 4For multi-view frame 121, the target timestamp is calculated from the camera baseline 145 (7.0 time units) and the center viewpoint 148 (17.0 time units). Using the camera baseline 145 and the center viewpoint 148, the view... Figure 1 The target timestamp has a time unit of 7.0, and is viewed as Figure 2 The target timestamp has 15.0 time units, viewed Figure 3 The target timestamp has 23.0 time units and is viewed as... Figure 4 The target timestamp has 31.0 time units. Each view 124 is separated by a camera baseline 145 represented in time units (e.g., 8.0 time units). Additionally, intermediate views 124 (e.g., viewpoints) Figure 2 Heshi Figure 3 The target timestamps are located on either side of the central viewpoint 148 (e.g., time-isolated). Therefore, in order to construct the multi-view frame 121, the target timestamps for each of its views are determined. As a result, a set of target timestamps is determined for each multi-view frame 121.

[0051] Real-time multi-view video conversion may also include generating views from a subset of 2D frames 115 having 2D video timestamps adjacent to the target timestamp. For example, when the target timestamp is determined, each view of the multi-view frames 121 is generated from a subset of the 2D frames. The subset of 2D frames 115 may be at least one pair of 2D frames 115 whose timestamps are adjacent to the target timestamp. For example, in order to generate views... Figure 1 The target timestamp is 7.0 time units. A subset of 2D frames 115 with 2D video timestamps adjacent to this target timestamp are frames A and B. To generate the video... Figure 2 The set of 2D frames 115 with 2D video timestamps adjacent to the target timestamp is frames B and C. To generate the video... Figure 3 The set of 2D frames 115 with 2D video timestamps adjacent to the target timestamp is frames C and D. Furthermore, in order to generate the video... Figure 4 The set of 2D frames 115 having a 2D video timestamp adjacent to the target timestamp is frame D and frame E. The 2D frames 115 having a 2D video timestamp adjacent to the target timestamp may include at least two 2D frames 115 that are closest to the target timestamp in terms of time.

[0052] When a subset of 2D frames 115 with 2D video timestamps adjacent to the target timestamp of a particular view is identified, view 124 is generated by combining the subset of 2D frames 115. For example, the view is generated by blending subsets of 2D frames. In some embodiments, blending may be linear blending. In other embodiments, blending may be motion-based blending. Blending may be a pixel-by-pixel blending operation. Regarding... Figure 7A and 7BAn example of generating view 124 from a subset of sampled 2D frames 115 using a blending operation is described in more detail.

[0053] Real-time multi-view video conversion may also include rendering multi-view video 118 for display, the multi-view video 118 comprising views of multi-view frames 121. For example, when generating each view of each multi-view frame 121, the multi-view frames may be formatted into multi-view video 118 in a predefined video format. Additionally, real-time multi-view video conversion may include rendering multi-view video 118 for display while receiving the video stream. This allows video to be presented in a multi-view format on the fly as 2D video is received. This will be about... Figure 5 Let's discuss this in more detail.

[0054] Return to reference Figure 2 Real-time multi-view conversion may include decoding a video stream to extract 2D frames from GPU memory 140. The 2D video stream 112 may be initially encoded according to a predefined encoding format. 2D frames 115 (e.g., frames A through E) may initially be encoded or otherwise compressed to reduce the file size of the 2D video stream. After decoding the 2D frames into GPU memory 140, real-time multi-view conversion may include loading the 2D frames from GPU memory 140 into CPU memory 130 to generate view 124 (e.g., view...). Figure 1 To see Figure 4 Specifically, real-time multi-view conversion can dump 2D frame 115 into buffer 128 in real time during decoding.

[0055] Figure 4 An example of a user interface for receiving multi-view parameters, based on an embodiment consistent with the principles described herein, is shown. Real-time multi-view video conversion can be implemented by an application that generates a user interface 150 for receiving user input. For example, real-time multi-view video conversion may include receiving user input via user interface 150, the user input including at least one of camera baseline 145 or center viewpoint 148. Additionally, real-time multi-view video conversion may include dynamically updating camera baseline 145 and center viewpoint 148 while rendering multi-view video 118. As a result, the user can dynamically and instantly modify various multi-view parameters as the 2D video stream is converted into multi-view video 118 to dynamically change the multi-view properties of the multi-view video.

[0056] User interface 150 may include a first area 153 displaying a rendered multi-view video 118. User interface 250 may include a second area 156 containing various user interface elements. User interface elements may be, for example, sliders, checkboxes, text fields, buttons, switches, etc. The second area 156 may overlap the first area 153 or be located next to the first area 153. Users can interact with the various user interface elements using a cursor, mouse, keyboard, or touchscreen interface.

[0057] User interface 150 can receive user selections for camera baseline 145, pan settings, zoom settings, convergence offset, clockwise / counterclockwise selection, 2D / multi-view selection, or any other video settings. Camera baseline 145 controls the target timestamp of the view in multi-view frames that can be dynamically generated in real time. For example, pan settings can shift the field of view, thereby specifying a center viewpoint 148. Panning left or right can proportionally shift the center viewpoint 148. Zoom settings can adjust the cropping area applied to each multi-view frame 121. For example, a user can pinch to zoom or otherwise manipulate the zoom slider to specify the area in the rendered multi-view video to zoom in or out from it. 2D frame 115 or multi-view frame 121 can be dynamically processed in response to user-specified video settings.

[0058] Users can specify a convergence offset. The convergence offset can be applied to multi-view frame 121 to change the position of the convergence plane as the multi-view video is rendered. The position of the convergence plane can be modified so that it moves closer to or further away from the viewer. Assuming horizontal parallax (e.g., capturing a scene as it moves horizontally relative to the camera), the convergence offset can be modified by performing a horizontal pixel shift operation on one or more views. The horizontal pixel shift operation can involve cropping the left or right edge of each view to effectively shift the view horizontally. Therefore, the view to the left of center viewpoint 148 can be shifted further to the left, while the view to the right of center viewpoint 148 can be shifted further to the right. Alternatively, the view to the left of center viewpoint 148 can be shifted further to the right, while the view to the right of center viewpoint 148 can be shifted further to the left. The amount and / or direction of the horizontal shift correspond to the convergence offset.

[0059] Users can also specify the view order by selecting a clockwise or counterclockwise selector. For example, Figure 3 It depicts a clockwise direction, in which the view Figure 1 Positioned as the leftmost view, while the view Figure 4 It is positioned as the rightmost view. The view order can be changed to counter-clockwise, making the view... Figure 4 Positioned as the leftmost view, Figure 3 Positioned as the left center view, view Figure 2 Positioned as the right center view, and viewed Figure 1 It is positioned as the rightmost view. Users will expect to modify the view order (e.g., clockwise or counterclockwise) depending on the direction of the overall movement of the scene (e.g., panning left, panning right, etc.).

[0060] The user interface 150 may include various other image processing operations that adjust the final rendering of the multi-view video. User-specified parameters can be provided in real-time during the real-time multi-view video conversion.

[0061] In some embodiments, the user interface 150 includes a selector to choose between rendering the 2D video stream in its original 2D format or rendering the 2D video stream as multi-view video. Viewers can cause the computing device to switch between rendering multi-view video and rendering the 2D video stream in response to user input received via the user interface. Selecting the 2D format can bypass multi-view video conversion.

[0062] Figure 5 An example of rendering multi-view video for display is shown, based on an embodiment consistent with the principles described herein. For example, Figure 5 An example of rendering a multi-view video 118 for display is depicted, the multi-view video including multi-view frames 121, each multi-view frame including a corresponding set of different views 124 (e.g., views...). Figure 1 To see Figure 4 As described above, the application can provide real-time multi-view video conversion that generates multi-view video including multi-view frames 121 from a 2D video stream. During multi-view video generation, the application can interface with graphics pipeline 134 to render the multi-view video including multi-view frames 121 on a multi-view display 160. Graphics pipeline 134 can map each view 124 of each multi-view frame 121 to a corresponding pixel position on the multi-view display 160. As a result, each view 124 can correspond to a corresponding principal angular direction 161. Although... Figure 5 An example of a multi-view display 160 is provided, but other implementations can be used to display multi-view images. For example, some implementations may require special glasses to perceive the different views.

[0063] Figure 6 An example is shown of real-time conversion and rendering of multi-view video for display while receiving a 2D video stream as input, according to an embodiment consistent with the principles described herein. Figure 6 As shown, the embodiment is for real-time multi-view video conversion, which allows rendering multi-view video for display while receiving video streams. Figure 6 It describes the various computational processes that can be pipelined to influence parallel processing.

[0064] Real-time multi-view conversion can be initiated by receiving a 2D video stream 165. The 2D video stream may include a series of 2D frames 115, which are formatted into encoded 2D frames 168 according to the video encoding format. Specifically, the encoded 2D frames 168 contained within the 2D video stream are shown.

[0065] As the 2D video stream is received, real-time multi-view video conversion can perform decoding 171. Decoding may involve decompressing or otherwise extracting 2D frames from the 2D video stream. Graphics pipeline 134, a software decoder, or a hardware decoder can perform decoding 171. Encoded 2D frames 168 are decoded into decoded 2D frames 174. Next, as the 2D frames are decoded, real-time multi-view video conversion can perform frame dumping 177. Frame dumping 177 may involve filling buffer 128 with decoded 2D frames 174 as they are being decoded. In this respect, buffer 128 may be an address space in CPU memory or other system memory containing the most recently decoded 2D frame 174. In some embodiments, buffer 128 is a circular buffer that replaces older decoded 2D frames with new decoded 2D frames.

[0066] Next, as buffer 128 is loaded with the most recently decoded 2D frame 174, real-time multi-view video conversion performs view generation 180 to generate a view for each multi-view frame. Figure 6 The diagram illustrates how specific multi-view frames 186 are generated from multiple decoded 2D frames 174 sampled from self-buffer 128. For example, this could involve generating each view from a subset of 2D frames having 2D video timestamps adjacent to the target timestamp of the view. Simultaneously with view generation, multi-view parameters 183 can be identified. Multi-view parameters 183 may include data, for example, indicating camera baseline 145, center viewpoint 148, convergence offset, or other parameters that control how the views of the multi-view frames should be generated and which 2D frames are sampled during view generation. Multi-view parameters 183 can be identified by reading them from memory. Multi-view parameters 183 can be specified by a user via a user interface. As a result, view generation 180 can be controlled in real-time by a user who provides user input to change how 2D frames are sampled for view generation.

[0067] Next, real-time multi-view video conversion may include a display 189. Display 189 may involve rendering the multi-view video for display while receiving a 2D video stream. The multi-view video consists of a time-series of multi-view frames 186 forming a portion of the rendered multi-view video 192. A graphics pipeline can map pixels from different views to corresponding locations on the multi-view display to present the multi-view video to the user.

[0068] Figure 7A and Figure 7BAn example of a view generated by blending a subset of 2D frames is provided. Specifically, Figure 7A An example is shown of performing a weighted blend of temporally adjacent 2D frames proportional to their temporal difference to generate a view, according to an embodiment consistent with the principles described herein. In this document, "weighted blend" is defined as the sum or other combination of two variables (e.g., pixel values), where each variable is weighted according to a corresponding coefficient (e.g., weight). For example, to generate a view... Figure 1 , calculation view Figure 1 The target timestamp is 7.0 time units. The subset of temporally adjacent 2D frames identified based on the target timestamp is frame A and frame B. Weighted blending can be a weighted blend proportional to the time difference between temporally adjacent frames (e.g., frame A and frame B). The coefficients in the weighted blending correspond to the time difference. For example, the blending operation could involve determining the time difference between the target timestamp and the 2D video timestamps adjacent to the target timestamp. For example, the video... Figure 1 It can have a time difference of 7.0 time units relative to frame A (195a) and a time difference of 2.0 time units relative to frame B (195b). Therefore, compared to frame A, the view... Figure 1 It is closer to frame B in terms of quantization. When determining these time differences 195a and 195b, the view can be generated by performing a weighted blending proportional to the time difference. Figure 1 To generate view 124. This may involve performing linear pixel blending weighted according to these time differences. As a result, compared to frame A, view 124... Figure 1 It will look more like frame B.

[0069] Figure 7B An example is shown of performing motion-based blending of temporally adjacent 2D frames to generate a view, according to an embodiment consistent with the principles described herein. For example, using motion estimation techniques, optical flow between temporally adjacent 2D frames can be determined. Optical flow is a set of data comprising vectors expressing how features move from an initial frame to subsequent frames. In this respect, optical flow includes the velocity of different groups of objects or pixels from one frame to the next. Optical flow can be formatted as a vector matrix. Figure 7B Frame B, overlaid with optical flow data, illustrates the velocities of various objects relative to frame A. When determining the optical flow between frames A and B, a view can be generated based on mixing these temporally adjacent frames using optical flow. Figure 1 .

[0070] Real-time multi-view video conversion can be embodied in processor-based systems (such as, for example, computing devices). In this regard, embodiments are directed to a system configured to perform real-time multi-view video conversion, the system including a central processing unit (CPU) and a memory storing a plurality of instructions that, when executed, cause the CPU to perform various operations to implement real-time multi-view video conversion. The following describes... Figure 10An example describing the system.

[0071] When executed, multiple instructions enable the CPU to load a series of two-dimensional (2D) frames into a memory buffer, each 2D frame corresponding to a corresponding 2D video timestamp; determine the target timestamp of the view for the multi-view frame based on the camera baseline and center viewpoint; and generate a view from a subset of 2D frames having 2D video timestamps adjacent to the target timestamp. Furthermore, the view of the multi-view frame is configured to be part of a multi-view video rendered on a multi-view display.

[0072] Figure 8 An example of an architecture for converting a 2D video stream into a multi-view video in real time, based on an embodiment consistent with the principles described herein, is shown. Figure 8 A processor-based system is shown that executes instructions to perform real-time multi-view video conversion. Specifically, Figure 8 A video source 203 is depicted as a streaming 2D video consisting of a series of 2D frames, each 2D frame corresponding to a corresponding 2D video timestamp. The video source 203 can be similar to the one described above. Figure 2 The video source described is 127.

[0073] Processor-based systems may include an echo buffer module 206, which is implemented by an instruction set stored in memory and executable by a processor (e.g., CPU). The echo buffer module 206 can implement various application programming interfaces (APIs), function calls, or commands to receive 2D video streams, convert 2D frames into multi-view frames, and render multi-view video output in real time. The echo buffer module 206 can interface with various memory devices to load and retrieve video content (e.g., frames). The echo buffer module 206 may include a frame dump 209. The frame dump 209 is configured to load a series of two-dimensional (2D) frames into a memory buffer. The memory buffer may be a circular buffer 212 for frames.

[0074] In some embodiments, frame dump 209 interfaces with a graphics pipeline having a graphics processing unit (GPU) configured to decode a video stream comprising a series of 2D frames. The GPU can be configured to load 2D frames into GPU memory as decoded 2D frames. Frame dump 209 can then load a series of 2D frames from GPU memory into a memory buffer (e.g., the circular buffer of frame 212). Thus, a series of 2D frames is configured to be loaded from GPU memory into a memory buffer. Figure 8 As shown, the memory buffer stores 2D frames 215, including frames A through N.

[0075] The echo buffer module 206 can determine the target timestamp of the view in a multi-view frame based on the camera baseline and center viewpoint. This can be similar to... Figure 4 The example illustrates a camera baseline 145 and a center viewpoint 148 used to determine the target timestamp for a particular view. The echo buffer module 206 may include an output renderer 217 configured to generate a view from a subset of 2D frames having 2D video timestamps adjacent to the target timestamp. For example, the example above regarding... Figure 7A The hybrid operation discussed in 7B can be used to generate views based on temporally adjacent 2D frames. For example, as mentioned above... Figure 7A The CPU, as discussed, can be configured to determine the time difference between a target timestamp and a 2D video timestamp adjacent to the target timestamp; and to perform a weighted blending proportional to the time difference to generate a view. (As mentioned above regarding...) Figure 7B The CPU can also be configured to execute motion-based blends of 2D frames that are temporally adjacent.

[0076] The output renderer 217 can generate a multi-view video 220 with multiple view frames. Each multi-view frame consists of a different set of views. Views can be generated by blending temporally adjacent 2D frames. The multi-view video 220 can be formatted in a suitable video file format and stored in memory. The multi-view video 220 can be configured to be rendered on a multi-view display. (The above is about...) Figure 5 An example of rendering multi-view video for display was discussed. (See above regarding...) Figure 6 The multi-view video discussed can be configured to render on multiple view displays while the video stream is being decoded.

[0077] The echo buffer module 206 may be part of an application executed by the CPU to provide a user interface, or otherwise work in conjunction with an application executed by the CPU to provide a user interface. The user interface may be similar to that described above. Figure 4 The user interface is discussed. The application can be implemented as multiple instructions that can be executed by the CPU to generate a user interface configured for display on a multi-view display. Additionally, the CPU can receive user input via the user interface, including at least one of a camera baseline and a center viewpoint. The CPU can dynamically update at least one of the camera baseline and the center viewpoint while the multi-view video is being rendered. The CPU can also switch between rendering multi-view video and rendering a 2D video stream in response to user input received via the user interface. For example, selecting a 2D video format can bypass the echo buffer module 206 to directly route the received 2D video to the graphics pipeline for rendering without real-time multi-view video conversion.

[0078] Figure 9A flowchart is shown of a system and method for performing real-time multi-view video conversion according to an embodiment consistent with the principles described herein. Figure 9 The flowchart provides an example of different types of functionality implemented by a computing device that executes an instruction set. Figure 9 The flowchart can also be viewed as an example of elements depicting a computer-implemented method according to one or more embodiments. Figure 9 The flowchart can also be viewed as depicting the operations of a non-transitory computer-readable storage medium storing executable instructions that perform multi-view video conversion operations in real time when executed by a processor of a computer system.

[0079] At item 304, the operation involves receiving a video stream comprising a series of two-dimensional (2D) frames, each 2D frame corresponding to a corresponding 2D video timestamp. A 2D frame can be analogous to... Figure 3 2D frame 115 is under discussion.

[0080] At item 307, the operation involves determining the target timestamp of the view for a multi-view frame. (See above regarding...) Figure 3 The target timestamp discussed may be adjacent to a set of 2D video timestamps corresponding to a set of 2D frames. In some embodiments, the operation includes determining the target timestamp based on at least one of a user-specified camera baseline or a user-specified center viewpoint. For example, a user interface may be used to obtain a timestamp similar to the one described above. Figure 4 User input for discussion.

[0081] At item 310, the operation involves blending a collection of 2D frames to generate a view. For example, as mentioned above... Figure 7A The blending discussed may involve determining the time difference between a target timestamp and a 2D video timestamp adjacent to the target timestamp; and performing a weighted blending proportional to the time difference to generate a view. As mentioned above... Figure 7B The mixture discussed can also be a motion-based mixture.

[0082] At item 313, the operation includes rendering multi-view video for display while receiving a video stream; the multi-view video comprises views of multiple view frames. For example, Figure 5 A multi-view display is described, which renders multi-view video in real time as 2D video is received and converted to a multi-view format. While the multi-view video is being converted and rendered in real time, operation can involve switching from rendering multi-view video to rendering a 2D video stream in response to user input received via the user interface.

[0083] The above discussion Figure 9A flowchart can illustrate a system or method for real-time multi-view video conversion, embodied as a set of instructions stored in memory and executable by a processor. If embodied in software, each box can represent a module, segment, or portion of code comprising instructions for implementing specified logical functions (or more). Instructions can be embodied in source code comprising human-readable statements written in a programming language, object code compiled from source code, or machine code comprising numerical instructions recognizable by a suitable execution system (such as a processor, computing device). Machine code can be converted from source code, etc. If embodied in hardware, each box can represent a circuit or multiple interconnected circuits for implementing specified logical functions (or more).

[0084] although Figure 9 The flowchart illustrates a specific execution order, but it will be understood that the execution order may differ from the depicted order. For example, the execution order of two or more boxes may be disrupted relative to the illustrated order. Furthermore, the two or more boxes shown may be executed simultaneously or partially simultaneously. Additionally, in some embodiments, one or more boxes may be skipped or omitted, or one or more boxes may be executed simultaneously.

[0085] Figure 10 This is a schematic block diagram illustrating an example of a computing device performing real-time multi-view video conversion according to embodiments consistent with the principles described herein. The computing device 1000 may include a system of components performing various computing operations for a user of the computing device 1000. The computing device 1000 may be a laptop computer, tablet computer, smartphone, touchscreen system, smart display system, other client device, server, or other computing device. The computing device 1000 may include various components such as, for example, processors (or more) 1003, memory 1006, input / output (I / O) components 1 (or more) 009, a display 1012, and possibly other components. These components may be coupled to a bus 1015, which serves as a local interface to allow the components of the computing device 1000 to communicate with each other. Although the components of the computing device 1000 are shown as being contained within the computing device 1000, it should be understood that at least some components may be coupled to the computing device 1000 via external connections. For example, components can be externally inserted into or otherwise connected to computing device 1000 via external ports, sockets, plugs, connectors, or wireless links.

[0086] Processor 1003 may include a central processing unit (CPU), a graphics processing unit (GPU), any other integrated circuit that performs computational processing operations, or any combination thereof. Processor (or multiple processors) 1003 may include one or more processing cores. Processor (or multiple processors) 1003 includes circuitry for executing instructions. Instructions include, for example, computer code, programs, logic, or other machine-readable instructions received and executed by processor (or multiple processors) 1003 to perform computational functions embodied in the instructions. Processor (or multiple processors) 1003 may execute instructions to manipulate or generate data. For example, processor (or multiple processors) 1003 may receive input data (e.g., an image), process the input data according to an instruction set, and generate output data (e.g., the processed image). As another example, processor (or multiple processors) 1003 may receive instructions and generate new instructions for subsequent execution. Processor 1003 may include components for implementing a graphics pipeline (e.g., ...). Figure 2 The graphics pipeline 134) is used to render video, images, or frames generated by the application. For example, the processor (or multiple processors) 1003 may include one or more GPU cores, vector processors, scaler processes, decoders, or hardware accelerators.

[0087] Memory 1006 may include one or more memory components. Memory 1006 is defined herein as including any one or both of volatile and non-volatile memory. Volatile memory components are those that do not retain information when power is off. Volatile memory may include, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), magnetic random access memory (MRAM), or other volatile memory structures. System memory (e.g., main memory, cache, etc.) may be implemented using volatile memory. System memory refers to fast memory that can temporarily store data or instructions for rapid read and write access to assist processor (or multiple processors) 1003. Images (e.g., still images, video frames) may be stored or loaded in memory 1006 for subsequent access.

[0088] Non-volatile memory components are those components that retain information when power is off. Non-volatile memory includes read-only memory (ROM), hard disk drives, solid-state drives, USB flash drives, memory cards accessed via memory card readers, floppy disks accessed via associated floppy disk drives, optical disks accessed via optical disk drives, and magnetic tapes accessed via suitable magnetic tape drives. ROM can include, for example, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other similar memory devices. Storage memory can be implemented using non-volatile memory to provide long-term retention of data and instructions.

[0089] Memory 1006 can refer to a combination of volatile and non-volatile memory used to store instructions and data. For example, data and instructions can be stored in non-volatile memory and loaded into volatile memory for processing by processor (or multiple) 1003. Execution of instructions can include, for example, machine code translated by a compiler into a format capable of being loaded from non-volatile memory into volatile memory and then executed by processor 1003, source code converted to an appropriate format, such as object code capable of being loaded into volatile memory for execution by processor 1003, or source code interpreted by another executable program to generate instructions in volatile memory and executed by processor 1003, etc. Instructions can be stored or loaded into any part or component of memory 1006, including, for example, RAM, ROM, system memory, storage devices, or any combination thereof.

[0090] Although memory 1006 is shown as separate from other components of computing device 1000, it should be recognized that memory 1006 may be at least partially embedded or otherwise integrated into one or more components. For example, processor(s) 1003 may include onboard memory registers or caches to perform processing operations.

[0091] I / O components (or multiple components) 1009 include, for example, a touchscreen, speaker, microphone, button, switch, dial pad, camera, sensor, accelerometer, or other components that receive user input or generate output directed to the user. I / O components (or multiple components) 1009 may receive user input and convert it into data for storage in memory 1006 or for processing by processors (or multiple components) 1003. I / O components (or multiple components) 1009 may receive data output from memory 1006 or processors (or multiple components) 1003 and convert it into a user-perceived format (e.g., sound, haptic response, visual information, etc.).

[0092] One type of I / O component 1009 is a display 1012. Display 1012 may include a multi-view display (e.g., multi-view display 160), a multi-view display combined with a 2D display, or any other display that presents graphical content. A capacitive touchscreen layer used as I / O component 1009 may be layered within the display to allow the user to provide input while simultaneously perceiving visual output. Processors (or more) 1003 may generate data formatted for presentation as images or frames on display 1012. Processors 1003 (or more) may execute instructions to render images or frames for the user on display 1012. Camera I / O component 1009 may be used for video capture processing that captures video that can be converted into multi-view video.

[0093] Bus 1015 facilitates communication of instructions and data between processors (or more) 1003, memory 1006, I / O components (or more) 1009, display 1012, and any other components of computing device 1000. Bus 1015 may include address translators, address decoders, structures, conductive traces, wires, ports, plugs, sockets, and other connectors to allow communication of data and instructions.

[0094] Instructions within memory 1006 can be embodied in various forms to implement at least a portion of the software stack. For example, instructions can be embodied as operating system 1031, applications (or more) 1034, device drivers (e.g., display driver 1037), firmware (e.g., display firmware 1040), or other software components. Operating system 1031 is a software platform that supports the basic functions of computing device 1000, such as scheduling tasks, controlling I / O components 1009, providing access to hardware resources, managing power, and supporting applications 1034.

[0095] Applications (or more) 1034 execute on operating system 1031 and can access the hardware resources of computing device 1000 via operating system 1031. In this respect, the execution of applications (or more) 1034 is at least partially controlled by operating system 1031. Applications (or more) 1034 may be user-level software programs that provide advanced functions, services, and other capabilities to users. In some embodiments, application 1034 may be a dedicated "app" that is downloadable or otherwise accessible to the user on computing device 1000. Users can launch applications (or more) 1034 via a user interface provided by operating system 1031. Applications (or more) 1034 may be developed by developers and defined in various source code formats. Application 1034 may be developed using several programming or scripting languages, such as C, C++, C#, Objective C, etc. Swift Perl, PHP, Visual Ruby, Go, or other programming languages. Applications (or multiple applications) 1034 can be compiled into object code by a compiler or interpreted by an interpreter for execution by processor 1003. The various embodiments discussed herein can be implemented as at least a portion of application 1034.

[0096] Device drivers, such as display driver 1037, include instructions that allow operating system 1031 to communicate with various I / O components 1009. Each I / O component 1009 may have its own device driver. Device drivers can be installed such that they are stored in storage devices and loaded into system memory. For example, during installation, display driver 1037 translates high-level display instructions received from operating system 1031 into lower-level instructions implemented by display 1012 to display images.

[0097] Firmware (such as, for example, display firmware 1040) may include machine code or assembly code that allows I / O component 1009 or display 1012 to perform low-level operations. Firmware can translate electrical signals of specific components into higher-level instructions or data. For example, display firmware 1040 can control how display 1012 activates individual pixels at a low level by adjusting voltage or current signals. Firmware may be stored in and executed directly from non-volatile memory. For example, display firmware 1040 may be embodied in a ROM chip coupled to display 1012, such that the ROM chip is decoupled from other storage and system memory of computing device 1000. Display 1012 may include processing circuitry for executing display firmware 1040.

[0098] Operating system 1031, application 1034(one or more), driver (e.g., display driver 1037), firmware (e.g., display firmware 1040), and potentially other instruction sets may all include instructions executable by processor 1003(one or more) or other processing circuitry of computing device 1000 to perform the functions and operations discussed above. Although the instructions described herein may be embodied in software or code executed by processor (one or more) 1003 as described above, alternatively, the instructions may also be embodied in dedicated hardware or a combination of software and dedicated hardware. For example, the functions and operations performed by the instructions discussed above may be implemented as circuits or state machines employing any one or a combination of several techniques. These techniques may include, but are not limited to, discrete logic circuits having logic gates for implementing various logical functions when one or more data signals are applied, application-specific integrated circuits (ASICs) with appropriate logic gates, field-programmable gate arrays (FPGAs), or other components.

[0099] In some embodiments, instructions for performing the functions and operations discussed above may be embodied in a non-transitory computer-readable storage medium. The computer-readable storage medium may or may not be part of the computing device 1000. Instructions may include, for example, statements, code, or declarations that can be retrieved from the computer-readable medium and executed by processing circuitry (e.g., processor(s) 1003). In the context of this disclosure, "computer-readable medium" can be any medium capable of containing, storing, or maintaining the instructions described herein, which are used by or in conjunction with an instruction execution system (e.g., computing device 1000).

[0100] Computer-readable media can include any of a number of physical media, such as, for example, magnetic, optical, or semiconductor media. More specific examples of suitable computer-readable media may include, but are not limited to, magnetic magnetic disks, magnetic hard disks, memory cards, solid-state drives, USB flash drives, or optical discs. Furthermore, computer-readable media can be random access memory (RAM), including, for example, static random access memory (SRAM), dynamic random access memory (DRAM), or magnetic random access memory (MRAM). Additionally, computer-readable media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0101] The computing device 1000 can perform any operation or implement any function as described above. For example, the flowcharts and processing flows discussed above can be executed by the computing device 1000, which executes instructions and processes data. Although the computing device 1000 is shown as a single device, the embodiments are not limited thereto. In some embodiments, the computing device 1000 can offload instruction processing in a distributed manner, such that multiple computing devices 1000 operate together to execute instructions that can be stored or loaded in a distributed arrangement of computing components. For example, at least some instructions or data can be stored, loaded, or executed in a cloud-based system in conjunction with the operation of the computing devices 1000.

[0102] Therefore, examples and embodiments of real-time multi-view video conversion have been described. This can be done in real time as a 2D video stream is received. Each view is generated to produce a view for each multi-view frame by determining a target timestamp, identifying temporally adjacent 2D video frames based on the target timestamp, and blending the temporally adjacent 2D video frames. The target timestamp can be determined based on the camera baseline and / or center viewpoint. It should be understood that the above examples are merely illustrative of some of the many specific examples illustrating the principles described herein. It is evident that those skilled in the art will be able to readily devise many other arrangements without departing from the scope defined by the appended claims.

Claims

1. A computer-implemented method for real-time multi-view video conversion, the method comprising: A video stream comprising two-dimensional 2D frames is received by a computing device, each 2D frame corresponding to a corresponding 2D video timestamp; The computing device identifies a camera baseline and a center viewpoint, wherein the camera baseline defines the spacing between views of a multi-view frame, and the center viewpoint is the midpoint of the view range of the multi-view frame; The computing device determines the corresponding target timestamp for each view of the multi-view frame based on the camera baseline and the center viewpoint; The computing device uses the generated target timestamp to generate views in the multi-view frames from a corresponding subset of 2D frames having 2D video timestamps adjacent to the target timestamp for each corresponding view; as well as The computing device renders a multi-view video for display, the multi-view video including the multi-view frames.

2. The method of claim 1, further comprising: The video stream is decoded to extract the 2D frames from the graphics processing unit memory of the computing device.

3. The method of claim 2, further comprising loading the 2D frame from the graphics processing unit memory to the central processing unit memory of the computing device to generate the view.

4. The method of claim 1, further comprising: The multi-view video for display is rendered while the video stream is received by the computing device.

5. The method according to claim 1, further comprising: The computing device receives user input via a user interface, the user input including at least one of the camera baseline and the center viewpoint.

6. The method according to claim 5, further comprising: While the computing device renders the multi-view video, at least one of the camera baseline and the center viewpoint is dynamically updated.

7. The method according to claim 1, wherein, The view is generated by mixing a subset of the 2D frames.

8. The method according to claim 1, wherein, Generating the views in the multi-view frame includes a first view: The computing device determines the time difference between the target timestamp of the first view and the 2D video timestamp adjacent to the target timestamp of the first view; as well as The computing device performs a weighted mixing proportional to the time difference to generate the first view.

9. A system configured to perform real-time multi-view video conversion, the system comprising: Central processing unit; as well as The memory stores multiple instructions, which, when executed, cause the central processing unit to: A series of 2D frames are loaded into a memory buffer, with each 2D frame corresponding to a 2D video timestamp. The corresponding target timestamp for each view of a multi-view frame is determined based on a camera baseline and a center viewpoint, wherein the camera baseline defines the interval between views of the multi-view frame, and the center viewpoint is the midpoint of the view range of the multi-view frame; and Using the generated target timestamp, views in the multi-view frames are generated from a corresponding subset of 2D frames having 2D video timestamps adjacent to the target timestamp for each corresponding view. The view in the multiview frame is part of a multiview video configured to be rendered on a multiview display.

10. The system according to claim 9, further comprising: A graphics processing unit is configured to decode a video stream comprising the series of 2D frames and load the 2D frames into a graphics processing unit memory, wherein the series of 2D frames is configured to be loaded from the graphics processing unit memory into the memory buffer.

11. The system according to claim 10, wherein, The multi-view video is configured to be rendered on the multi-view display while the video stream is being decoded.

12. The system according to claim 9, wherein, The instruction also causes the central processing unit to: Generate a user interface configured to be displayed on the multi-view display; and User input is received via the user interface, and the user input includes at least one of the camera baseline and the center viewpoint.

13. The system according to claim 12, wherein, The instruction also causes the central processing unit to: At least one of the camera baseline and the center viewpoint is dynamically updated while the multi-view video is being rendered.

14. The system according to claim 9, wherein, The instruction also causes the central processing unit to: Generate a user interface configured to be displayed on the multi-view display; and In response to user input received via the user interface, switching is made between rendering the multi-view video and rendering the 2D video stream.

15. The system according to claim 9, wherein, The view is generated by blending a subset of the 2D frames.

16. The system according to claim 9, wherein, The instruction also causes the central processing unit to view the first view in the multi-view frame: Determine the time difference between the target timestamp of the first view and the 2D video timestamp adjacent to the target timestamp of the first view; and A weighted blending proportional to the time difference is performed to generate the first view.

17. A non-transitory computer-readable storage medium storing executable instructions that, when executed by a processor of a computer system, perform a multi-view video conversion operation in real time, the operation comprising: Receive a video stream consisting of a series of two-dimensional 2D frames, each 2D frame corresponding to a corresponding 2D video timestamp; The target timestamp for each view in a multi-view frame is determined based on the camera baseline and the center viewpoint, wherein the target timestamp for each corresponding view is adjacent to a set of 2D video timestamps corresponding to a set of 2D frames, wherein the camera baseline defines the interval between views of the multi-view frame, and the center viewpoint is the midpoint of the view range of the multi-view frame. Each set of 2D frames is blended to generate the corresponding view in the multi-view frame; as well as While receiving the video stream, a multi-view video for display is rendered, the multi-view video including the multi-view frames.

18. The non-transitory computer-readable storage medium according to claim 17, wherein, The operation also includes: The target timestamp is determined based on at least one of a user-specified camera baseline and a user-specified center viewpoint.

19. The non-transitory computer-readable storage medium according to claim 17, wherein, The operation also includes targeting the first view in the multi-view frame: Determine the time difference between the target timestamp of the first view and the 2D video timestamp adjacent to the target timestamp of the first view; as well as A weighted blending proportional to the time difference is performed to generate the first view.

20. The non-transitory computer-readable storage medium according to claim 17, wherein, The operation also includes: In response to user input received via the user interface, the system switches between rendering the multi-view video and rendering the 2D video stream.

Citation Information

Patent Citations

  • Method and system for creating three-dimensional viewable video from a single video stream

    US20120013711A1