Method and system for multiview video rendering, multiview cache preparation, and real-time multiview video transformation - Patents.com

Abstract

The method and system for multiview video rendering, multiview video cache preparation, and real-time multiview video conversion employs video information having multiview frames corresponding to multiview video timestamps to generate views of the multiview frames using cache data entries of the multiview video cache corresponding to target timestamps. The cache data entries may include a first cache data entry corresponding to a first multiview frame before the target timestamp and a second cache data entry corresponding to a second multiview frame after the target timestamp. The target timestamp for a particular multiview frame to be rendered may be based on user input. The system may include a processor circuit and a memory storing a plurality of instructions that, when executed by the processor circuit, cause the processor circuit to perform one or both of multiview video rendering and multiview video cache preparation.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 63 / 295,861, filed January 1, 2022, which is incorporated by reference in its entirety.

[0002] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT Not applicable. [Background technology]

[0003] A video stream includes a series of frames, where each frame may be made up of a two-dimensional (2D) image. The video stream may be compressed according to a video encoding specification to reduce the video file size and alleviate network bandwidth. The video stream may be received by a computing device from a variety of sources. The video stream may be decoded and rendered for display by a graphics pipeline. Displaying the video occurs by rendering these frames at a particular frame rate.

[0004] The emergence of display technologies such as holographic displays, three-dimensional (3D) displays, volumetric displays, light field displays, and other similar displays may provide a more immersive viewing experience than traditional 2D video displays. Techniques for converting 2D video into different formats may involve pre-processing efforts before the video can be rendered.

[0005] Various features of examples and embodiments according to the principles described herein may be more readily understood by reference to the following detailed description taken in conjunction with the accompanying drawings, in which like reference numerals indicate like structural elements and in which: [Brief description of the drawings]

[0006] [Figure 1]1 provides an overview of a process for generating multi-view video, according to an embodiment consistent with principles described herein. [Diagram 2] 1 illustrates an example of a multiview video and a corresponding multiview video cache, according to an embodiment consistent with principles described herein. [Diagram 3] 1 illustrates an example of a multiview frame and corresponding multiview video cache data entries, according to an embodiment consistent with principles described herein. [Figure 4] 1 illustrates an example of rendering multi-view video for display according to an embodiment consistent with principles described herein. [Diagram 5] 1 illustrates an example of using a multi-view video cache, according to an embodiment consistent with principles described herein. [Figure 6] 1 illustrates an example of blending information from multiple cache data entries according to an embodiment consistent with principles described herein. [Figure 7] 1 shows a flowchart of a method for generating a multi-view video cache, according to an embodiment consistent with principles described herein. [Figure 8] 1 shows a flowchart of a method for using information from a multiview video cache, according to an embodiment consistent with principles described herein. [Figure 9] 1 shows a flowchart of a method for providing or using a user interface for controlling timeline scrubbing for multiview video, according to an embodiment consistent with principles described herein. [Figure 10] 1 shows a schematic block diagram of a computing device configured to perform real-time multi-view video conversion in an example according to an embodiment consistent with principles described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Certain examples and embodiments have other features in addition to or in place of those shown in the above-referenced drawings. These and other features are described in detail below with reference to the above-referenced drawings.

[0008] Examples and embodiments according to principles described herein provide techniques for creating and using a multiview video cache. Among other benefits, the cache helps reduce latency, which may improve the user experience in video timeline searching, sometimes referred to as "timeline scrubbing," during playback of a multiview video using a multiview display. Embodiments are directed to generating and registering a multiview video cache having cache data entries corresponding to each different multiview video timestamp. Each cache data entry may include a copy of a two-dimensional (2D) frame for a particular timestamp in the multiview video. According to embodiments, the cache may be generated by pre-processing the multiview video.

[0009] According to an embodiment, a multiview video rendering system may be configured to receive an indication of a target timestamp for a particular multiview frame to render, for example based on user input indicating a timestamp of a particular time within the multiview video. A pair of cache data entries corresponding to the target timestamp may be retrieved from the multiview video cache, and the cache data entries may be used together to generate a multiview frame corresponding to the target timestamp. The pair of cache data entries corresponding to the target timestamp may include an entry that is closest in time to an entry that precedes the target timestamp and another entry that is closest in time to an entry that follows the target timestamp. A graphics pipeline may receive image information from the cache data entries and blend the image information, thereby generating a new multiview frame for display. Processing efficiencies are achieved by storing image information for multiple different views of a multiview image in a single data element or data location such that multiple frame data accesses may be avoided. Further processing efficiencies may be achieved by combining or blending information from multiple different cache data entries in substantially real time, rather than establishing a new echo buffer each time a new target timestamp is indicated for playback.

[0010] FIG. 1 provides an overview of a process for converting a two-dimensional (2D) video stream into a multi-view video according to an embodiment consistent with the principles described herein. The video capture process 103 includes capturing a video of a scene 105. The scene 105 may be a physical arrangement of objects of interest to a viewer. The video capture process 103 may take place in a studio or any other physical environment. The objective is to visually capture the scene 105 using a camera 107. The camera 107 may capture the video, which is composed of a series of time-sequential images (called frames). The camera may move along a particular direction 109 relative to the scene to capture different angles of the scene 105. The direction 109 may be primarily linear or circular to obtain different viewing angles of the scene.

[0011] The camera 107 may be mounted on a vehicle that moves relative to the scene 105. The vehicle may be a drone or other unmanned aerial vehicle (UAV). In some examples, the scene 105 may include one or more objects placed on a turntable that rotates relative to the fixed camera 107. The video capture process 103 ultimately converts the visual details of the scene occupying a physical space into a 2D video stream 112. The 2D video stream may be composed of a series of time-sequenced 2D frames 115. Each 2D frame 115 represents a single view of the scene at a particular point in time. The time may be recorded as a timestamp unique to the corresponding 2D frame. Rendering the 2D frames in the sequence at a target frame rate results in the 2D video perceived by the viewer.

[0012] Although the 2D video stream 112 is described as a video of a physical scene, it should be understood that the 2D video stream 112 may be generated by moving a virtual camera relative to a 3D model. For example, gaming engines and other 3D model generators may enable the creation of 2D video by applying a virtual camera (defined in terms of position and orientation) to a 3D model (e.g., a volumetric model of the scene defined in terms of a point cloud, surface mesh, texture map, etc.).

[0013] The camera 107 may transmit or upload the 2D video stream to a repository over a network. The repository may include memory storage, a server, cloud-based storage, or other computing system configured to receive and store video files. The network may include the Internet, an intranet, a wireless network, a wired network, or any other network that facilitates the flow of data. The 2D video stream 112 may be encoded according to a particular encoding format. The encoding includes applying lossy or lossless compression to reduce the file size to fit the network bandwidth. The video encoding format may be, for example, the H.264 format, the H.265 format, any format specified by the Moving Picture Experts Group (MPEG), etc. Additionally, the 2D video stream 112 may be packetized for transmission over an Internet Protocol (IP) network.

[0014] The multiview video 118 may be generated from the 2D video stream 112 using a multiview conversion process 119. The multiview conversion process 119 is a computer-implemented process that converts the 2D video stream into a multiview video 118 that can be rendered for display. The multiview conversion process 119 may be implemented as a software program, routine, or module executable by a processor. The multiview conversion process 119 may receive the 2D video stream 112 as a compressed, packetized data stream. For example, the multiview conversion process 119 may retrieve the 2D video stream 112 from a repository using an application programming interface (API) or other function calls. The 2D video stream may be live video, such as when the video capture process 103 transmits the 2D video stream as it is capturing video of the scene 105, in which case the 2D video is streamed to the computing system implementing the multiview conversion process. In some embodiments, the 2D video stream 112 comprises a pre-recorded (not live) video configured to be streamed to a computing device upon request.

[0015] The multiview video 118 may be stored in memory as one or more computer files or formats. The multiview video 118 includes a series of multiview frames 121, denoted as multiview frame A through multiview frame N. The multiview frames 121 may have respective timestamps and are displayed chronologically at a specified frame rate, expressed as frames per second (FPS). The multiview frames 121 thus include a multiview image. Each multiview frame 121 is composed of multiple views 124. For example, multiview frame A is denoted as having views 1 through n, which may be similarly true for each multiview frame 121 in the multiview video 118. In some embodiments, when rendered on a multiview display, each view 124 of the multiview frame 121 corresponds to a respective principal angular direction of a surface normal to the multiview display. As a result, multiple different views 124 may be perceived simultaneously by a viewer. This results in a three-dimensional (3D) output image or light field image content. An illustrated example of a multi-view display and different view directions is given in FIG.

[0016] According to an embodiment, a scene 105 may be captured or recorded from different view directions simultaneously by multiple cameras 107. Multi-view frames 121 of a multi-view video 118 may thus be provided directly based on the captured image content and without the multi-view transformation process 119 described above for original 2D content.

[0017] A "two-dimensional display" or "2D display" is defined herein as a display configured to provide a view of an image that is substantially the same regardless of the direction from which the image is viewed (i.e., within a defined viewing angle or range of the 2D display). A conventional liquid crystal display (LCD) found on many smartphones and computer monitors is an example of a 2D display. In contrast, a "multi-view display" is defined herein as an electronic display or display system configured to provide different views of a multi-view image to or from different viewing directions simultaneously from a user's viewpoint. In particular, the different views may represent different perspective views of a particular multi-view frame 121.

[0018] As used herein, a "timestamp" is defined as a value indicating a relative or absolute time in any unit of time (e.g., seconds, days / hours, etc.). Each frame (e.g., each multiview frame, or each 2D frame that comprises a multiview frame) may include a timestamp that organizes the frames in a chronological order. According to an embodiment, the multiview frames of a multiview video 118, when rendered in order by timestamp, comprise a multiview video program.

[0019] As used herein, the article "a" is intended to have its ordinary meaning in the patent art, namely, "one or more." For example, "processor" means one or more processors, and similarly, "memory" herein means "one or more memory components."

[0020] A multi-view display may be implemented using a variety of techniques that support the presentation of different image views so that they are perceived simultaneously. One example of a multi-view display employs a diffraction grating to control the predominant angular direction of light that comprises the different views. According to some embodiments, the multi-view display may be a light field display that presents multiple light beams of different colors and different directions that correspond to the different views. In some examples, the light field display is a so-called "glasses-free" three-dimensional (3D) display that may use a diffraction grating to provide an autostereoscopic representation of the multi-view image without the need for dedicated eyewear to perceive depth. In some embodiments, the multi-view display may require glasses or other eyewear to control which view is perceived by each eye of the user.

[0021] Embodiments of the principles described herein are directed to a computer-implemented method of multi-view video rendering, which may alternatively be characterized as a computer-implemented method of multi-view image cache generation and usage. The computer-implemented method of multi-view video rendering may be configured to reduce latency in timeline scrubbing operations, according to some embodiments. Figure 10, described in more detail below, provides an example of a computing device configured to perform one or both of the computer-implemented method of multi-view video rendering or utilize a multi-view image cache generated by the computer-implemented method of multi-view video rendering.

[0022] According to various embodiments, a computer-implemented method for multiview video rendering includes receiving, using a computing device, video information having a plurality of multiview frames, each multiview frame of the plurality of multiview frames in the received video information including a plurality of two-dimensional (2D) frames corresponding to a respective multiview video timestamp and corresponding to a different view of a scene within the multiview frame.

[0023] The computer-implemented method of multiview video rendering further includes determining a target timestamp for a particular multiview frame to render based on. In various embodiments, the particular multiview frame may be determined based on user input. The computer-implemented method of multiview video rendering further includes retrieving a pair of cache data entries from a cache (e.g., a multiview video cache) that correspond to the target timestamp. A first cache data entry of the pair of cache data entries retrieved from the cache includes a respective first image frame group corresponding to a first multiview frame that is earlier than the target timestamp, and a second cache data entry of the pair of cache data entries includes a respective second image frame group corresponding to a second multiview frame that is later than the target timestamp. In some embodiments, retrieving the pair of cache data entries from the cache includes identifying two cache data entries from the cache that have a timestamp that is closest in time to the target timestamp.

[0024] The computer-implemented method of multi-view video rendering further includes generating a view of the particular multi-view frame using the first group of image frames and the second group of image frames from the cache, and rendering a multi-view video for display by the computing device, the multi-view video including the view of the particular multi-view frame.

[0025] In some embodiments, the computer-implemented method of multiview video rendering may further include generating a cache having cache data entries corresponding to each different timestamp. Generating the cache includes selecting a number of multiview frames that is less than all of the video information. Generating the cache may further include registering in the cache cache data entries corresponding to each different multiview video timestamp. In particular, generating a cache having cache data entries corresponding to each different timestamp may include at least one of the cache data entries including an image frame group representing a number of views that is less than all of the corresponding multiview frames. Furthermore, each cache data entry may include a copy of a 2D frame for a particular respective multiview frame timestamp. According to some embodiments, the number of cache data entries for a particular multiview video program may be a number that is at least a magnitude less than the number of multiview frames in the received video information.

[0026] In various of these embodiments, each cache data entry may include a tiled image having tiles each corresponding to a different 2D frame of a respective multiview frame. In some embodiments, tiles in a tiled image may include copies of the corresponding 2D frames themselves. In other embodiments, tiles of a tiled image may include low-resolution copies of corresponding 2D frames from the video information. In some embodiments, cache data entries in the cache may have timestamps that are not temporally adjacent. In some embodiments, the number of cache data entries in the cache may be based on a timeline scrubbing resolution. In some embodiments, generating the cache includes using 2D frames corresponding to fewer than all of the multiview frames of the video information.

[0027] In other embodiments, the computer-implemented method of multiview video rendering may further include generating a cache having cache data entries corresponding to respective different timestamps, in these embodiments, at least one of the cache data entries may include an image frame group representing fewer than all of the views of the corresponding multiview frame.

[0028] In some embodiments, generating a view of the particular multi-view frame includes using a virtual camera. In some embodiments, generating a view of the particular multi-view frame includes generating a respective view using 2D frames from each of a first cache data entry and a second cache data entry corresponding to the particular multi-view frame. In some of these embodiments, generating the respective view includes blending information from the 2D frames from each of the first cache data entry and the second cache data entry. Blending the information may include, for example, weighting the influence of each of the 2D frames in proportion to a time difference of each of the cache data entries relative to a target timestamp.

[0029] Another embodiment of the principles described herein is directed to a computer-implemented method of preparing a multiview video cache. The computer-implemented method of preparing a multiview cache includes receiving, by a computing device, multiview video information including two-dimensional (2D) frames. According to various embodiments, each 2D frame corresponds to a respective view of the multiview video and corresponds to a multiview video timestamp. The computer-implemented method of preparing a multiview video cache further includes registering cache data entries in a cache, each corresponding to a different multiview video timestamp. Each cache data entry includes a copy of the 2D frame for a particular respective timestamp. In some embodiments, the cache data entries correspond to multiview video timestamps that are not temporally adjacent. In some embodiments, at least one of the cache data entries includes fewer than all of the 2D frames from the video information for a particular timestamp.

[0030] In some embodiments, the computer-implemented method of preparing a multiview video cache further includes generating a first cache data entry for the cache corresponding to a first timestamp. In these embodiments, the first cache data entry includes a tiled image, with tiles of the tiled image corresponding to each of the copies of the 2D frame for the first timestamp. In some embodiments, the tiles may be low-resolution copies of the 2D frame for the first timestamp.

[0031] Yet another embodiment of the principles described herein is directed to using a multiview video cache. Using the multiview video cache may include retrieving a pair of cache data entries from the cache that correspond to a target timestamp. A first cache data entry and a second cache data entry of the pair may include a first image frame group and a second image frame group, respectively, where the first image frame group corresponds to a temporally closest first multiview frame before the target timestamp and the second image frame group corresponds to a temporally closest second multiview frame after the target timestamp. Using the multiview video cache may further include generating a multiview frame using information from the first image frame group and the second image frame group, for example, by blending information from each of the frame groups to generate a new multiview image frame for the target timestamp.

[0032] 2 illustrates a multiview video 118, a multiview video cache 206, a graphics pipeline 212, and an output buffer 218. The multiview video 118 may include a number of multiview video frames having respective timestamps. Together, the multiview frames of the multiview video 118, when rendered in order by timestamp, comprise a portion of a multiview video program. In the example of FIG. 2, the multiview video 118 includes a first multiview frame or multiview frame A 202 corresponding to a first timestamp, a last multiview frame or multiview frame N 204 corresponding to an Nth timestamp, and a number of other multiview frames having respective timestamps between the first timestamp and the Nth timestamp.

[0033] According to an embodiment, a number of cache data entries may be registered in the multiview video cache 206. Each cache data entry may include a group of images or a group of image frames and has a particular timestamp. For example, the group of images may correspond to different 2D views of a multiview frame for a particular timestamp. In general, the timestamp of a particular cache data entry corresponds to the timestamp for a particular frame of the multiview video 118. However, there may be fewer cache data entries present in the multiview video cache 206 than there are multiview frames present in the multiview video 118. In other words, fewer than all of the frames in the multiview video 118 may correspond to a data entry in the multiview video cache 206. Various attributes of a cache data entry, such as cache entry A 208 or cache entry N 210, are further described below in the example of FIG. 3.

[0034] According to an embodiment, certain frames of the multiview video 118 may be selected for use in or inclusion in one or more image frame groups and stored in the multiview video cache 206. The number of frames of the multiview video 118 selected for use in the multiview video cache 206 and the timestamp difference between adjacent image frame groups in the multiview video cache 206 may affect the timeline scrubbing resolution. For example, if the timeline scrubbing resolution is greater, relatively more frames of the multiview video 118 may be selected for inclusion in the multiview video cache 206. Thus, a greater resolution timeline scrub refers to a multiview video cache 206 that includes more cache data entries or image frame groups for an overall number of program timestamps than a lesser resolution timeline scrub has for the same number of program timestamps. In general, the cache data entries may have timestamps that are relatively evenly (e.g., periodically) distributed throughout the timeline of the multiview video 118, although other distributions of image frame groups may be used. For example, cache data entries may be more focused on program content that is of most interest to the viewer.

[0035] The multiview video cache 206 may be defined as an address space that is a portion of a memory space accessed by a processor circuit. The processor circuit may be a general processor that executes instructions, supports an operating system, and provides applications (e.g., user-level applications). The memory circuit may be system memory accessible to an operating system or applications executing on the computing device. The computing device may include a graphics pipeline 212 that may receive video frame information, for example, from the multiview video 118 or the multiview video cache 206.

[0036] A "graphics pipeline" as used herein is defined as a computer-implemented environment that renders image data for display. The graphics pipeline 212 may include one or more graphics processing units, i.e., GPUs 214, or other specialized processing circuitry optimized for rendering image content to a screen. For example, the GPUs 214 may include a vector processor that executes a set of instructions that operate in parallel on an array of data. The graphics pipeline 212 may include a graphics card, a graphics driver, or other hardware and software used to render graphics. The graphics pipeline 212 may be configured to render images on a multi-view display. The graphics pipeline 212 may map pixels onto corresponding locations on a display and control the display to emit light to display the rendered image. The graphics pipeline 212 may also include a GPU memory 216. The GPU memory 216 may include one or more memory devices used by the graphics pipeline 212.

[0037] In some embodiments, the graphics pipeline 212 is a subsystem separate from one or more other processor circuits of the computing device. In some embodiments, the graphics pipeline 212 is implemented solely as software by the processor circuitry. For example, the processor circuitry may execute software modules that operate as the graphics pipeline 212 without dedicated graphics hardware. In some embodiments, parts of the graphics pipeline 212 are implemented in dedicated hardware and other parts are implemented as software modules.

[0038] The graphics pipeline 212 may be used to decode the video stream to extract video frames in the GPU memory 216. For example, the graphics pipeline 212 may decode a 2D or 3D video stream to identify frames according to an encoding format. For example, in an MPEG encoding format, the frames may be formatted as I-frames, P-frames, B-frames, etc. The graphics pipeline 212 may store the decoded frames in the GPU memory 216. These decoded 2D frames may then be loaded into the output buffer 218, such as using a frame dumping process. Although FIG. 2 illustrates the graphics pipeline 212 that may perform the decoding and frame extraction, it should be understood that a dedicated hardware or software decoder may be used to load the frames into the output buffer 218 after decoding.

[0039] 3 illustrates an example of a multiview video cache 206 according to an embodiment consistent with principles described herein. The multiview video cache 206 may include multiple cache data entries with different timestamps corresponding to different portions of a multiview video.

[0040] In the example of FIG. 3, the multiview video cache 206 includes a first cache data entry 302 with a timestamp of 0, a second cache data entry 304 with a timestamp of 100, a third cache data entry 306 with a timestamp of 200, and a fourth cache data entry 308 with a timestamp of 300. Additional or fewer cache data entries may be used in the multiview video cache 206 as well, depending on the desired timeline scrubbing resolution, etc. The first through fourth cache data entries of the multiview video cache 206 are periodically spaced one hundred (100) timestamp units apart. Similarly, other periodic or non-periodic intervals between cache data entries may be used. The particular timestamp values ​​described herein are representative or nominal values ​​selected for illustrative purposes.

[0041] According to an embodiment, each cache data entry in the multiview video cache 206 includes a group of images having the same timestamp. For example, the group of images may correspond to different views or 2D frames from the same multiview frame. FIG. 3 diagrammatically illustrates a particular multiview frame 310 corresponding to a timestamp 200 and its relationship to the third cache data entry 306. The particular multiview frame 310 includes four separate and separate frames or views (e.g., 2D View 1-2D View 4) corresponding to different view directions at the timestamp 200. According to an embodiment, a copy of each of View 1-View 4 from the multiview frame 310 may be included in the third cache data entry 306.

[0042] The groups of images within a cache data entry may be stored in various configurations or formats. According to an embodiment, the images within each group may be combined to form a macro image, which may be stored as a cache data entry in an image file format or the like. According to further embodiments, the macro image may include an image composed of multiple tiled images. For example, the third cache data entry 306 may be a macro image in an image file format, the macro image including multiple tiles each corresponding to a view within a particular multi-view frame 310. According to the embodiment of FIG. 3, each cache data entry includes a different macro image of four tiled images. Additional or fewer tiled images may be used as well. The number of tiles may depend, for example, on the number of views within the multi-view frame being cached.

[0043] According to an embodiment, to further reduce the size of the cache data entry, the number of tiles in a particular cache data entry may be less than the number of views in the multiview video. In this embodiment, any "missing" tiles may be generated, for example, by applying interpolation to other tiles in the particular cache data entry or by blending information from other tiles. Other techniques, such as using a virtual camera, may be used to generate the tile information as well.

[0044] According to an embodiment, the copy of a view from a particular multiview frame 310 may be a lower resolution copy of the source multiview frame. That is, the resolution of the image or macro image stored in the cache data entry may be lower than its corresponding source image. Thus, a cache data entry for a multiview frame 310 may contain less data than the multiview frame 310. In another embodiment, a cache data entry for a multiview frame 310 may contain less data than one view of the multiview frame 310.

[0045] Combining views into a tiled macro image may be convenient in terms of computation time and other resource handling. However, other cache data entry configurations may be used. For example, different image files for a particular cache data entry may be stored in consecutive cache memory addresses. In another example, a particular cache data entry may include a reference image and a vector map that defines a view image for a particular timestamp. Other techniques may be used as well.

[0046] Figure 4 illustrates an example of rendering a multiview video for display according to an embodiment consistent with principles described herein. For example, Figure 4 illustrates an example of rendering a multiview video 118 for display, where the multiview video 118 includes multiview frames 121, each including a respective set of different views 124 (e.g., views 1 through 4 in an example four-view multiview display).

[0047] The example of FIG. 4 includes a user interface 406 that may be used to communicate with or provide instructions to the graphics pipeline 212, which in turn may use the multiview video 118 and / or information from the multiview video cache 206 to prepare a viewable output.

[0048] As described above, the application may provide a multiview video transform to generate a multiview video that includes multiview frames 121, and may optionally include or use multiview frames generated using one or more cache data entries from multiview video cache 206. Upon generating the multiview video, the application may interface with graphics pipeline 212 to render the multiview video that includes multiview frames 121 on multiview display 160. Graphics pipeline 212 may map each view 124 of each multiview frame 121 to a corresponding pixel location on multiview display 160. As a result, each view 124 may correspond to a respective principal angular direction 161.

[0049] The user interface 406 may include a timeline 404 representing a period of the multiview video 118. According to an embodiment, the user interface 406 may be configured to receive user input indicating a target timestamp 402 on the timeline 404 at which to begin playback. According to some embodiments, a user may search or "scrub" the timeline 404 by moving the target timestamp 402 to any position along the timeline 404. Information regarding the target timestamp 402 may be provided to the graphics pipeline 212, which in response may identify the multiview frame or frames required to begin playback at or near the target timestamp 402. In some embodiments, if the graphics pipeline 212 acquires multiple full resolution image frames to establish sufficient content in the output buffer 218, a relatively long latency may occur between the user input to the user interface 406 and the resulting output from the multiview display 160. According to embodiments described herein, latency may be reduced using information from multiple cache data entries in the multi-view video cache 206, where the multiple cache data entries are closest in time (e.g., before and after) to the target timestamp 402.

[0050] 4 provides an example of a multi-view display 160 and user interface 406, other implementations may be used to display the multi-view images or receive user input. For example, some implementations may require specialized glasses to perceive the different views, or other graphical representations of the timeline 404 may be used.

[0051] Figure 5 illustrates an example of using multiview video cache 206 according to an embodiment consistent with principles described herein. In the example of Figure 5, a user input may indicate a timeline scrub to a target timestamp of 70. The graphics pipeline 212 may receive information regarding the target timestamp and access cached data entries from the multiview video cache 206 to initiate a multiview video transformation or playback of the multiview video streams that begins at the target timestamp or is perceived by the user to begin at the target timestamp.

[0052] The multiview video cache 206 may or may not include a cache data entry for a target timestamp of 70. In the example of FIG. 5, the multiview video cache 206 does not include a cache data entry with timestamp 70. Thus, a pair of cache data entries corresponding to the target timestamp may be selected by the graphics pipeline 212. The graphics pipeline 212 may select the first cache data entry 302 with timestamp 0 because it is closest in time to and before the target timestamp of 70. The graphics pipeline 212 may select the second cache data entry 304 with timestamp 100 because it is closest in time to and after the target timestamp of 70. The first cache data entry 302 may include a first group of image frames corresponding to a first multiview frame that is earlier than the target timestamp, and the second cache data entry 304 may include a second group of image frames corresponding to a second multiview frame that is later than the target timestamp.

[0053] Multiview video transformation using cache data entries includes generating new multiview frame views from images in the first cache data entry 302 and the second cache data entry 304. For example, each view of the new multiview frame is generated from a respective portion of the macro image stored in the pair of cache data entries. The portions may include or represent at least a pair of 2D frames that are adjacent (or closest in time) to the target timestamp.

[0054] For example, to generate a first view, or view 1, the target timestamp is 70 time units. The first cache data entry 302 and the second cache data entry 304 each contain a macro image that includes tiles corresponding to view 1 at timestamps earlier and later than the target timestamp 70, respectively. FIG. 6 illustrates a first tile 602 that is part of the macro image stored as the first cache data entry 302, and a second tile 604 that is part of the macro image stored as the second cache data entry 304. In FIG. 6, the tiles are alphanumeric with a first letter indicating the corresponding view (e.g., view 1, 2, 3, or 4) and a second letter indicating the corresponding cache data entry (e.g., "A" for the first cache data entry 302, "B" for the second cache data entry 304, etc.). In the example of FIG. 6, a first tile 602 and a second tile 604 each correspond to the top left portion of a respective macro image in the cache, although different tiles or image portions or arrangements may be used as well.

[0055] The graphics pipeline 212 may receive a first tile 602 (Tile 1A) and a second tile 604 (Tile 1B) and may use the tile information together to generate a first new view 606 corresponding to a target timestamp of 70 for a first viewing direction (View 1). Similarly, the graphics pipeline 212 may generate a different new view for a second viewing direction (View 2), etc., at timestamp 70, using Tile 2A from the first cache data entry 302 and Tile 2B from the second cache data entry 304, etc. The newly generated view may provide initial conditions, such as depth information, to the graphics pipeline 212 for use in a new (or “scrubbed”) playback stream that effectively begins at the target timestamp.

[0056] According to an embodiment, a new view may be generated by combining or using together tiles from cache data entries. For example, a new view may be generated by blending image portions. In some embodiments, the blending may be a linear blend or an extrapolation. In other embodiments, the blending may be a motion-based blend. The blending may include a pixel-by-pixel blending operation. Examples of using blending operations to generate a new view from different image portions are further described below.

[0057] According to an embodiment, the multiview video transformation may also include rendering the multiview video for display, where the multiview video includes the newly generated views based on the cache data entries. For example, upon generating the new views, the views may be formatted into multiview frames for the multiview video in a predefined video format.

[0058] FIG. 6 provides an example of a first new view 606 generated by blending a subset of image portions. In particular, FIG. 6 illustrates an example of performing a weighted blend proportional to the time difference of temporally adjacent image portions to generate a view, according to an embodiment consistent with principles described herein. A "weighted blend" is defined herein as a sum or other combination of two variables (e.g., pixel values), where each variable is weighted according to a respective coefficient (e.g., weight). For example, to generate view 1, a target timestamp is determined for view 1, which is 70 time units in FIG. 6. A subset of temporally adjacent image portions is identified from the cache data entries based on the target timestamp, which are tile 1A and tile 1B in the example of FIG. 6. The weighted blend may be proportional to the time difference between the temporally adjacent image portions (e.g., between tile 1A and tile 1B). The coefficient in the weighted blend corresponds to the time difference. For example, the blending operation may include determining a time difference between a target timestamp (70) and cache data entry timestamps adjacent to the target timestamp (e.g., timestamps 0 and 100). For example, view 1 may have a time difference of 70 time units with respect to tile 1A and a time difference of 30 time units with respect to tile 1B. Thus, view 1 is quantitatively closer to tile 1B than to tile 1A. Upon determining these time differences, view 1 may be generated by performing a weighted blend in proportion to the time difference to generate the first new view 606. This may include performing a linear pixel blend weighted according to the time difference. As a result, the first new view 606 may appear to be more similar to the image portion of tile 1B than to the image portion of tile 1A.

[0059] According to an embodiment, motion-based blending may be used to generate new views according to embodiments consistent with principles described herein. For example, using motion estimation techniques, optical flow between image portions of a first tile 602 and image portions of a second tile 604 may be determined. Optical flow is a set of data that includes vectors that represent how features or image objects move from an initial frame to a subsequent frame. In this regard, optical flow encompasses the velocity of different objects or pixel groupings from one frame to the next. Optical flow may be formatted as a matrix of vectors. Once the optical flow between tiles is determined, new views may be generated based on blending these temporally adjacent image portions using the optical flow.

[0060] Figure 7 shows a flowchart of a first method 700 of generating a multiview video cache according to an embodiment consistent with principles described herein. Figure 8 shows a flowchart of a second method 800 of using information from a multiview video cache according to an embodiment consistent with principles described herein. Figure 9 shows a flowchart of a third method 900 of providing or using a user interface for controlling timeline scrubbing for multiview video according to an embodiment consistent with principles described herein.

[0061] The flowcharts of Figures 7, 8, and 9 provide various examples of functions that may be performed by a computing device executing a set of instructions. The flowcharts may be viewed as illustrating examples of elements of a computer-implemented method according to one or more embodiments. The flowcharts may also be viewed as illustrating the operation of a non-transitory computer-readable storage medium that stores executable instructions that, when executed by a processor circuitry of a computer system, cause the computer system to perform operations for multi-view video processing or transformation.

[0062] 7, at block 702, operations of a first method 700 may include receiving multiview video information including two-dimensional (2D) frames. According to an embodiment, each 2D frame corresponds to a respective view of the multiview video or multiview frames. Each multiview frame, and accordingly each corresponding 2D frame, may have an associated multiview video timestamp.

[0063] At block 704, the operations may include receiving a video timeline scrub resolution indicator. The scrub resolution indicator may be a preset value or may be a value specified by a user. The scrub resolution indicator may specify a resolution or granularity at which timeline scrubbing is available. Higher resolution scrubbing may include or use a larger or higher resolution cache for multiview video playback.

[0064] At block 706, the operations may include allocating a multiview video cache. Allocating the cache may include allocating a data store or data structure in a computing device resident memory or a processor resident memory. According to an embodiment, attributes such as size, location, or format, among others, may be defined at cache allocation time. The size of the cache may depend, for example, on the resolution indicator received at block 704.

[0065] At block 708, the operations may include selecting a cache timestamp interval based on the resolution indicator received at block 704. The cache timestamp interval may indicate how often cache data entries are registered throughout all or part of the multiview video. A smaller interval corresponds to a higher frequency and corresponds to higher resolution timeline scrubbing. According to an embodiment, the timestamp interval is greater than the timestamp unit interval. The cache data entries correspond to multiview video timestamps that are not temporally adjacent when the timestamp interval is greater than the unit interval. According to an embodiment, the cache timestamp interval may be automatically optimized based on system resources or performance capabilities, or playback quality requirements (e.g., specified by a user or a user profile).

[0066] In block 710, the operations may include registering or generating a first cache data entry for a first timestamp of the multiview video. The first cache data entry may include a group of images, such as a macro image or a group of images that may be tiled in other image formats. The tiles in the cache data entry may be copies or representations of some or all of the 2D frames of the multiview frame having the first timestamp. According to an embodiment, the tiles may be low-resolution copies of the source images.

[0067] At block 712, the operations may include registering or generating a second cache data entry for a second timestamp of the multiview video. The second timestamp may be spaced in time from the first timestamp by a timestamp interval (e.g., selected at block 708). The second cache data entry may include a group of images, such as macro images or other image formats that may be tiled. The tiles in the cache data entry may be copies or representations of some or all of the 2D frames of the multiview frame having the second timestamp.

[0068] 8 illustrates a flowchart of a second method 800 of using information from a multiview video cache, according to an embodiment consistent with principles described herein. At block 802, operation of the second method 800 may include determining a target timestamp for a particular multiview frame to render. According to an embodiment, the particular multiview frame to render may be based on user input or instructions, such as may be received by a user interface.

[0069] At block 804, the operations may include retrieving a pair of cache data entries from the multiview video cache that correspond to the target timestamp determined at block 802. According to an embodiment, the first and second cache data entries of the pair include a respective first and second image frame group, where the first image frame group corresponds to a first multiview frame that is earlier than the target timestamp and the second image frame group corresponds to a second multiview frame that is later than the target timestamp. According to an embodiment, each image frame group includes a macro image or a collection of multiple images in one file or location. For example, the macro image may be a tiled image.

[0070] In block 806, the operations may include generating a view of the particular multi-view frame using a pair of cache data entries. For example, block 806 may include using a first group of image frames and a second group of image frames from the cache. According to an embodiment, block 806 may include applying various techniques to blend or combine information from each of the cache data entries to generate the multiple new views. For example, blending the information may include weighting the influence of each of the frames in proportion to a time difference of each of the cache data entries relative to a target timestamp. According to an embodiment, generating a view of the particular multi-view frame includes generating a respective view using 2D frames from each of the first cache data entry and the second cache data entry corresponding to the same view direction.

[0071] At block 808, operations may include rendering multiview video for display using the generated views of the particular multiview frame (e.g., at block 806). For example, block 808 may include providing the generated views to graphics pipeline 212 for further processing and display using a multiview video display.

[0072] 9 illustrates a flowchart of a third method 900 of providing or using a user interface for controlling timeline scrubbing for a multiview video, according to an embodiment consistent with principles described herein. At block 902, the operations may include providing a timeline interface. The timeline interface may include a portion of a user interface. The timeline interface may provide a representation, such as a graphical representation, of a timeline or period of the multiview video. The timeline interface may include an index of a cache data entry in a corresponding portion on the timeline interface. The index may indicate a user-specific seek point in the multiview video with the lowest playback latency.

[0073] In block 904, the operations may include receiving a target timestamp from a timeline interface. For example, the timeline interface may be configured to receive user input. The user input may indicate a particular timestamp of the multi-view video. Information regarding the target timestamp may be used by the system to trigger one or more other operations, such as according to the example of FIG. 8. For example, information regarding the target timestamp may be used in block 802.

[0074] The flowcharts of FIG. 7, FIG. 8, and FIG. 9 may illustrate various systems or methods of multi-view video processing embodied as a set of instructions stored in a memory and executable by a processor circuit. When embodied in software, each block may represent a module, segment, or portion of code that includes instructions for performing a specified logical function. The instructions may be embodied in the form of source code that includes human-readable statements written in a programming language, object code compiled from source code, or machine code that includes a number of instructions recognizable by a suitable execution system, such as a processor circuit of a computing device. The machine code may be translated from the source code, or the like. When embodied in hardware, each block may represent a circuit or several interconnected circuits that perform the specified logical function.

[0075] Although the flow chart shows a particular order of execution, it should be understood that the order of execution may differ from that which is depicted. For example, the order of execution of two or more blocks may be swapped relative to the order depicted. Also, two or more blocks depicted may be performed concurrently or with partial concurrence. Further, in some embodiments, one or more of the blocks may be skipped or omitted, or may be performed concurrently.

[0076] According to other embodiments of the principles described herein, a system is configured to perform real-time multi-view video transformation. According to various embodiments, the multi-view video transformation may be embodied in a processor-based system, such as, for example, a computing device. In this regard, embodiments are directed to a system configured to perform multi-view video transformation where the system includes a processor circuit or central processing unit (CPU) and a memory circuit. The memory circuit is configured to store a plurality of instructions, which, when executed by the processor circuit, cause the processor circuit to perform various operations to perform the multi-view video transformation. For example, the various operations may include or use information from the multi-view video cache 206. The plurality of instructions, when executed, may cause the processor circuit to generate cache data entries in the multi-view video cache 206 corresponding to respective timestamps of the multi-view video. According to some embodiments, the instructions may cause the processor circuit to generate and store respective tiled images as cache data entries, with tiles of each tiled image corresponding to respective copies of the 2D frames of the different views for the corresponding timestamps.

[0077] FIG. 10 illustrates a schematic block diagram of a computing device 1000 configured to perform real-time multi-view video conversion in an example according to an embodiment consistent with principles described herein. The computing device 1000 may include a system of components that perform various computing operations for a user of the computing device 1000. The computing device 1000 may be a laptop, a tablet, a smartphone, a touch screen system, an intelligent display system, other client devices, a server, or other computing devices. The computing device 1000 may include various components such as, for example, a processor 1003, a memory 1006, input / output (I / O) components 1009, a display 1012, and potentially other components. These components may be coupled to a bus 1015 that serves as a local interface that allows the components of the computing device 1000 to communicate with each other. Although the components of the computing device 1000 are shown as being included within the computing device 1000, it should be understood that at least some of the components may be coupled to the computing device 1000 through external connections. For example, a component may externally plug into or connect to computing device 1000 via an external port, socket, plug, connector, or wireless link.

[0078] The processor 1003 may include processor circuitry, such as a central processing unit (CPU), a graphics processing unit (GPU), any other integrated circuit that performs computing processing operations, or any combination thereof. The processor 1003 may include one or more processing cores. The processor 1003 includes circuitry that executes instructions. The instructions include, for example, computer code, programs, logic, or other machine-readable instructions that are received and executed by the processor 1003 to perform the computing function embodied in the instructions. The processor 1003 may execute instructions that operate on data or generate data. For example, the processor 1003 may receive input data (e.g., an image), process the input data according to an instruction set, and generate output data (e.g., a processed image). As another example, the processor 1003 may receive instructions and generate new instructions for subsequent execution. The processor 1003 may include hardware that implements a graphics pipeline (e.g., graphics pipeline 134 of FIG. 2) to render videos, images, or frames generated by an application. For example, the processor 1003 may include one or more GPU cores, vector processors, scalar processors, decoders, or hardware accelerators.

[0079] The memory 1006 may include one or more memory components. The memory 1006 is defined herein as including either or both volatile and non-volatile memory. A volatile memory component is one that does not retain information upon loss of power. Volatile memory may include, for example, random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), magnetic random access memory (MRAM), or other volatile memory structures. System memory (e.g., main memory, cache, etc.) may be implemented using volatile memory. System memory refers to high-speed memory that may temporarily store data or instructions for fast read and write access to assist the processor 1003. Images (e.g., still images, video frames) may be stored or loaded into the memory 1006 for subsequent access.

[0080] A non-volatile memory component is one that retains information upon loss of power. Non-volatile memory includes read only memory (ROM), hard disk drives, solid state drives, USB flash drives, memory cards accessed via memory card readers, floppy disks accessed via associated floppy disk drives, optical disks accessed via optical disk drives, magnetic tapes accessed via suitable tape drives. ROM may include, for example, programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), or other similar memory devices. Storage memory may be implemented using non-volatile memory that provides long-term retention of data and instructions. According to various embodiments, the multi-view video cache 206 may be implemented using volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory.

[0081] Memory 1006 may refer to a combination of volatile and non-volatile memory used to store instructions and data. For example, data and instructions may be stored in non-volatile memory and loaded into volatile memory for processing by processor 1003. Execution of instructions may include, for example, a compiled program that is converted into machine code in a format that can be loaded from non-volatile memory into volatile memory and then executed by processor 1003, source code that is converted into a suitable format such as object code that can be loaded into volatile memory for execution by processor 1003, or source code that is interpreted by another executable program to generate instructions in volatile memory and executed by processor 1003, etc. Instructions may be stored or loaded in any portion or component of memory 1006 including, for example, RAM, ROM, system memory, storage, or any combination thereof.

[0082] Although the memory 1006 is shown as separate from other components of the computing device 1000, it should be understood that the memory 1006 may be at least partially embedded or integrated with one or more components. For example, the processor 1003 may include on-board memory registers or cache for performing processing operations.

[0083] According to various embodiments, the memory 1006 is configured to store a plurality of instructions that, when executed by the processor 1003, cause the processor to receive a target timestamp for a particular multiview frame to be rendered and to retrieve a pair of cache data entries corresponding to the target timestamp from the cache. A first cache data entry and a second cache data entry of the pair include a first image frame group and a second image frame group, respectively. According to various embodiments, the first image frame group corresponds to a first multiview frame that is earlier than the target timestamp, and the second image frame group corresponds to a second multiview frame that is later than the target timestamp. When executed by the processor 1003, the plurality of instructions stored in the memory 1006 further cause the processor 1003 to generate a multiview frame using information from the first image frame group and the second image frame group. In various embodiments, the multiview frame includes a portion of a multiview video configured to be rendered on a multiview display.

[0084] In some embodiments, the memory 1006 further stores instructions that, when executed, cause the processor 1003 to generate a user interface configured to be displayed on a multi-view display. Additionally, the stored instructions, when executed, may cause the processor 1003 to generate a user interface configured to be displayed on a multi-view display. In some embodiments, the memory 1006 may also store instructions that cause the processor 1003 to generate a multi-view frame, in some embodiments by blending information from corresponding portions of the first image frame group and the second image frame group.

[0085] Also shown in Figure 10 is an I / O component 1009, which may include, for example, a touch screen, speaker, microphone, buttons, switches, dials, cameras, sensors, accelerometers, or other components that receive user input or generate output for a user. The I / O component 1009 may receive user input and convert it into data for storage in memory 1006 or for processing by processor 1003. The I / O component 1009 may receive data output by memory 1006 or processor 1003 and convert them into a format perceived by the user (e.g., sound, tactile responses, visual information, etc.).

[0086] One type of I / O component 1009 is a display 1012. The display 1012 may include a multi-view display (e.g., multi-view display 160), a multi-view display, a multi-view display combined with a 2D display, or any other display that presents graphical content. A capacitive touch screen layer, functioning as the I / O component 1009, may be layered within the display to allow a user to provide input while simultaneously perceiving visual output. The processor 1003 may generate data that is formatted as an image or frame for presentation on the display 1012. The processor 1003 may execute instructions to render an image or frame on the display 1012 for a user. The camera I / O component 1009 may be used for a video capture process that captures video that may be converted to multi-view video.

[0087] 10 may facilitate communication of instructions and data between the processor 1003, the memory 1006, the I / O components 1009, the display 1012, and any other components of the computing device 1000. The bus 1015 may include address translators, address decoders, fabric, conductive traces, conductive wires, ports, plugs, sockets, and other connectors that enable communication of data and instructions.

[0088] According to various embodiments, the instructions in memory 1006 may be embodied in various forms in a manner that implements at least a portion of a software stack. For example, the instructions may be embodied as an operating system 1031, an application 1034, a device driver (e.g., a display driver 1037), firmware (e.g., a display firmware 1040), or other software components. The operating system 1031 is a software platform that supports basic functions of the computing device 1000, such as scheduling tasks, controlling the I / O components 1009, providing access to hardware resources, managing power, and supporting applications 1034.

[0089] The applications 1034 shown in FIG. 10 may execute on the operating system 1031 and access the hardware resources of the computing device 1000 through the operating system 1031. In this regard, the execution of the applications 1034 is controlled at least in part by the operating system 1031. The applications 1034 may be user-level software programs that provide high-level functionality, services, and other features to the user. In some embodiments, the applications 1034 may be dedicated "apps" downloadable or accessible to the user on the computing device 1000. The user may launch the applications 1034 through a user interface provided by the operating system 1031. The applications 1034 may be developed by developers and may be defined in various source code formats. The applications 1034 may be developed using a number of programming or scripting languages, such as, for example, C, C++, C#, Objective C, Java, Swift, JavaScript, Perl, PHP, Visual Basic, Python, Ruby, Go, or other programming languages. The application 1034 may be compiled by a compiler into object code or interpreted by an interpreter for execution by the processor 1003. Various embodiments described herein may be implemented as at least a portion of the application 1034.

[0090] For example, device drivers, such as display driver 1037, contain instructions that enable operating system 1031 to communicate with various I / O components 1009. Each I / O component 1009 may have its own device driver. Device drivers may be installed such that they are stored in storage and loaded into system memory. For example, upon installation, display driver 1037 translates high-level display instructions received from operating system 1031 into low-level instructions that are performed by display 1012 to display images.

[0091] For example, firmware such as the display firmware 1040 may include machine code or assembly code that enables the I / O components 1009 or the display 1012 to perform low-level operations. The firmware may translate the electrical signals of a particular component into high-level instructions or data. For example, the display firmware 1040 may control how the display 1012 operates individual pixels at a low level by adjusting voltage or current signals. The firmware may be stored in and executed directly from non-volatile memory. For example, the display firmware 1040 may be embodied in a ROM chip coupled to the display 1012 such that the ROM chip is separate from other storage and system memory of the computing device 1000. The display 1012 may include processing circuitry for executing the display firmware 1040.

[0092] The operating system 1031, applications 1034, drivers (e.g., display driver 1037), firmware (e.g., display firmware 1040), and potentially other instruction sets may each include instructions executable by the processor 1003 or other processing circuitry of the computing device 1000 to perform the functions and operations described above. The instructions described herein may be embodied in software or code executed by the processor 1003 as described above, although, alternatively, the instructions may also be embodied in dedicated hardware or a combination of software and dedicated hardware. For example, the functions and operations performed by the instructions described above may be implemented as circuits or state machines employing any one or combination of several technologies. These technologies may include, but are not limited to, discrete logic circuits having logic gates for performing various logical functions upon application of one or more data signals, application specific integrated circuits (ASICs) having appropriate logic gates, field programmable gate arrays (FPGAs), or other components, and the like.

[0093] In some embodiments, instructions for performing the functions and operations described above may be embodied in a non-transitory computer-readable storage medium. The computer-readable storage medium may or may not be part of the computing device 1000. The instructions may include, for example, statements, code, or declarations that may be fetched from the computer-readable medium and executed by a processing circuit (e.g., processor 1003). In the context of this disclosure, a "computer-readable medium" may be any medium that can contain, store, or maintain the instructions described herein for use by or associated with an instruction execution system, such as, for example, the computing device 1000.

[0094] The computer-readable medium may include any one of many physical media, such as, for example, magnetic media, optical media, or semiconductor media. More specific examples of suitable computer-readable media may include, but are not limited to, magnetic tape, magnetic floppy diskette, magnetic hard drive, memory card, solid-state drive, USB flash drive, or optical disk. The computer-readable medium may also be, for example, random access memory (RAM), including static random access memory (SRAM) and dynamic random access memory (DRAM) or magnetic random access memory (MRAM). In addition, the computer-readable medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or other types of memory devices.

[0095] Computing device 1000 may perform any of the operations or implement the functions described above. For example, the flowcharts and process flows described above may be performed by computing device 1000 executing instructions and processing data. Although computing device 1000 is shown as a single device, the embodiments are not so limited. In some embodiments, computing device 1000 may offload processing of instructions in a distributed manner, such that multiple computing devices 1000 operate together to execute instructions that may be stored or loaded in a distributed arrangement of computing components. For example, at least some instructions or data may be stored, loaded, or executed in a cloud-based system operating in conjunction with computing device 1000.

[0096] Thus, examples and embodiments of a computer-implemented method of multiview video rendering, a computer-implemented method of preparing a multiview video cache, and a system configured to perform real-time multiview video transformation employing a multiview video cache have been described. The multiview video cache may similarly include at least a pair of cache data entries corresponding to a target timestamp, where a first cache data entry and a second cache data entry of the pair may include a first image frame group and a second image frame group, respectively. The first image frame group may correspond to a first multiview frame that is earlier than the target timestamp, and the second image frame group may correspond to a second multiview frame that is later than the target timestamp. A view of a particular multiview frame that corresponds to a target timestamp may be generated using information from the first image frame group and the second image frame group from the cache. It should be understood that the above examples merely illustrate some of the many specific examples that represent the principles described herein. Clearly, those skilled in the art may readily devise numerous other configurations without departing from the scope defined by the following claims. [Explanation of symbols]

[0097] 103 Video Capture Process 105 Scenes 107 Camera 109 directions 112 2D Video Streams 115 2D Frame 118 Multiview Video 119 Multi-view transformation process 121 Multiview Frames 124 Views 160 Multi-view Display 161 Principal angular direction 202 Multiview Frame A 204 Multiview Frame N 206 Multiview Video Cache 208 Cache Entry A 210 Cache Entry N 212 Graphics Pipeline 214 GPU 216 GPU Memory 218 Output Buffer 302 First cache data entry 304 Second cache data entry 306 Third cache data entry 308 Fourth cache data entry 310 Multiview Frame 402 Target Timestamp 404 Timeline 406 User Interface 602 1st Tile 604 Second Tile 606 New Views 1000 Computing Devices 1003 Processor 1006 Memory 1009 I / O Components 1012 Display 1015 Bus 1031 Operating Systems 1034 Applications 1037 Display Driver 1040 Display Firmware

Claims

1. 1. A computer-implemented method for multiview video rendering, comprising: receiving, using a computing device, video information having a plurality of multi-view frames, each multi-view frame of the plurality of multi-view frames corresponding to a respective multi-view video timestamp and including a plurality of two-dimensional (2D) frames corresponding to different views of a scene within the multi-view frame; determining a target timestamp for a particular multiview frame to be rendered based on user input; retrieving from a multiview video cache a pair of cache data entries corresponding to the target timestamp, wherein a first cache data entry of the pair of cache data entries includes a respective first image frame group corresponding to a first multiview frame that is earlier than the target timestamp and a second cache data entry of the pair of cache data entries includes a respective second image frame group corresponding to a second multiview frame that is later than the target timestamp; generating a view of the particular multiview frame using the first group of image frames and the second group of image frames from the cache; Rendering a multiview video for display by the computing device, the multiview video including the view of the particular multiview frame; 4. A computer-implemented method comprising:

2. 2. The computer-implemented method of claim 1, further comprising: generating the cache having cache data entries corresponding to different respective timestamps, each cache data entry including a tiled image, each tile of the tiled image corresponding to a different 2D frame of a respective multiview frame.

3. The computer-implemented method of claim 2 , wherein the tiles of the tiled image include low-resolution copies of corresponding 2D frames from the video information.

4. 3. The computer-implemented method of claim 2, wherein the cache data entries in the cache have timestamps that are not contiguous in time.

5. The computer-implemented method of claim 2 , wherein the number of cache data entries in the cache is based on a timeline scrubbing resolution.

6. The computer-implemented method of claim 2 , wherein generating the cache comprises using 2D frames corresponding to fewer than all of the multi-view frames of the video information.

7. 2. The computer-implemented method of claim 1, comprising generating the cache having cache data entries corresponding to different timestamps, at least one of the cache data entries including an image frame group representing fewer than all of the views of a corresponding multiview frame.

8. The computer-implemented method of claim 1 , wherein generating a view of the particular multiview frame includes using a virtual camera.

9. 2. The computer-implemented method of claim 1 , wherein generating the view of the particular multiview frame comprises generating a respective view using a 2D frame from each of the first cache data entry and the second cache data entry corresponding to the particular multiview frame.

10. 10. The computer-implemented method of claim 9, wherein generating the respective views comprises blending information from the 2D frames from each of the first cache data entry and the second cache data entry.

11. 11. The computer-implemented method of claim 10, wherein blending the information comprises weighting an influence of each of the 2D frames in proportion to a time difference of each of the cache data entries relative to the target timestamp.

12. 2. The computer-implemented method of claim 1, wherein retrieving the pair of cache data entries from the cache comprises identifying two cache data entries from the cache having timestamps that are closest in time to the target timestamp.

13. 1. A computer-implemented method for preparing a multiview video cache, comprising: receiving, by a computing device, multi-view video information including two-dimensional (2D) frames, each 2D frame corresponding to a respective view of the multi-view video and corresponding to a multi-view video timestamp; registering cache data entries in the cache corresponding to respective different multi-view video timestamps, each cache data entry including a copy of the 2D frame for a particular respective timestamp; 4. A computer-implemented method comprising:

14. The computer-implemented method of claim 13 , wherein the cache data entries correspond to multi-view video timestamps that are not temporally adjacent.

15. 14. The computer-implemented method of claim 13, wherein at least one of the cache data entries includes fewer than all of the 2D frames from the video information for a particular timestamp.

16. 14. The computer-implemented method of claim 13, further comprising: generating a first cache data entry in the cache corresponding to a first timestamp, the first cache data entry including a tiled image, tiles of the tiled image corresponding to respective ones of the copies of the 2D frame for the first timestamp.

17. The computer-implemented method of claim 16 , wherein the tile is a low-resolution copy of the 2D frame for the first timestamp.

18. 1. A system configured to perform real-time multi-view video conversion, the system comprising: A processor circuit; A memory storing a plurality of instructions, the plurality of instructions, when executed by the processor circuit, causing the processor circuit to: receiving a target timestamp for a particular multiview frame to be rendered; retrieving from a cache a pair of cache data entries corresponding to the target timestamp, wherein a first cache data entry and a second cache data entry of the pair of cache data entries include a first image frame group and a second image frame group, respectively, the first image frame group corresponding to a first multiview frame that is earlier than the target timestamp and the second image frame group corresponding to a second multiview frame that is later than the target timestamp; generating a multiview frame using information from the first group of image frames and the second group of image frames; The memory and Equipped with The system, wherein the multiview frames comprise a portion of a multiview video configured to be rendered on a multiview display.

19. The memory includes: generating a user interface configured to be displayed on the multi-view display; receiving a user input via the user interface, the user input indicating the target timestamp; 20. The system of claim 18, further comprising instructions for:

20. 20. The system of claim 18, wherein the memory stores instructions that cause the processor circuit to generate the multiview frame by blending information from corresponding portions of the first image frame group and the second image frame group.

Images