Interactive video playback techniques enabling high-fidelity scaling

Abstract

During presentation of a first video (400), in response to a zoom command (206), a second video (408) is presented in combination with the first video. The first and second videos are generated from substantially the same camera location as each other, at substantially the same time, and with substantially the same resolution. However, the second video is generated by a physical or virtual lens having a field of view (FOV) that is smaller than the FOV of the physical or virtual lens used in generating the first video. Modules (508, 510) are described for using a matching metric to correctly position the second video over the inner video to make it appear seamless.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION This application relates generally to a module for zooming video without loss of resolution. [Background technology]

[0002] As recognized herein, when zooming a video, at high levels of zoom, the image becomes pixelated. This can be mitigated by providing video with very high resolution, but such video consumes excessive storage and bandwidth. Currently, there is no adequate solution to the above computer-related technical problem. Summary of the Invention

[0003] Thus, in one aspect, a device includes at least one storage device including instructions executable by at least one processor, while not being a transitory signal, the executable instructions causing the processor to present a first video having a first object having a first size, and in response to a zoom command, present a second video having the first object having a second size larger than the first size, and align frames of the second video with frames of the first video using, at least in part, a ratio of the number of pixels in the first video to the number of pixels in the second video in a single dimension.

[0004] In an example implementation of this first aspect, the instructions may be executable to identify a horizontal offset of the second video relative to a region of interest (ROI) of a frame of the first video, identify a vertical offset of the second video relative to the ROI of the frame of the first video, and use those offsets to align the frame of the second video with the frame of the first video. In such an implementation, the device may include first and second physical or virtual cameras having respective first and second fields of view (FOVs) configured to generate the respective first and second videos, the first FOV being larger than the second FOV, and the first and second cameras capturing images of a first object simultaneously with each other.

[0005] If desired, the instructions may be further executable to disable auto-exposure for the camera to facilitate blending of the first and second videos.

[0006] In the above example, the instructions may be executable to temporally synchronize the first and second videos and encode the first and second videos as respective first and second bitstreams. The instructions may be executable to simultaneously decode both bitstreams using respective first and second decoders. In other embodiments, the instructions may be executable to compress the first and second bitstreams into a single bitstream and use a single decoder to decode the single bitstream.

[0007] In another aspect, a video player includes at least one processor configured to output pixels of a first video having a first field of view (FOV) and a second video having a second FOV smaller than the first FOV. The processor is configured to execute at least one decoding module (DM) and at least one rendering module (RM) to output the pixels. The DM includes at least one decoder, and the RM includes at least one shader. The processor is configured to provide at least a portion of the first and second videos to at least one display in response to a zoom command, at least in part using the DM and / or the RM.

[0008] In some implementations of this second aspect, the processor may be configured to align the second video with the first video using a alignment metric for a region of interest (ROI) in the first video. Aligning the videos may use a fixed alignment metric, or the alignment metrics may change over time, in which case they may be received in metadata in a bitstream decoded by the DM or calculated using motion estimation and image matching.

[0009] In an implementation of the second aspect, a zoom command establishes a magnification level (ML), and the processor may be configured to use the ML to determine which portions of the first and second videos are made visible on the display. In such a case, the processor may be configured to set upper and lower limits on the ML to avoid magnification levels that result in image quality degradation. The processor may also be configured to decrease the number of visible pixels of the first video and increase the number of visible pixels of the second video in response to a zoom command that increases the ML.

[0010] In an implementation of the second aspect, the processor may be configured to execute at least one shader of the RM to use a magnification level (ML) associated with a zoom command, a matching metric, and a frame number of an input bitstream associated with the video for synchronization to create the perception of watching a single video rather than two separate videos. The processor may be configured to feather the videos using the at least one shader to mask boundaries between the videos.

[0011] In some embodiments of the second aspect, the processor may be configured to skip rendering of at least a portion of the second video when a magnification level (ML) established by a zoom command is a first ML such that a portion of the second video is not decoded and at least one decoder of the DM is in an inactive state. In response to a change in the ML, the exemplary processor may be further configured to change the at least one decoder from an inactive state to an active state only when a current frame being decoded is a key frame.

[0012] In another aspect, a method includes receiving at least first, second, and third bitstreams representing respective first, second, and third videos. The method includes, in response to at least a first requested magnification level (ML), decoding the first bitstream with a first decoder to render the first video, decoding the second bitstream with a second decoder to render the second video, and presenting the first and second videos on a display. The method also includes, in response to a second requested ML greater than the first requested ML, decoding the third bitstream with the first decoder to render the third video, decoding the second bitstream with the second decoder to render the second video, and presenting the second and third videos on the display.

[0013] In some embodiments of this aspect, the method includes using a bitstream identifier (ID) passed from at least one of the decoders to display decoded pixels of each bitstream according to the first or second request ML. In response to a change in at least one bitstream ID, the rendering can be updated to use different textures and sampling coordinates.

[0014] In an example embodiment of this third aspect, the method may include using the same instance of a first decoder to process multiple bitstreams, where keyframes of the bitstreams are aligned and evenly spaced according to how quickly a user can increase or decrease ML, and pre-calculating keyframe positions and offsets for each bitstream. Further, the example method may include using at least one decoder to predict the next bitstream to be processed based on the requested ML, and decoding the next bitstream to render decoded pixels before the decoded pixels are made visible on a display to facilitate the rate of change of ML.

[0015] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram of an exemplary system in accordance with the present principles; [Figure 2] An embodiment of logic consistent with the present principles is illustrated in flow chart form. [Figure 3] Illustrates that the user zooms by moving forward along the Z axis. [Figure 4] 1 illustrates a schematic diagram of zooming. [Figure 5] 10 shows a schematic representation of an offset between videos. [Figure 5A] FIG. 2 is a block diagram of an example rendering module and decoding module. [Figure 6] Illustrates the view from five cameras. [Figure 7] Illustrates multi-FOV and multi-position content capture. DETAILED DESCRIPTION OF THE INVENTION

[0017] The present disclosure generally relates to computer ecosystems, including aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks, including wireless networks operating on 5G or ATSC 3.0. The systems herein may include a server component and a client component, which may be connected via a network such that data may be exchanged between the client and server components. The client components may include one or more computing devices, including game consoles such as Sony PlayStation® or Microsoft, Nintendo, or other game consoles, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart TVs, Internet-enabled televisions), portable computers such as laptops and tablet computers, and other mobile devices, including smartphones, as well as additional examples described below. These client devices may operate in a variety of operating environments. For example, some client computers may employ, by way of example, the Linux operating system, a Microsoft operating system, a Unix operating system, or an Apple or Google operating system. These operating environments may be used to run one or more browsing programs, such as browsers made by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by the Internet servers described below. Additionally, operating environments according to present principles may be used to run one or more computer game programs.

[0018] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, clients and servers may be connected via a local intranet or virtual private network. The server or controller may be instantiated by a game console such as a Sony PlayStation™, a personal computer, or the like.

[0019] Information may be exchanged between the client and the server over a network. For this purpose and for security, the server and / or client may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community, such as an online social website or a gamer network, to network members.

[0020] The processor may be a single-chip or multi-chip processor capable of implementing logic through various wiring such as address lines, data lines, and control lines, and registers and shift registers.

[0021] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or shown in the drawings may be combined, interchanged, or excluded from other embodiments.

[0022] "A system having at least one of A, B, and C" (and similarly "a system having at least one of A, B, or C" and "a system having at least one of A, B, and C") includes systems having A only, B only, C only, A and B together, A and C together, B and C together, and / or A, B, and C together.

[0023] Referring now specifically to FIG. 1 , an exemplary system 10 is shown, which may include one or more of the exemplary devices described above and further below in accordance with the present principles. The first exemplary device included in system 10 is a consumer electronics (CE) device, such as, for example, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). AVD 12 may alternatively be a computerized Internet-enabled (“smart”) phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or headset such as smart glasses or a VR headset, another wearable computerized device, a computerized Internet-enabled music player, computerized Internet-enabled headphones, a computerized Internet-enabled implantable device such as an implantable skin device, etc. It should be understood that AVD 12, regardless of its nature, is configured to implement the present principles (e.g., communicate with other CE devices to implement the present principles, execute the logic described herein, and perform any other functions and / or operations described herein).

[0024] Accordingly, to implement these principles, AVD 12 may be established by some or all of the components shown in Figure 1. For example, AVD 12 may include one or more touch-enabled displays 14, which may be implemented by high-definition or ultra-high-definition "4K" or higher flat screens. Touch-enabled display 14 may include, for example, a capacitive or resistive touch-sensing layer having a grid of electrodes for touch sensing consistent with the present principles.

[0025] The AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD 12 to control it. The example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, or a LAN, under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that the processor 24 controls the AVD 12 to implement the present principles, including other elements of the AVD 12 described herein, such as controlling the display 14 to present images thereon and receiving input therefrom. It should further be noted that the network interface 20 may be a wired or wireless modem or router, or a wireless telephone transceiver, or other suitable interface, such as the Wi-Fi transceiver discussed above.

[0026] In addition to the above, AVD 12 may include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI®) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 to present audio from AVD 12 to a user through the headphones. For example, input port 26 may be connected, via wire or wireless, to a cable or satellite source 26a of audio-video content. Thus, source 26a may be a separate or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. When implemented as a game console, source 26a may include some or all of the components described below in connection with CE device 48.

[0027] AVD 12 may further include one or more computer memory / computer-readable storage media 28, such as non-transitory disk-based or solid-state storage, possibly embodied within the AVD chassis as a standalone device, or as a personal video recording device (PVR) or video disk player either internal or external to the AVD chassis for playing AV programs, or as removable memory media or a server as described below. In some embodiments, AVD 12 may also include a position or location receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular tower and provide that information to processor 24 and / or to determine the altitude at which AVD 12 is disposed in conjunction with processor 24. Component 30 may also be implemented by an inertial measurement unit (IMU), typically including a combination of accelerometers, gyroscopes, and magnetometers, or by event-based sensors to determine the location and orientation of AVD 12 in three dimensions.

[0028] Continuing with the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into AVD 12 and controllable by processor 24 to collect photos / images and / or video in accordance with present principles. AVD 12 may also include a Bluetooth® transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth® and / or NFC technology, respectively. An exemplary NFC element can be a radio frequency identification (RFID) element.

[0029] Continuing further, AVD 12 may include one or more auxiliary sensors 38 (e.g., motion sensors such as pressure-sensitive sensors, accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) that provide input to processor 24. For example, one or more of auxiliary sensors 38 may include one or more pressure sensors forming a layer of touch-enabled display 14 itself, and may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc.

[0030] The AVD 12 may also include an over-the-air television broadcast port 40 for receiving OTA television broadcasts, which provides input to the processor 24. In addition to the above, it should be noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR Data Association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, and may also be a kinetic energy harvester that can convert kinetic energy into power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may be included. One or more haptic / vibration generators 47 may be provided to generate haptic signals that can be sensed by a person holding or in contact with the device. The haptic generator 47 may vibrate all or a portion of the AVD 12 using an electric motor connected to an off-center and / or off-balance weight via a rotatable shaft of the motor, such that the shaft can rotate under the control of the motor (which in turn may be controlled by a processor such as processor 24) to create simulations of vibrations of various frequencies and / or amplitudes, and forces in various directions.

[0031] 1 , in addition to AVD 12, system 10 may include one or more other CE device types. In one embodiment, first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or through a server, as described below, while second CE device 50 may include similar components as first CE device 48. In the illustrated embodiment, second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a head-up transparent or non-transparent display for presenting AR / MR content or VR content, respectively.

[0032] In the illustrated embodiment, only two CE devices are shown, but it should be understood that a fewer or greater number of devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for AVD 12.

[0033] Referring now to the aforementioned at least one server 52, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other devices of Figure 1 via network 22, and may indeed facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephone transceiver.

[0034] Thus, in some embodiments, server 52 may be an entire internet server or server "farm," and may include and perform "cloud" functionality such that, in an exemplary embodiment for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers located in or near the same room as the other devices shown in FIG.

[0035] The components shown in the following figures may include some or all of the components shown in Figure 1. The user interfaces (UIs) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0036] 2 illustrates that, in an embodiment, "N" videos are generated in block 200 by each virtual or physical camera and associated physical or virtual lens. N can be an integer greater than or equal to 2. In one embodiment, N is equal to 5.

[0037] In one embodiment, each of the N videos has the same resolution, such as, but not limited to, 4K. However, in other embodiments, the N videos may not all have the same resolution.

[0038] In either case, in one embodiment, the videos may be taken at the same or substantially the same time and from the same or substantially the same location. "Substantially the same location" means, for example, within the constraints of physically placing two cameras in the same place; the cameras may be closely positioned side-by-side despite being separated by the width of the camera housings. "Substantially the same time" means at the same real or virtual time, or within a few seconds of each other.

[0039] However, a first video may be generated using a physical or virtual lens having a first field of view (FOV), a second video may be generated using a physical or virtual lens having a second FOV smaller than the first FOV, and so on, with each successive video being generated with a successively smaller FOV than the preceding video in the chain. However, each FOV may be centered at the same location or point or center. Note that in addition to or instead of successively smaller FOVs, the physical or virtual cameras may have successively shorter focal lengths.

[0040] Moving to block 202, the videos are synchronized with each other, for example, by aligning key frames in each video with each other, and in a particular embodiment, encoding the videos as H264. Alignment is described further below.

[0041] When a user wishes to play a video, it is presented in block 204 using a first video, i.e., the video with the widest FOV. In block 206, as the user zooms in using an input device or by moving their head along the Z axis while wearing the HMD presenting the video, a video with the next smaller FOV is combined with the first video, eventually replacing it. Continued zooming results in successive videos presented with successively smaller FOVs, so that zooming is emulated without loss of fidelity. Thus, during playback, content from the telephoto camera is inserted into content from the wide-angle camera according to a pre-calculated alignment metric, creating the perception of watching a single video. Due to the precise alignment, the presence of the inner video displayed within the outer video is not apparent to the viewer.

[0042] FIG. 3 illustrates a user 300 wearing an HMD 302 that zooms by moving along a Z axis 304 .

[0043] FIG. 4 continues to further illustrate. Note that while FIG. 4 illustrates an implementation in which a scene is captured at different locations in addition to using different FOVs, FIG. 6, described below, illustrates a case in which more than two videos are captured from the same location. More specifically, in the example shown in FIG. 4, multiple (e.g., three) lenses are used with different FOVs to capture three videos from the same real or virtual camera position, and the same three lenses, each with a different FOV, are used to capture three videos from a second position. Thus, after recording, six videos are captured simultaneously.

[0044] A first video 400 is shown in its widest mode 402. As the user zooms in, the video is shown in its standard mode 404, and finally, under continued zoom, in its telephoto mode 406, with each mode filling the display. It should be understood that the transition between the three modes shown is continuous and gradual as the user zooms, and that for simplicity only three common modes are shown.

[0045] When the zoom in telephoto mode 406 of the first video reaches a threshold limit, further zooming results in combining the first video with the second video 408 in maximum wide-angle mode 410. It should be understood that the second video 408 may eventually or immediately replace the first video entirely as the zoom progresses from the telephoto mode 406 of the first video to the wide-angle mode 410 of the second video 408.

[0046] As the user continues to zoom in, the second video 408 is shown in its standard angle mode 412 and eventually, under continued zoom, in its telephoto mode 414, with each mode filling the display.

[0047] Continued zooming from the telephoto mode 414 of the second video results in the second video being combined with a third video 416 having a maximum wide-angle mode 418. It should be understood that the third video 414 may eventually or immediately replace the second video entirely as the zoom progresses from the telephoto mode 414 of the second video to the wide-angle mode 418 of the third video 416.

[0048] As the user continues to zoom in, the third video 416 is shown in its standard angle mode 420 and eventually, under continued zoom, in its telephoto mode 422, with each mode filling the display. Note that if the scene is captured from only a single position, steps 408-422 are not available.

[0049] While FIG. 4 illustrates the use of three videos generated by physical or virtual lenses, each having successively smaller FOVs, it should be understood that, consistent with the principles of FIG. 4, only two videos need be used, or more than three videos may be used.

[0050] Note that multiple videos, each with a progressively smaller FOV, can be generated for multiple likely regions of user focus. A central focus can be used as a baseline, and then offsets from that point in terms of distance and direction can be used and transmitted as metadata to indicate when the user is focusing on a point separated from the central focus by the offset. For each offset, a series of nested videos can be pre-computed or computed on the fly for a particular focus as the user focuses on that point. If the user happens to focus on a point for which no nested videos with progressively smaller FOVs exist, traditional scaling techniques can be used.

[0051] A heatmap of previous user focus for each scene can be used to determine which points in the scene should have a series of nested videos generated for them: only the video of the area on which the user focused can be decoded.

[0052] Reference is now made to FIG. 5 for a description of a match metric that may be determined before or during capture.

[0053] An insertion ratio (R) can be determined to be the ratio of the number of pixels in the outer video (wider FOV) to the number of pixels in the inner video (narrower FOV) in a single dimension. In Figure 5, W0 is the width in pixels of the outer video, W1 is the width in pixels of the inner video, and after alignment, R = W0 / W1. The insertion ratio depends on the focal lengths of the two cameras and the resolution of the camera sensors. Note that the inner video may have the same resolution as the outer video, but the inner video may be displayed at a smaller size after alignment.

[0054] The horizontal offset (Oh) is shown in Figure 5 and is the horizontal offset of the inner video or ROI measured from the center of the frame of the outer video. Similarly, the vertical offset (Ov) is the vertical offset of the inner video or ROI measured from the center of the frame of the outer video.

[0055] FIG. 5 illustrates that frames of a wider FOV and a narrower FOV video are aligned during display using the above offset with an alignment metric. Specifically, the camera positions are determined, and two cameras with different FOVs simultaneously capture the same scene. In the simplest case, Oh = Ov = 0, and the ROI is the center of the video frame. An interpolation ratio between the two can be achieved by using an FOV of 60 for the wide-angle lens and an FOV of approximately 32.2 for the telephoto lens. Note that the video captured from the telephoto lens may have the same resolution as the video captured from the wide-angle lens. The interpolation ratio is not the ratio of the number of pixels in the first video to the number of pixels in the second video in a single dimension during capture. The interpolation ratio is determined based on how these videos are displayed after alignment. In this example, both videos may be captured as 3840 x 2160. However, after alignment, the inner video is initially displayed as 1920 x 1080. The interpolation ratio is therefore 3840 / 1920 = 2. By disabling automatic features such as auto-exposure on the camera, it becomes easier to blend the two frames together during display.

[0056] See FIG. 5A. The raw video from the two cameras, labeled 500 and 502, is synchronized and encoded as two separate bitstreams. In this case, two decoders are used to decode both bitstreams simultaneously. In other embodiments using one decoder, the video data from each camera may be compressed as a single bitstream but may be independently decodable, for example, as HEVC tiles. In either case, the video player 504, which generates output pixels for the display 506, includes a decoding module (DM) 508 and a rendering module (RM) 510. The DM, in turn, includes one or more decoders 512 capable of decoding the compressed bitstream. The RM includes GPU shaders that can sample the video texture and render it to the display.

[0057] The match metric can be fixed or can change over time. In the fixed case, the match metric can be sent to the DM and / or RM only once. In the case of a dynamic match metric, the DM and / or RM can be updated with each change in the metric. One way to achieve this is to pass the match metric as metadata in the compressed bitstream. In other embodiments, the match metric can be calculated automatically using motion estimation and image matching algorithms.

[0058] A video player that renders decoded video data on a display accepts magnification control from a user using a device such as a mouse or video game controller. The magnification level (ML) selected by the user is used to determine the portions of the outer and inner videos that are visible on the display. The system can place upper and lower bounds on ML to avoid magnification levels that result in image quality degradation. When the user zooms in, the value of ML increases, and when the user zooms out, ML decreases. As ML increases, the number of visible pixels in the outer video decreases and the number of visible pixels in the inner video increases. The RM's GPU shader uses the ML value, a matching metric, and the frame number of each bitstream for synchronization to create the perception of watching a single video rather than two separate videos. In other embodiments, an additional "feathering" step can be performed by the shader to mask the boundary at the junction of the inner and outer videos.

[0059] When the ML is small and the number of visible pixels of the inner video is small, rendering of the inner video can be skipped without a noticeable difference in the image quality of the displayed video. When decoded video data of the inner video is not being displayed, decoding of the non-displayed video data can be eliminated, thereby improving system performance and efficiency. One way this can be achieved is by utilizing the ML to determine which video bitstreams need to be decoded and rendering only frames from the actively decoded bitstreams. When a decoder is in an active state, access units (AUs) of the bitstreams are successfully decoded, and the decoded video data is sent to the RM for rendering to the display. When a decoder is in an inactive state, decoding of the AUs can be partially or completely skipped, and video data for bitstreams corresponding to the inactive decoders is not rendered to the display.

[0060] When the ML changes, a decoder in an active state may become inactive, and vice versa. Switching a decoder from an active state to an inactive state can be done immediately, but switching from an inactive state to an active state may not be done immediately. The reason for this is that the current AU may depend on the previous AU, and if decoding of the previous AU was skipped when the decoder was in the inactive state, the current AU may have an error when decoded. To avoid this problem, switching from an inactive state to an active state may be performed only when the current AU is a key frame (IDR frame). To support this, a seeking state may be used in which, when the ML exceeds a threshold, an inactive decoder first switches to a seeking state in which the decoder waits for an IDR. When the current AU is an IDR, the decoder switches from the seeking state to the active state. The DM passes the bitstream ID of the active decoder to the RM and passes invalid IDs for decoders in the seeking or inactive state to the RM. The RM uses these IDs to render only valid pixels to the display.

[0061] For applications requiring high magnification levels or a smoother transition from a zoomed-out field of view to a zoomed-in field of view, more than two camera fields of view may be required. For such use cases, more than two cameras with varying focal lengths or FOVs may be used. As before, the same scene is captured simultaneously from a single position using these cameras.

[0062] An example of fields of view that can be captured using five cameras is shown in FIG. 6 (the five fields of view are labeled "Wide 1," "Wide 2," "Telephoto 1," "Telephoto 2," and "Standard").

[0063] The video data from each camera in Figure 6 can be synchronized and compressed as individual bitstreams or independently decodable substreams. While all of these streams can be simultaneously decoded and selectively rendered according to the desired ML, a more efficient approach is to decode only the streams that will ultimately be displayed. The number of decoders required in the DM can be equal to the maximum number of video streams being simultaneously rendered at any given moment. For the setup shown in Figure 5 of one outer video and one inner video, the number of decoders required can be limited to two, even if more than two video streams are used. This is achieved using a "stream switching" strategy described below.

[0064] The stream processed by each decoder is determined by the value of ML. When the application starts, the first decoder (D1) can process the widest bitstream (B1), and the second decoder (D2) can process the second bitstream (B2) with a lower FOV. As the user increases ML, there comes a point where the pixels of B1 are no longer rendered on the display. D1 then transitions to a seek state and prepares to decode the next bitstream (B3) in the field of view list. The RM uses the bitstream ID and alignment metric passed by the decoder to display the decoded pixels of each bitstream at the appropriate magnification level. When the RM detects a change in the bitstream ID, it updates the rendering process to use the correct texture and sampling coordinates.

[0065] In other embodiments, the following steps can be performed during the encoding process to facilitate smooth stream switching.

[0066] First, the bitstreams use similar coding structures so that the same instance of a decoder can process AUs from multiple bitstreams without requiring extra memory. The IDRs of different bitstreams can be aligned and evenly spaced according to how fast the user can increase or decrease the ML. Second, the IDR positions and AU offsets for each bitstream can be pre-computed to avoid doing this in the DM.

[0067] In further embodiments, the DM may include one or more extra decoders to predict the next bitstream to be processed based on ML and decode these streams before the decoded pixels are made visible on the display. This strategy can help increase the rate of change of ML. An alternative approach to achieve this is to encode the bitstream using only IDR.

[0068] Referring now to FIG. 7 , an alternative technique for applications requiring high magnification levels is multi-position content capture instead of multi-FOV content capture. Instead of capturing a scene from one position using cameras with different FOVs, the scene can be captured using the same FOV but at different positions 700, 702 in the direction of scene capture. In other embodiments, both multi-FOV and multi-position content capture can be used together, as shown in FIG. 7 . In other embodiments, the RM can include a stage for distortion correction between the multi-position content or multi-viewpoint content. In other embodiments, audio is also captured from different positions, and the audio stream is also switched according to the ML for a more immersive experience.

[0069] Although particular embodiments are shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. 1. A method comprising: receiving at least first, second, and third bitstreams representing respective first, second, and third videos; decoding the first bitstream with a first decoder to render the first video and decoding the second bitstream with a second decoder to render the second video in response to at least a first requested magnification level (ML); presenting the first and second videos on a display; in response to a second request ML greater than the first request ML, decoding the third bitstream with the first decoder to render the third video and decoding the second bitstream with the second decoder to render the second video; presenting the second and third videos on a display; A method wherein the first, second, and third videos are generated for multiple likely regions of user focus by physical or virtual lenses, each having an increasingly smaller FOV for the first, second, and third videos.

2. 2. The method of claim 1, wherein a central focus is used at the baseline and an offset in terms of distance and direction from the central focus is used and transmitted as metadata to indicate when a user is focusing on a point separated from the central focus by the offset.

3. The method of claim 2 , wherein for each offset, a series of nested videos are pre-computed or computed on the fly for a particular focus point as a user focuses on a particular point.

4. The method of claim 3 , wherein a heat map of previous user focus for each scene is used to determine which points within a scene should have the series of nested videos generated for them.

5. using a bitstream identifier (ID) passed from at least one of the decoders to display decoded pixels of each bitstream according to the first or second request ML; and updating the rendering to use different textures and sampling coordinates in response to a change in at least one bitstream ID.

6. using the same instance of the first decoder to process multiple bitstreams, where the keyframes of the bitstreams are aligned and evenly spaced according to how fast a user can increase or decrease the ML; and pre-calculating keyframe positions and offsets for each bitstream.

7. using at least one decoder to predict a next bitstream to be processed based on a requested ML; and decoding the next bitstream to render decoded pixels before the decoded pixels are made visible on the display to expedite the rate of change of the ML.

Images