Interactive video playback techniques enabling high fidelity scaling

Abstract

During presentation of a first video (400), in response to a zoom command (206), a second video (408) is presented in combination with the first video. The first and second videos are generated from substantially the same camera location as each other, at substantially the same time, and with substantially the same resolution. However, the second video is generated by a physical or virtual lens having a field of view (FOV) that is smaller than the FOV of the physical or virtual lens used in generating the first video. Modules (508, 510) are described for using a matching metric to correctly position the second video over the inner video to make it appear seamless.

Description

[Technical field]

[0001] FIELD This application relates generally to a module for zooming a video without losing resolution. [Background technology]

[0002] As recognized herein, when zooming a video, at high levels of zoom, the image becomes pixelated. This can be mitigated by providing a video with a very high resolution, but such a video consumes excessive storage and bandwidth. Currently, there is no adequate solution to the above computer-related technical problem. Summary of the Invention

[0003] Thus, in one aspect, a device includes at least one storage device including instructions executable by at least one processor, the instructions being executable to cause the processor to present a first video having a first object having a first size, the instructions being executable to, in response to a zoom command, present a second video having a first object having a second size larger than the first size, and align frames of the second video with frames of the first video using, at least in part, a ratio of a number of pixels in the first video to a number of pixels in the second video in a single dimension.

[0004] In an example implementation of this first aspect, the instructions may be executable to identify a horizontal offset of the second video relative to a region of interest (ROI) of a frame of the first video, identify a vertical offset of the second video relative to the ROI of the frame of the first video, and align the frame of the second video to the frame of the first video using the offsets. In such an implementation, the device may include first and second physical or virtual cameras having respective first and second fields of view (FOVs) configured to generate the respective first and second videos, the first FOV being larger than the second FOV, and the first and second cameras capturing images of a first object simultaneously with each other.

[0005] If desired, the instructions may be further executable to disable auto-exposure for the camera to facilitate blending of the first and second videos.

[0006] In the above example, the instructions may be executable to temporally synchronize the first and second videos and encode the first and second videos as respective first and second bitstreams. The instructions may be executable to simultaneously decode both bitstreams using respective first and second decoders. In other embodiments, the instructions may be executable to compress the first and second bitstreams into a single bitstream and use a single decoder to decode the single bitstream.

[0007] In another aspect, a video player includes at least one processor configured to output pixels of a first video having a first field of view (FOV) and a second video having a second FOV smaller than the first FOV. The processor is configured to execute at least one decoding module (DM) and at least one rendering module (RM) to output the pixels. The DM includes at least one decoder and the RM includes at least one shader. The processor is configured to provide at least a portion of the first and second videos to at least one display in response to a zoom command, at least in part using the DM and / or the RM.

[0008] In some implementations of this second aspect, the processor may be configured to align the second video with the first video using a alignment metric for a region of interest (ROI) in the first video. Aligning the videos may use a fixed alignment metric, or the alignment metrics may change over time, in which case they may be received in metadata in a bitstream decoded by the DM or calculated using motion estimation and image matching.

[0009] In an implementation of the second aspect, the zoom command establishes a magnification level (ML), and the processor may be configured to use the ML to determine which portions of the first and second videos are made visible on the display. In such a case, the processor may be configured to set upper and lower limits on the ML to avoid magnification levels that result in image quality degradation. The processor may also be configured to decrease the number of visible pixels of the first video and increase the number of visible pixels of the second video in response to a zoom command that increases the ML.

[0010] In an implementation of the second aspect, the processor may be configured to execute at least one shader of the RM to use a magnification level (ML) associated with a zoom command, a matching metric, and a frame number of an input bitstream associated with the video for synchronization to create the perception of watching a single video rather than two separate videos. The processor may be configured to feather the videos using at least one shader to mask boundaries between the videos.

[0011] In some embodiments of the second aspect, the processor may be configured to skip rendering at least a portion of the second video when a magnification level (ML) established by the zoom command is a first ML such that a portion of the second video is not decoded and at least one decoder of the DM is in an inactive state. In response to a change in the ML, the example processor may be further configured to change the at least one decoder from an inactive state to an active state only when a current frame being decoded is a key frame.

[0012] In another aspect, a method includes receiving at least a first, second, and third bitstreams representing respective first, second, and third videos. The method includes, in response to at least a first requested magnification level (ML), decoding the first bitstream with a first decoder to render the first video, decoding the second bitstream with a second decoder to render the second video, and presenting the first and second videos on a display. The method also includes, in response to a second requested ML greater than the first requested ML, decoding the third bitstream with the first decoder to render the third video, decoding the second bitstream with the second decoder to render the second video, and presenting the second and third videos on a display.

[0013] In some embodiments of this aspect, the method includes using a bitstream identifier (ID) passed from at least one of the decoders to display decoded pixels of each bitstream according to the first or second request ML. In response to a change in the at least one bitstream ID, the rendering may be updated to use different textures and sampling coordinates.

[0014] In an example embodiment of this third aspect, the method may include using the same instance of a first decoder to process multiple bitstreams, where the keyframes of the bitstreams are aligned and evenly spaced according to how fast a user can increase or decrease the ML, and pre-calculating keyframe positions and offsets for each bitstream. Further, an example method may include using at least one decoder to predict the next bitstream to be processed based on the required ML, and decoding the next bitstream to render the decoded pixels before the decoded pixels are made visible on a display to facilitate the rate of change of ML.

[0015] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram of an exemplary system in accordance with the present principles; [Diagram 2] An embodiment of logic consistent with the present principles is illustrated in flow chart form. [Diagram 3] Illustrates a user zooming by moving forward along the Z axis. [Figure 4] 1 illustrates a schematic diagram of a zoom. [Diagram 5] 3 illustrates a schematic representation of an offset between videos. [Figure 5A] FIG. 2 is a block diagram of an example rendering and decoding module. [Figure 6] Illustrates the views from five cameras. [Figure 7] Illustrates multi-FOV and multi-position content capture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] The present disclosure generally relates to a computer ecosystem including aspects of a consumer electronics (CE) device network, such as, but not limited to, a computer gaming network, including wireless networks operating on 5G or ATSC 3.0. The system herein may include a server component and a client component that may be connected via a network such that data may be exchanged between the client component and the server component. The client components may include one or more computing devices including gaming consoles, such as Sony PlayStation® or Microsoft, or Nintendo or other manufacturer gaming consoles, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart televisions, Internet-enabled televisions), portable computers, such as laptops and tablet computers, other mobile devices including smartphones, as well as additional examples described below. These client devices may operate in a variety of operating environments. For example, some of the client computers may employ, as examples, a Linux operating system, a Microsoft operating system, or a Unix operating system, or an Apple or Google operating system. These operating environments may be used to run one or more browsing programs, such as browsers produced by Microsoft, Google, or Mozilla, or other browser programs that can access web sites hosted by the Internet servers described below. Additionally, operating environments according to present principles may be used to run one or more computer game programs.

[0018] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, clients and servers may be connected via a local intranet or a virtual private network. The servers or controllers may be instantiated by game consoles such as the Sony PlayStation®, personal computers, etc.

[0019] Information may be exchanged between the clients and the servers over a network. For this purpose and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements the method for providing a secure community, such as an online social website or a gamer network, to network members.

[0020] The processor may be a single-chip or multi-chip processor capable of implementing logic through various wiring such as address lines, data lines, and control lines, and registers and shift registers.

[0021] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or illustrated in the drawings may be combined, interchanged, or excluded from other embodiments.

[0022] "A system having at least one of A, B, and C" (also "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes systems having A only, B only, C only, A and B together, A and C together, B and C together, and / or A, B, and C together.

[0023] 1, an exemplary system 10 is shown that may include one or more of the exemplary devices described above and further below in accordance with the present principles. The first of the exemplary devices included in the system 10 is a consumer electronics (CE) device, such as an audio video device (AVD) 12, for example, but not limited to, an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). The AVD 12 may also alternatively be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or a headset, such as smart glasses or a VR headset, another wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headphones, a computerized Internet-enabled implantable device, such as an implantable skin device, etc. It should be understood that the AVD 12, regardless of what it is, is configured to implement the present principles (e.g., to communicate with other CE devices to implement the present principles, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0024] Thus, to realize such principles, AVD 12 may be established by some or all of the components shown in Figure 1. For example, AVD 12 may include one or more touch-enabled displays 14, which may be implemented by high-definition or ultra-high-definition "4K" or higher flat screens. Touch-enabled display 14 may include, for example, a capacitive or resistive touch-sensing layer having a grid of electrodes for touch sensing consistent with the present principles.

[0025] The AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD 12 to control the AVD 12. The example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that the processor 24 controls the AVD 12 to implement the present principles, including other elements of the AVD 12 described herein, such as controlling the display 14 to present images thereon and receiving input therefrom. It should further be noted that the network interface 20 may be a wired or wireless modem or router, or a wireless telephone transceiver, or other suitable interface, such as the Wi-Fi transceiver discussed above.

[0026] In addition to the above, the AVD 12 may include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to the AVD 12 to present audio from the AVD 12 to a user through the headphones. For example, the input port 26 may be connected, via wire or wirelessly, to a cable or satellite source 26a of audio-video content. Thus, the source 26a may be a separate or integrated set-top box, or a satellite receiver. Alternatively, the source 26a may be a game console or disc player containing the content. The source 26a when implemented as a game console may include some or all of the components described below in connection with the CE device 48.

[0027] The AVD 12 may further include one or more computer memory / computer readable storage media 28, such as non-transitory disk-based or solid-state storage, possibly embodied within the AVD chassis as a standalone device or as a personal video recording device (PVR) or video disk player either inside or outside the AVD chassis for playing AV programs, or as a removable memory medium or server as described below. In some embodiments, the AVD 12 may also include a position or location receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular tower and provide that information to the processor 24 and / or in conjunction with the processor 24 to determine the altitude at which the AVD 12 is disposed. The component 30 may also be implemented by an inertial measurement unit (IMU), typically including a combination of accelerometers, gyroscopes, and magnetometers, or by event-based sensors to determine the location and orientation of the AVD 12 in three dimensions.

[0028] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, event-based sensors, and / or cameras integrated into the AVD 12 and controllable by the processor 24 to collect pictures / images and / or videos in accordance with present principles. The AVD 12 may also include a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element may be a radio frequency identification (RFID) element.

[0029] Continuing further, the AVD 12 may include one or more auxiliary sensors 38 (e.g., pressure-sensitive sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors forming a layer of the touch-enabled display 14 itself, which may be, without limitation, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc.

[0030] The AVD 12 may also include an over-the-air television broadcast port 40 for receiving an OTA television broadcast that provides input to the processor 24. In addition to the above, it is noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR Data Association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, and may also be a kinetic energy harvester that may convert kinetic energy into power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may be included. One or more haptic / vibration generators 47 may be provided to generate haptic signals that can be sensed by a person holding or in contact with the device. The haptic generator 47 may vibrate all or a portion of the AVD 12 using an electric motor connected to an off-center and / or off-balance weight via a rotatable shaft of the motor, such that the shaft can rotate under the control of the motor (which in turn may be controlled by a processor such as processor 24) to create simulations of vibrations of various frequencies and / or amplitudes, as well as forces in various directions.

[0031] 1, in addition to the AVD 12, the system 10 may include one or more other CE device types. In one embodiment, the first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through a server, described below, while the second CE device 50 may include similar components as the first CE device 48. In the embodiment shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a head-up transparent or non-transparent display for presenting AR / MR or VR content, respectively.

[0032] In the illustrated embodiment, only two CE devices are shown, but it should be understood that a fewer or greater number of devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for AVD 12.

[0033] Referring now to the at least one server 52 mentioned above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other devices of Figure 1 via the network 22, and may indeed facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephone transceiver.

[0034] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm," and may include and perform "cloud" functionality such that, for example, in an exemplary embodiment for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG.

[0035] The components shown in the following figures may include some or all of the components shown in Figure 1. The user interfaces (UIs) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0036] 2 illustrates that in an embodiment, "N" videos are generated in block 200 by respective virtual or physical cameras and associated physical or virtual lenses. N can be an integer greater than or equal to 2. In one embodiment, N is equal to 5.

[0037] In one embodiment, each of the N videos has the same resolution, such as but not limited to 4K, however, in other embodiments, the N videos may not all have the same resolution.

[0038] In either case, in one embodiment, the videos may be taken at the same or substantially the same time and from the same or substantially the same location. "Substantially the same location" means, for example, within the constraints of physically placing two cameras at the same place, where the cameras may be closely positioned side-by-side despite being separated by the width of the camera housing. "Substantially the same time" means at the same real or virtual time, or within a few seconds of each other.

[0039] However, a first video may be generated using a physical or virtual lens having a first field of view (FOV), a second video may be generated using a physical or virtual lens having a second FOV smaller than the first FOV, and so on, with each successive video being generated with a successively smaller FOV than the preceding video in the chain. However, each FOV may be centered at the same location or point or center. It should be noted that in addition to or instead of successively smaller FOVs, the physical or virtual cameras may have successively shorter focal lengths.

[0040] Moving to block 202, the videos are synchronized with each other, for example, by aligning key frames in each video with each other, and in a specific embodiment, encoding the videos as H264. Alignment is described further below.

[0041] When a user wants to play a video, it is presented in block 204 using a first video, i.e., the video with the widest FOV. In block 206, when the user zooms in using an input device or by moving his head along the Z axis while wearing the HMD presenting the video, the video with the next smaller FOV is combined with the first video, eventually replacing the first video. Continued zooming results in successive videos presented with successively smaller FOVs, such that zooming is emulated without loss of fidelity. Thus, during playback, content from the telephoto camera is inserted into content from the wide-angle camera according to a pre-computed alignment metric, creating the perception of watching a single video. Due to the precise alignment, the presence of the inner video displayed within the outer video is not obvious to the viewer.

[0042] FIG. 3 illustrates a user 300 wearing an HMD 302 that zooms by moving along a Z axis 304.

[0043] Fig. 4 continues to further illustrate. Note that Fig. 4 illustrates an implementation in which a scene is captured at different locations in addition to using different FOVs, while Fig. 6 described below illustrates a case in which more than two videos are captured from the same location. More specifically, in the embodiment shown in Fig. 4, multiple (e.g., three) lenses are used with different FOVs to capture three videos from the same real or virtual camera position, and the same three lenses, each with a different FOV, are used to capture three videos from a second position. Thus, after recording, six videos are captured simultaneously.

[0044] A first video 400 is shown in its widest mode 402. As the user zooms in, the video is shown in its standard mode 404, and finally, under continued zoom, in its telephoto mode 406, with each mode filling the display. It should be understood that the transition between the three modes shown is continuous and gradual as the user zooms, and for simplicity only three common modes are shown.

[0045] When the zoom in the telephoto mode 406 of the first video reaches a threshold limit, further zooming results in combining the first video with the second video 408 in a maximum wide-angle mode 410. It should be understood that the second video 408 may eventually or immediately replace the first video entirely as the zoom progresses from the telephoto mode 406 of the first video to the wide-angle mode 410 of the second video 408.

[0046] As the user continues to zoom in, the second video 408 is shown in its standard angle mode 412, and eventually, under continued zoom, in its telephoto mode 414, with each mode filling the display.

[0047] Continued zooming from the telephoto mode 414 of the second video results in combining the second video with a third video 416 having a maximum wide-angle mode 418. It should be understood that the third video 414 may eventually or immediately replace the second video entirely as the zoom progresses from the telephoto mode 414 of the second video to the wide-angle mode 418 of the third video 416.

[0048] As the user continues to zoom in, the third video 416 is shown in its standard angle mode 420, and eventually, under continued zoom, in its telephoto mode 422, with each mode filling the display. Note that if the scene is only captured from a single position, steps 408-422 are not available.

[0049] Although FIG. 4 illustrates the use of three videos generated by physical or virtual lenses, each having a successively smaller FOV, it should be understood that consistent with the principles of FIG. 4, only two videos need be used, or more than three videos may be used.

[0050] Note that multiple videos, each with a progressively smaller FOV, may be generated for multiple likely regions of user focus. A central focus may be used at the baseline, and then offsets in terms of distance and direction from that point may be used and transmitted as metadata to indicate when the user is focusing on a point separated from the central focus by the offset. For each offset, a series of nested videos may be precomputed, or may be computed on the fly for a particular focus as the user focuses on the particular point. If it happens that the user focuses on a point for which there are no nested videos with progressively smaller FOVs, traditional magnification techniques may be used.

[0051] A heatmap of previous user focus for each scene can be used to determine which points in the scene should have a series of nested videos generated for them. Only the video of the areas on which the user focused can be decoded.

[0052] Reference is now made to FIG. 5 for a discussion of match metrics that may be determined prior to or during capture.

[0053] An insertion ratio (R) can be determined to be the ratio of the number of pixels in the outer video (wider FOV) to the number of pixels in the inner video (narrower FOV) in a single dimension. In FIG. 5, W0 is the width in pixels of the outer video, W1 is the width in pixels of the inner video, and after alignment, R=W0 / W1. The insertion ratio depends on the focal length of the two cameras and the resolution of the camera sensor. Note that the inner video may have the same resolution as the outer video, but the inner video may be displayed at a smaller size after alignment.

[0054] The horizontal offset (Oh) is shown in Figure 5 and is the horizontal offset of the inner video or ROI measured from the center of the frame of the outer video. Similarly, the vertical offset (Ov) is the vertical offset of the inner video or ROI measured from the center of the frame of the outer video.

[0055] FIG. 5 illustrates that frames of a wider FOV and a narrower FOV video are aligned during display with the alignment metric using the offsets described above. Specifically, the camera positions are determined, and two cameras with different FOVs capture the same scene simultaneously. In the simplest case, Oh=Ov=0, and the ROI is the center of the video frame. The insertion ratio of the two can be achieved by using an FOV of 60 for the wide-angle lens and an FOV of about 32.2 for the telephoto lens. Note that the video captured from the telephoto lens may have the same resolution as the video captured from the wide-angle lens. The insertion ratio is not the ratio of the number of pixels in the first video to the number of pixels in the second video in a single dimension during capture. The insertion ratio is determined based on how these videos are displayed after alignment. In this example, both videos may be captured as 3840×2160. However, after alignment, the inner video is initially displayed as 1920×1080. The interpolation ratio is therefore 3840 / 1920 = 2. By disabling automatic features such as auto exposure on the camera, it becomes easier to blend the two frames during display.

[0056] See FIG. 5A. The raw video from the two cameras, labeled 500, 502, is synchronized and encoded as two separate bitstreams. In this case, two decoders are used to decode both bitstreams simultaneously. In other embodiments using one decoder, the video data from each camera may be compressed as a single bitstream but may be independently decodable, for example as HEVC tiles. In either case, a video player 504, which generates output pixels for a display 506, includes a decoding module (DM) 508 and a rendering module (RM) 510. The DM, in turn, includes one or more decoders 512 that are capable of decoding the compressed bitstream. The RM includes GPU shaders that can sample the video texture and render it to the display.

[0057] The match metric can be fixed or can change over time. In the fixed case, the match metric can be sent to the DM and / or RM only once. In the case of a dynamic match metric, the DM and / or RM can be updated with each change of the metric. One way to achieve this is to pass the match metric as metadata in the compressed bitstream. In other embodiments, the match metric can be calculated automatically using motion estimation and image matching algorithms.

[0058] A video player that renders the decoded video data on a display accepts magnification control from a user using a device such as a mouse or a video game controller. The magnification level (ML) selected by the user is used to determine the portions of the outer and inner videos that are visible on the display. The system can place upper and lower bounds on ML to avoid magnification levels that result in image quality degradation. When the user is zooming in, the value of ML increases, and when the user is zooming out, ML decreases. When ML increases, the number of visible pixels in the outer video decreases and the number of visible pixels in the inner video increases. The RM's GPU shader uses the value of ML, a matching metric, and the frame number of each bitstream for synchronization to create the perception of watching a single video rather than two separate videos. In other embodiments, an additional "feathering" step can be performed by the shader to mask the boundary at the junction of the inner and outer videos.

[0059] When the ML is small and the number of visible pixels of the inner video is small, the rendering of the inner video can be skipped without any noticeable difference in the image quality of the displayed video. When the decoded video data of the inner video is not displayed, the decoding of the video data that is not displayed can be eliminated, thereby improving the performance and efficiency of the system. One of the ways this can be achieved is by utilizing the ML to determine which video bitstreams need to be decoded, and rendering only frames from the bitstreams that are actively being decoded. When the decoder is in an active state, the access units (AUs) of the bitstreams are decoded normally, and the decoded video data is sent to the RM for rendering to the display. When the decoder is in an inactive state, the decoding of the AUs can be skipped partially or completely, and the video data for the bitstreams corresponding to the inactive decoders is not rendered to the display.

[0060] When the ML changes, a decoder in the active state may become inactive and vice versa. Switching a decoder from the active state to the inactive state can be done immediately, but switching from the inactive state to the active state may not be done immediately. The reason for this is that the current AU may depend on the previous AU, and if the decoding of the previous AU is skipped when the decoder is in the inactive state, the current AU may have an error when decoded. To avoid this problem, switching from the inactive state to the active state may be performed only when the current AU is a key frame (IDR frame). To support this, a seeking state may be used where an inactive decoder first switches to a seeking state where the decoder waits for an IDR when the current AU is an IDR. When the current AU is an IDR, the decoder switches from the seeking state to the active state. The DM passes the bitstream IDs of the active decoders to the RM, and passes the invalid IDs for the decoders in the seeking or inactive state to the RM. The RM uses these IDs to render only valid pixels to the display.

[0061] For applications requiring high magnification levels or a smoother transition from zoomed-out to zoomed-in field of view, more than two camera fields of view may be required. For such use cases, more than two cameras with varying degrees of focal length or FOV may be used. As before, the same scene is captured simultaneously from a single position using these cameras.

[0062] An example of fields of view that can be captured using five cameras is shown in FIG. 6 (the five fields of view are labeled "Wide Angle 1", "Wide Angle 2", "Telephoto 1", "Telephoto 2", and "Standard").

[0063] The video data from each camera in Figure 6 may be synchronized and compressed as individual bitstreams or independently decodable substreams. All these streams may be decoded simultaneously and selectively rendered according to the desired ML, but a more efficient approach is to decode only the streams that will ultimately be displayed. The number of decoders required in the DM may be equal to the maximum number of video streams that are rendered simultaneously at any moment. For the setup shown in Figure 5 of one outer video and one inner video, the number of decoders required can be limited to two, even if more than two video streams are used. This is achieved using the "stream switching" strategy described below.

[0064] The stream processed by each decoder is determined by the value of ML. When the application starts, the first decoder (D1) can process the widest bitstream (B1) and the second decoder (D2) can process the second bitstream (B2) with a lower FOV. As the user increases the ML, there is a point where the pixels of B1 are no longer rendered on the display. D1 then transitions to a seek state and prepares to decode the next bitstream (B3) in the view list. The RM uses the bitstream ID and alignment metric passed from the decoder to display the decoded pixels of each bitstream at the appropriate degree of magnification. When the RM detects a change in the bitstream ID, it updates the rendering process to use the correct texture and sampling coordinates.

[0065] In other embodiments, the following steps may be performed during the encoding process to facilitate smooth stream switching.

[0066] First, the bitstreams use similar coding structures so that the same instance of the decoder can process AUs from multiple bitstreams without requiring extra memory. The IDRs of different bitstreams can be aligned and evenly spaced according to how fast the user can increase or decrease the ML. Second, the IDR positions and AU offsets for each bitstream can be pre-computed to avoid doing this in the DM.

[0067] In further embodiments, the DM may include one or more extra decoders to predict the next bitstreams to be processed based on ML and decode these streams before the decoded pixels are made visible on the display. This strategy can help increase the rate of change of ML. An alternative approach to achieve this is to code the bitstream using only IDR.

[0068] Now referring to FIG. 7, an alternative technique for applications requiring high magnification levels is multi-position content capture instead of multi-FOV content capture. Instead of capturing a scene from one position using cameras with different FOVs, the scene can be captured by using the same FOV but at different positions 700, 702 in the direction of scene capture. In other embodiments, both multi-FOV and multi-position content capture can be used together, as shown in FIG. 7. In other embodiments, the RM can include a stage for distortion correction between the multi-position content or the multi-view content. In other embodiments, the audio is also captured from different positions and the audio stream is also switched according to the ML for a more immersive experience.

[0069] Although particular embodiments are shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. A device, comprising: at least one storage device containing instructions executable by at least one processor that are not transitory signals, the instructions causing the processor to: presenting a first video having a first object having a first size; in response to a zoom command, presenting a second video having the first object having a second size larger than the first size; A device that aligns frames of the second video with frames of the first video using, at least in part, a ratio of a number of presented pixels in the first video to a number of presented pixels in the second video in a single dimension.

2. The instruction: Identifying a horizontal offset of the second video relative to a region of interest (ROI) of a frame of the first video; Identifying a vertical offset of the second video relative to the ROI of the frame of the first video; The device of claim 1 , operable to use the offset to align frames of the second video to frames of the first video.

3. 3. The device of claim 2, comprising first and second physical or virtual cameras having respective first and second fields of view (FOVs) configured to generate the respective first and second videos, the first FOV being larger than the second FOV, and the first and second cameras capturing images of the first object simultaneously with each other.

4. The device of claim 3 , wherein the instructions are executable to disable auto-exposure for the camera to facilitate blending of the first and second videos.

5. The instruction: synchronizing the first and second videos in time; The device of claim 3 , executable to encode the first and second videos as respective first and second bitstreams.

6. The device of claim 3 , wherein the instructions are executable to decode both bitstreams simultaneously using respective first and second decoders.

7. 4. The device of claim 3, wherein the instructions are executable to compress the first and second bitstreams into a single bitstream and to use a single decoder to decode the single bitstream.

8. 1. A video player comprising: At least one processor, the at least one processor comprising: outputting pixels of a first video having a first field of view (FOV) and a second video having a second FOV smaller than the first FOV; executing at least one decoding module (DM) and at least one rendering module (RM) to output the pixels, the DM comprising at least one decoder and the RM comprising at least one shader; and providing at least a portion of the first and second videos to at least one display in response to a zoom command, at least in part using the DM and / or RM.

9. The processor, 9. The video player of claim 8, configured to align the second video with the first video using a matching metric for a region of interest (ROI) in the first video.

10. The video player of claim 9 , wherein the aligning the videos uses a fixed alignment metric.

11. The video player of claim 9 , wherein the aligning of the videos uses a time-varying alignment metric.

12. The video player of claim 11 , wherein the alignment metric is received in metadata within a bitstream decoded by the DM.

13. The video player of claim 11 , wherein the processor is configured to calculate the match metric using motion estimation and image matching.

14. 9. The video player of claim 8, wherein the zoom command establishes a magnification level (ML), and the processor is configured to use the ML to determine what portions of the first and second videos are made visible on the display.

15. 15. The video player of claim 14, wherein the processor is configured to place upper and lower limits on ML to avoid magnification levels that result in image quality degradation.

16. 15. The video player of claim 14, wherein the processor is configured to decrease a number of visible pixels of the first video and increase a number of visible pixels of the second video in response to the zoom command that increases ML.

17. 10. The video player of claim 9, wherein the processor is configured to execute the at least one shader of the RM to use a magnification level (ML) associated with the zoom command, the alignment metric, and a frame number of an input bitstream associated with the video for synchronization to create the perception of watching a single video rather than two separate videos.

18. 20. The video player of claim 17, wherein the processor is configured to feather the videos using the at least one shader to mask boundaries between the videos.

19. 9. The video player of claim 8, wherein the processor is configured to skip rendering at least a portion of the second video when a magnification level (ML) established by the zoom command is a first ML such that a portion of the second video is not decoded and at least one decoder of the DM is in an inactive state.

20. 20. The video player of claim 19, wherein the processor is configured to, in response to a change in the ML, change the at least one decoder from an inactive state to an active state only when a current frame being decoded is a key frame.

21. 1. A method comprising: receiving at least first, second, and third bitstreams representing respective first, second, and third videos; decoding the first bitstream with a first decoder to render the first video and decoding the second bitstream with a second decoder to render the second video in response to at least a first requested magnification level (ML); presenting the first and second videos on a display; in response to a second request ML greater than the first request ML, decoding the third bitstream with the first decoder to render the third video and decoding the second bitstream with the second decoder to render the second video; presenting the second and third videos on a display.

22. using a bitstream identifier (ID) passed from at least one of said decoders to display decoded pixels of each bitstream according to said first or second request ML; 22. The method of claim 21, further comprising: updating the rendering to use different textures and sampling coordinates in response to a change in at least one bitstream ID.

23. using the same instance of the first decoder to process multiple bitstreams, where keyframes of the bitstreams are aligned and evenly spaced according to how fast a user can increase or decrease the ML; and pre-calculating keyframe positions and offsets for each bitstream.

24. using at least one decoder to predict a next bitstream to be processed based on a request ML; and decoding the subsequent bitstream to render decoded pixels before the decoded pixels are made visible on the display to facilitate a rate of change of the ML.

25. 15. The video player of claim 14, wherein the processor is configured to increase a number of visible pixels of the first video and decrease a number of visible pixels of the second video in response to the zoom command that decreases ML.

Images