Techniques for enabling high fidelity upscaling of video - Patent Application 20070123633

Abstract

During presentation of a first video (400), in response to a zoom command, a second video (408) is presented in combination with the first video. The first and second videos (400, 408) are generated from substantially the same camera position as each other, at substantially the same time, and with substantially the same resolution. However, the second video (408) is generated by a physical or virtual lens with a smaller field of view (FOV) than the FOV of the physical or virtual lens used to generate the first video. This technique creates the illusion of zooming without loss of resolution.

Description

[Technical field]

[0001] This application relates generally to techniques for zooming video without loss of resolution. [Background technology]

[0002] As recognized herein, when zooming a video, at high levels of zoom, the image becomes pixelated. This can be mitigated by providing a video with a very high resolution, but such a video consumes excessive storage and bandwidth. Currently, there is no adequate solution to the above computer-related technical problem. Summary of the Invention

[0003] Thus, in one aspect, at least one storage device that is not a temporary signal includes instructions executable by at least one processor, the instructions causing the processor to present a first video on a display, combine a second video with the first video in response to a zoom command, and present the second video combined with the first video on the display. The first video and the second video are generated from substantially the same camera position as each other, at substantially the same time, and with substantially the same resolution. However, to achieve the appearance of zooming without loss of fidelity, the second video is generated by a physical or virtual lens with a field of view (FOV) smaller than the FOV of the physical or virtual lens used to generate the first video. Alternatively, the second video may be generated by a camera with a shorter focal length than the first video.

[0004] The zoom command may be a first zoom command, and the instructions may be executable to present only the second video on the display in response to continued input of the first zoom command or input of a second zoom command. In some examples, the instructions may be executable to combine the second video with a third video and present the third video combined with the second video on the display in response to continued input of the first zoom command or input of a third zoom command after the second zoom command, where the first, second, and third videos may be generated from substantially the same camera position as one another, at substantially the same time, and with substantially the same resolution, but the third video is generated by a physical or virtual lens having a smaller FOV than the FOV of the physical or virtual lens used to generate the second video.

[0005] In effect, the processor may access a fourth video and a fifth video, each having a successively smaller FOV than the immediately preceding video, for use with successive input of zoom commands.

[0006] The display may be a head mounted display (HMD) such as a virtual reality (VR), three dimensional (3D) computer game display, or the like.

[0007] In another aspect, a method includes presenting a first video on a display in a wide-angle mode. In response to a zoom-in command, the method includes presenting the first video in a standard-angle mode and in response to a continued zoom-in command, presenting the first video in a telephoto mode. In response to the continued zoom-in command, the method includes presenting a second video on the display in the wide-angle mode.

[0008] In another aspect, an apparatus includes at least a processor programmed to present a first video on a display and, in response to a zoom command, present a second video on the display, the second video being generated by a physical or virtual lens having a field of view (FOV) smaller than a FOV of a physical or virtual lens used to generate the first video and / or based on a focal length shorter than a focal length at which the first video is presented.

[0009] The details of the present application, both as to its structure and operation, can best be understood in reference to the accompanying drawings, in which like reference numerals refer to like parts and in which: [Brief description of the drawings]

[0010] [Figure 1] 1 is a block diagram of an exemplary system in accordance with the present principles; [Diagram 2] Exemplary logic consistent with the present principles is illustrated in exemplary flow chart form. [Diagram 3] Illustrates a user zooming by stepping along the Z axis. [Figure 4] 1 illustrates a schematic diagram of a zoom. [Diagram 5] 3 illustrates a schematic representation of an offset between videos. [Figure 5A] FIG. 2 is a block diagram of an example rendering and decoding module. [Figure 6] Illustrates the views from five cameras. [Figure 7] Illustrates multi-FOV and multi-position content capture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present disclosure generally relates to computer ecosystems including aspects of consumer electronics (CE) device networks, including but not limited to computer gaming networks, including wireless networks operating on 5G or ATSC 3.0. The systems herein may include a server component and a client component that may be connected via a network such that data may be exchanged between the client component and the server component. The client components may include one or more computing devices including gaming consoles, such as Sony PlayStation® or Microsoft, or Nintendo, or other manufacturer's gaming consoles, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (e.g., smart televisions, Internet-enabled televisions), portable computers, such as laptops and tablet computers, and other mobile devices, including smartphones, as well as additional examples described below. These client devices may operate in a variety of operating environments. For example, some client computers may employ, as examples, a Linux operating system, a Microsoft operating system, or a Unix operating system, or an Apple or Google operating system. These operating environments may be used to run one or more browsing programs, such as browsers created by Microsoft, Google, or Mozilla, or other browser programs that can access websites hosted by Internet servers, as described below. Additionally, an operating environment consistent with the present principles may be used to run one or more computer game programs.

[0012] Servers and / or gateways may be used, which may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the clients and servers may be connected via a local intranet or a virtual private network. The servers or controllers may be exemplified by game consoles such as the Sony PlayStation®, personal computers, and the like.

[0013] Information may be exchanged between the clients and the servers over a network. For this purpose and for security, the servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers may form an apparatus that implements a method for providing a secure community, such as an online social website or a gamer network, to network members.

[0014] The processor may be a single-chip or multi-chip processor capable of implementing logic through various wiring such as address lines, data lines, and control lines, and registers and shift registers.

[0015] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or illustrated in the drawings may be combined, interchanged, or excluded from other embodiments.

[0016] "A system having at least one of A, B, and C" (also "a system having at least one of A, B, or C" and "a system having at least one of A, B, C") includes systems having A only, B only, C only, A and B together, A and C together, B and C together, and / or A, B, and C together.

[0017] Referring now specifically to FIG. 1, an exemplary system 10 is shown, which may include one or more of the exemplary devices described above and further described below in accordance with the present principles. A first of the exemplary devices included in the system 10 is a consumer electronics (CE) device, such as, but not limited to, an audio-video device (AVD) 12, such as an Internet-enabled TV having a TV tuner (equivalently, a set-top box that controls the TV). The AVD 12 may also alternatively be a computerized Internet-enabled ("smart") phone, a tablet computer, a notebook computer, a head-mounted device (HMD) and / or a headset, such as smart glasses or a VR headset, another wearable computerized device, a computerized Internet-enabled music player, a computerized Internet-enabled headphones, a computerized Internet-enabled implantable device, such as an implantable skin device. It should nevertheless be understood that the AVD 12 is configured to implement the present principles (e.g., to communicate with other CE devices to implement the present principles, to execute the logic described herein, and to perform any other functions and / or operations described herein).

[0018] Thus, to realize such principles, AVD 12 may be established by some or all of the components shown in Figure 1. For example, AVD 12 may include one or more touch-enabled displays 14, which may be implemented by high-definition or ultra-high-definition "4K" or higher flat screens. Touch-enabled display(s) 14 may include, for example, a capacitive or resistive touch-sensing layer having a grid of electrodes for touch sensing consistent with the present principles.

[0019] The AVD 12 may also include one or more speakers 16 for outputting audio in accordance with the present principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to the AVD 12 to control the AVD 12. The exemplary AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22, such as the Internet, a WAN, a LAN, etc., under the control of one or more processors 24. Thus, the interface 20 may be a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that the processor 24 controls the AVD 12 to implement the present principles, including other elements of the AVD 12 described herein, such as controlling the display 14 to present images thereon and receiving input therefrom. It should further be noted that the network interface 20 may be a wired or wireless modem or router, or a wireless telephone transceiver, or other suitable interface, such as the Wi-Fi transceiver discussed above.

[0020] In addition to the above, AVD 12 may also include one or more input and / or output ports 26, such as a High-Definition Multimedia Interface (HDMI) port or a Universal Serial Bus (USB) port for physically connecting to another CE device, and / or a headphone port for connecting headphones to AVD 12 to present audio from AVD 12 to a user through the headphones. For example, input port 26 may be connected to a source 26a of cable or satellite audio-video content via wire or wireless. Thus, source 26a may be a separate or integrated set-top box, or a satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. Source 26a when implemented as a game console may include some or all of the components described below in connection with CE device 48.

[0021] The AVD 12 may further include one or more computer memory / computer readable storage media 28, such as non-transitory disk-based or solid-state storage, possibly embodied within the AVD chassis as a standalone device or as a personal video recording device (PVR) or video disk player either inside or outside the AVD chassis for playing AV programs, or as a removable memory medium or server as described below. In some embodiments, the AVD 12 may also include a location or position receiver, such as, but not limited to, a cellular receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographic location information from a satellite or cellular tower and provide that information to the processor 24 and / or in conjunction with the processor 24 to determine the altitude at which the AVD 12 is disposed. The component 30 may also be implemented by an inertial measurement unit (IMU), typically including a combination of accelerometers, gyroscopes, and magnetometers, or by event-based sensors to determine the position and orientation of the AVD 12 in three dimensions.

[0022] Continuing with the description of the AVD 12, in some embodiments, the AVD 12 may include one or more cameras 32, such as an infrared camera, a digital camera such as a webcam, event-based sensors, and / or cameras integrated into the AVD 12 and controllable by the processor 24, capable of collecting photos / images and / or videos in accordance with the present principles. The AVD 12 may also include a Bluetooth transceiver 34 and other near field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An exemplary NFC element can be a radio frequency identification (RFID) element.

[0023] Continuing further, the AVD 12 may include one or more auxiliary sensors 38 (e.g., pressure-sensitive sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (e.g., for sensing gesture commands)) that provide input to the processor 24. For example, one or more of the auxiliary sensors 38 may include one or more pressure sensors forming a layer of the touch-enabled display 14 itself, which may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc.

[0024] The AVD 12 may also include an OTA television broadcast port 40 for receiving terrestrial television broadcasts that provide input to the processor 24. In addition to the above, it is noted that the AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR Data Association (IRDA) device. A battery (not shown) may be provided to power the AVD 12, and may also be a kinetic energy harvester that may convert kinetic energy into power to charge the battery and / or power the AVD 12. A graphics processing unit (GPU) 44 and a field programmable gate array 46 may be included. One or more haptics / vibration generators 47 may be provided to generate haptic signals that can be sensed by a person holding or in contact with the device. The haptic generator 47 may vibrate all or a portion of the AVD 12 using an electric motor connected to a centered and / or off-balanced weight via a rotatable shaft of the motor, such that the shaft can rotate under the control of the motor (and which may in turn be controlled by a processor such as processor 24) to generate vibrations of various frequencies and / or amplitudes, as well as simulated forces in various directions.

[0025] 1, in addition to the AVD 12, the system 10 may include one or more other CE device types. In one embodiment, the first CE device 48 may be a computer game console that can be used to transmit computer game audio and video to the AVD 12 via commands sent directly to the AVD 12 and / or through a server, described below, while the second CE device 50 may include similar components as the first CE device 48. In the embodiment shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include a head-up transparent or non-transparent display for presenting AR / MR or VR content, respectively.

[0026] In the illustrated embodiment, only two CE devices are shown, but it should be understood that fewer or more devices may be used. The devices herein may implement some or all of the components shown for AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for AVD 12.

[0027] Referring now to the at least one server 52 mentioned above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56, such as disk-based or solid-state storage, and at least one network interface 58 that, under the control of the server processor 54, enables communication with other devices of Figure 1 via the network 22, and may indeed facilitate communication between the server and client devices in accordance with the present principles. It should be noted that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as, for example, a wireless telephone transceiver.

[0028] Thus, in some embodiments, server 52 may be an entire Internet server or server "farm" and may include and perform "cloud" functionality such that, for example, in an exemplary embodiment for a network gaming application, devices of system 10 may access the "cloud" environment via server 52. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown in FIG.

[0029] The components shown in the following figures may include some or all of the components shown in Figure 1. Any user interfaces (UIs) described herein may be integrated and / or extended, and UI elements may be mixed and matched between UIs.

[0030] 2 illustrates that in an embodiment, "N" videos are generated by each virtual or physical camera and associated physical or virtual lens. N can be an integer greater than or equal to 2. In one embodiment, N is equal to 5.

[0031] In one embodiment, each of the N videos has the same resolution, such as but not limited to 4K, however, in other embodiments, the N videos may not all have the same resolution.

[0032] In either case, in one embodiment, the videos may be taken from the same or substantially the same location and at the same or substantially the same time. "Substantially the same location" means, for example, within the constraints of physically locating two cameras at the same place; the cameras may be closely positioned side-by-side, albeit separated by the width of the camera housings. "Substantially the same time" means at the same real or virtual time, or within a few seconds of each other.

[0033] However, a first video is generated using a physical or virtual lens having a first field of view (FOV), a second video is generated using a physical or virtual lens having a second FOV smaller than the first FOV, and each successive video is generated with a successively smaller FOV than the preceding video in the chain. However, each FOV may be centered on the same location or point or center. It is noted that in addition to or instead of successively smaller FOVs, the physical or virtual cameras may have successively shorter focal lengths.

[0034] Moving to block 202, the videos are synchronized with each other, for example, by aligning key frames in each video with each other, and in a specific embodiment, encoding the videos as H264. Alignment is described further below.

[0035] When a user wants to play a video, the video is presented in block 204 using a first video, i.e., the video with the widest FOV. In block 206, when the user zooms in using an input device or by moving his head along the Z axis while wearing the HMD presenting the video, the video with the next smaller FOV is combined with the first video, eventually replacing the first video. Continued zooming results in successive videos presented with successively smaller FOVs, such that zooming is emulated without loss of fidelity. Thus, during playback, content from the telephoto camera is inserted into content from the wide-angle camera according to a pre-computed alignment metric to create the perception of watching a single video. With accurate alignment, the presence of the inner video displayed within the outer video is not obvious to the viewer.

[0036] FIG. 3 illustrates a user 300 wearing an HMD 302 that zooms by moving along a Z axis 304.

[0037] Fig. 4 continues to further illustrate. Note that Fig. 4 illustrates an implementation in which, in addition to using different FOVs, the scene is captured at different positions, while Fig. 6 described below illustrates the case in which three or more videos are captured from the same position. More specifically, in the embodiment shown in Fig. 4, multiple (e.g., three) lenses are used with different FOVs to capture three videos from the same physical or virtual camera arrangement, and the same three lenses with respective different FOVs are used to capture three videos from a second arrangement. Thus, after recording, six videos are captured simultaneously.

[0038] A first video 400 is shown in its maximum wide-angle mode 402. As the user zooms in, the video is shown in its standard-angle mode 404, and finally, under continued zoom, in its telephoto mode 406, with each mode occupying more and more of the display. It should be understood that the transition between the three modes shown is continuous and gradual as the user zooms, but for simplicity only three common modes are shown.

[0039] When the zoom in the telephoto mode 406 of the first video reaches a threshold limit, further zooming results in combining the first video with the second video 408 in its maximum wide-angle mode 410. It should be understood that the second video 408 may eventually or immediately completely replace the first video as the zoom progresses from the telephoto mode 406 of the first video to the wide-angle mode 410 of the second video 408.

[0040] As the user continues to zoom in, the second video 408 is shown in its standard angle mode 412, and eventually, under continued zoom, in its telephoto mode 414, with each mode occupying the display.

[0041] Continued zooming from the telephoto mode 414 of the second video results in combining the second video with the third video 416 having its maximum wide-angle mode 418. It should be understood that the third video 414 may eventually or immediately replace the second video entirely as the zoom progresses from the telephoto mode 414 of the second video to the wide-angle mode 418 of the third video 416.

[0042] As the user continues to zoom in, the third video 416 is shown in its standard angle mode 420, and eventually, under continued zoom, in its telephoto mode 422, with each mode occupying the display. Note that if the scene is only captured from a single position, steps 408-422 are not available.

[0043] Although FIG. 4 illustrates the use of three videos generated by physical or virtual lenses, each having a successively smaller FOV, it should be understood that consistent with the principles of FIG. 4, only two videos need be used, or four or more videos may be used.

[0044] Note that multiple videos, each with a smaller FOV, may be generated for multiple likely regions of user focus. A central focus may be used at the baseline, and then offsets in terms of distance and direction from that point may be used and transmitted as metadata to indicate when the user is focusing on a point that is the offset away from the central focus. For each offset, a series of nested videos may be precomputed or may be computed on the fly for a particular focus as the user focuses on the particular point. If the user happens to focus on a point where there are no nested videos with smaller FOVs, traditional magnification techniques may be used.

[0045] A heatmap of previous user focus for each scene can be used to determine which points in the scene should have a set of nested videos generated for them: only the video of the areas on which the user focused can be decoded.

[0046] Reference is now made to FIG. 5 for a discussion of match metrics that may be determined prior to or during capture.

[0047] The insertion ratio (R) can be determined to be the ratio of the number of pixels in the outer video (wider FOV) to the number of pixels in the inner video (narrower FOV) in a single dimension. In Figure 5, W0 is the width in pixels of the outer video and W1 is the width in pixels of the inner video, and after alignment, R = W0 / W1. The insertion ratio depends on the focal lengths of the two cameras and the resolution of the camera sensors.

[0048] The horizontal offset (Oh) is shown in Figure 5 and is the horizontal offset of the inner video or ROI measured from the center of the frame of the outer video. Similarly, the vertical offset (Ov) is the vertical offset of the inner video or ROI measured from the center of the frame of the outer video.

[0049] FIG. 5 illustrates that frames of videos with wider and narrower FOVs are aligned during display using the above offsets with the alignment metric. Specifically, a camera arrangement is determined, where two cameras with different FOVs capture the same scene simultaneously. In the simplest case, Oh=Ov=0, and the ROI is the center of the video frame. An interpolation ratio of 2 could be achieved by using an FOV of 60 for the wide-angle lens and an FOV of about 32.2 for the telephoto lens. Disabling automatic features such as autoexposure on the camera facilitates blending of the two frames during display.

[0050] See FIG. 5A. The raw video from the two cameras, labeled 500, 502, is synchronized and encoded as two separate bitstreams. In this case, two decoders are used to decode both bitstreams simultaneously. In other embodiments using one decoder, the video data from each camera may be compressed as a single bitstream but may be independently decodable, for example as HEVC tiles. In either case, a video player 504, which generates output pixels for a display 506, includes a decoding module (DM) 508 and a rendering module (RM) 510. The DM, in turn, includes one or more decoders 512 that are capable of decoding the compressed bitstream(s). The RM includes GPU shaders that can sample the video texture and render it to the display.

[0051] The match metric may be fixed or may change over time. In the fixed case, the match metric may be sent to the DM and / or RM only once. In the case of a dynamic match metric, the DM and / or RM may be updated with each change in the metric. One way to achieve this is to pass the match metric as metadata in the compressed bitstream. In other embodiments, the match metric may be calculated automatically using motion estimation and image matching algorithms.

[0052] A video player that renders the decoded video data on a display accepts magnification control from a user using a device such as a mouse or a video game controller. The magnification level (ML) selected by the user is used to determine the portions of the outer and inner videos that will be visible on the display. The system can place upper and lower bounds on ML to avoid magnification levels that result in image quality degradation. When the user is zooming in, the value of ML increases, and when the user is zooming out, ML decreases. When ML increases, the number of visible pixels in the outer video decreases and the number of visible pixels in the inner video increases. The RM's GPU shader uses the value of ML, a matching metric, and the frame number of each bitstream for synchronization to create the perception of watching a single video rather than two separate videos. In other embodiments, an additional "feathering" step can be performed by the shader to mask the boundary at the junction of the inner and outer videos.

[0053] When the ML is small and the number of visible pixels of the inner video is small, the rendering of the inner video can be skipped without noticeable difference in the image quality of the displayed video. When the decoded video data of the inner video is not displayed, the decoding of the video data that is not displayed can be omitted, thereby improving the performance and efficiency of the system. One of the ways this can be achieved is to utilize the ML to determine which video bitstreams need to be decoded, and render only frames from the bitstreams that are actively decoded. When the decoder is in an active state, the access units (AUs) of the bitstreams are decoded normally, and the decoded video data is sent to the RM for rendering to the display. When the decoder is in an inactive state, the decoding of the AUs can be skipped partially or completely, and the video data for the bitstreams corresponding to the inactive decoders is not rendered to the display.

[0054] When the ML changes, a decoder in the active state may become inactive and vice versa. Switching a decoder from the active state to the inactive state can be done immediately, but switching from the inactive state to the active state may not be done immediately. The reason for this is that the current AU may depend on the previous AU, and if the decoding of the previous AU is skipped when the decoder is in the inactive state, the current AU may have an error when decoded. To avoid this problem, switching from the inactive state to the active state may be performed only when the current AU is a key frame (IDR frame). To support this, a seek state may be used, in which when the ML exceeds a threshold, the inactive decoder first switches to a seek state where the decoder waits for an IDR. When the current AU is an IDR, the decoder switches from the seek state to the active state. The DM passes the bitstream IDs of the active decoder to the RM, and passes the invalid IDs for the decoders in the seek state or in the inactive state to the RM. The RM uses these IDs to render only valid pixels to the display.

[0055] For applications requiring high magnification levels or smoother transitions from zoomed-out to zoomed-in views, more than two camera views may be required. For such use cases, three or more cameras with varying degrees of focal length or FOV may be used. As before, the same scene is captured from a single position simultaneously using these cameras.

[0056] An example of the fields of view that can be captured using five cameras is shown in FIG. 6 (the five fields of view are labeled "Wide Angle 1", "Wide Angle 2", "Telephoto 1", "Telephoto 2", and "Standard").

[0057] The video data from each camera in Figure 6 may be synchronized and compressed as individual bitstreams or independently decodable substreams. All these streams may be simultaneously decoded and selectively rendered according to the desired ML, but a more efficient approach is to decode only the streams that will ultimately be displayed. The number of decoders required in the DM may be equal to the maximum number of video streams that are rendered simultaneously at any moment. The number of decoders required for the setup shown in Figure 5 of one outer video and one inner video can be limited to two, even if more than two video streams are used. This is achieved using the "stream switching" strategy described below.

[0058] The stream processed by each decoder is determined by the value of ML. When the application starts, the first decoder (D1) can process the widest bitstream (B1) and the second decoder (D2) can process the second bitstream (B2) with a lower FOV. As the user increases the ML, there is a point where the pixels of B1 are no longer rendered on the display. D1 then transitions to a seek state and prepares to decode the next bitstream (B3) in the viewing list. The RM uses the bitstream ID and alignment metric passed from the decoder to display the decoded pixels of each bitstream at the appropriate magnification. When the RM detects a change in the bitstream ID, it updates the rendering process to use the correct texture and sampling coordinates.

[0059] In other embodiments, to facilitate smooth stream switching, the following steps may be performed during the encoding process.

[0060] First, the bitstreams use similar coding settings so that the same instance of the decoder can process AUs from multiple bitstreams without requiring additional memory. The IDRs of different bitstreams can be aligned and evenly spaced according to the rate at which the user can increase or decrease the ML. Second, the IDR positions and AU offsets for each bitstream can be pre-computed to avoid doing the same calculations in the DM.

[0061] In further embodiments, the DM may include one or more additional decoders to predict the next bitstreams to be processed based on ML and decode these streams before the decoded pixels are made visible on the display. This strategy can help increase the rate of change of ML. An alternative approach to achieve this is to code the bitstream using only IDR.

[0062] Now referring to FIG. 7, an alternative technique for applications requiring high magnification levels is multi-location content capture instead of multi-FOV content capture. Instead of capturing a scene from one location using cameras with different FOVs, the scene can be captured at locations 700, 702 using the same FOV but with different scene capture directions. In other embodiments, both multi-FOV and multi-location content capture can be used together, as shown in FIG. 7. In other embodiments, the RM can include a stage for distortion correction between multi-location content or multi-view content. In other embodiments, audio is also captured from different locations and audio streams are also switched according to ML for a more immersive experience.

[0063] Although particular embodiments are shown and described in detail herein, it should be understood that the subject matter encompassed by the present invention is limited only by the scope of the claims.

Claims

1. A device, comprising: at least one storage device containing instructions executable by at least one processor other than a transitory signal, said instructions causing said processor to: Presenting the first video on a display; a device that, in response to a zoom command, combines a second video with the first video and causes the second video combined with the first video to be presented on the display, wherein the first video and the second video are generated from substantially the same camera position as each other and at substantially the same resolution, and the second video is generated by a physical or virtual lens having a field of view (FOV) that is smaller than the FOV of a physical or virtual lens used to generate the first video, to provide the appearance of zooming without loss of resolution.

2. The device of claim 1 , wherein the first video and the second video are generated by a virtual lens.

3. The device of claim 1 , wherein the first video and the second video are generated by a physical lens.

4. 2. The device of claim 1, wherein the zoom command is a first zoom command and the instructions are executable to present only the second video on the display in response to continued input of the first zoom command or input of a second zoom command.

5. 5. The device of claim 4, wherein the instructions are executable to combine the second video with a third video in response to continued input of the first zoom command or input of a third zoom command after the second zoom command, and to present the third video combined with the second video on the display, wherein the first, second, and third videos are generated from substantially the same camera position as one another, at substantially the same time, and with substantially the same resolution, and the third video is generated by a physical or virtual lens having a smaller FOV than the FOV of a physical or virtual lens used to generate the second video.

6. 6. The device of claim 5, wherein the processor has access to a fourth video and a fifth video, each having a successively smaller FOV than the immediately preceding video, for use with successive input of zoom commands.

7. The device of claim 1 comprising the display.

8. The device of claim 7 , wherein the display comprises a head mounted display (HMD).

9. 1. A method comprising: presenting the first video on a display; and in response to a zoom-in command, presenting a second video on the display.

10. 10. The method of claim 9, comprising presenting the first video in a wide-angle mode and then, in response to a first zoom command, presenting the video in a standard-angle mode.

11. The method of claim 10 , comprising presenting the video in a telephoto mode after a second zoom command that follows the first zoom command.

12. The method of claim 11 , wherein the zoom command that causes the presentation of the second video is a third zoom command following the second zoom command, and the second video is presented in a wide-angle mode.

13. 10. The method of claim 9, wherein the first video is generated using a lens having a first field of view (FOV) and the second video is generated using a lens having a second FOV that is smaller than the first FOV.

14. The method of claim 13 , wherein the first video and the second video are generated to be from cameras located at the same place and at the same time.

15. 10. The method of claim 9, wherein the first video and the second video are captured using the same camera field of view from respective first and second physical or virtual camera positions.

16. 1. An apparatus comprising: The method includes at least one processor, the processor comprising: Presenting the first video on a display; The device is programmed to present a second video on the display in response to a zoom command, the second video being generated by a physical or virtual lens having a field of view (FOV) smaller than a FOV of a physical or virtual lens used to generate the first video, and / or the second video being generated based on a focal length shorter than a focal length at which the first video is presented.

17. The apparatus of claim 16 , wherein the first video and the second video have substantially the same resolution.

18. The apparatus of claim 16 , wherein the first video and the second video are generated by a virtual lens.

19. The apparatus of claim 16 , wherein the first video and the second video are generated by a physical lens.

20. 17. The device of claim 16, wherein the zoom command is a first zoom command, and the processor is programmed to present only the second video on the display in response to continued input of the first zoom command or input of a second zoom command.

21. 21. The apparatus of claim 20, wherein the processor is programmed to, in response to continued input of the first zoom command or input of a third zoom command after the second zoom command, combine the second video with a third video and present the third video combined with the second video on the display, wherein the first, second, and third videos are generated from substantially the same camera position as one another, at substantially the same time, and with substantially the same resolution, and the third video is generated by a physical or virtual lens having a smaller FOV than the FOV of a physical or virtual lens used to generate the second video.

22. 22. The apparatus of claim 21, wherein the processor has access to a fourth video and a fifth video, each having a successively smaller FOV than the immediately preceding video, for use with successive input of zoom commands.

23. The apparatus of claim 16 , wherein the first video and the second video are generated from substantially the same camera position and at substantially the same time as each other.

24. The apparatus of claim 16 , wherein the first video and the second video are generated from respective first and second camera positions that are spaced apart from one another.

25. 10. The method of claim 9, comprising reducing required computational power by selectively decoding only content in the video represented by at least one bitstream that is made visible.

26. 10. The method of claim 9, comprising smoothly switching between respective bitstreams of camera views, at least in part by using similar encoding settings, such that the same instance of a decoder can process bitstream access units (AUs) from the multiple bitstreams without requiring additional memory, where keyframes of the different bitstreams are aligned and evenly spaced according to a rate at which a user can increase or decrease the magnification level, and where keyframe positions and AU offsets for each bitstream are pre-calculated to avoid performing the same calculations in the decoding module.

27. 10. The method of claim 9, comprising: executing a decoding module (DM) including one or more additional decoders to predict a next bitstream to be processed; and decoding the next bitstream before decoded pixels from the next bitstream are visible on the display.

28. 10. The method of claim 9, comprising encoding a bitstream associated with the video using only keyframes.

29. 10. The method of claim 9, including switching audio streams as well as switching video when zooming.

Images