Zoom in or zoom out with slow-motion video capture

Abstract

An aspect is described for a device for generating video content, the device including a first camera and one or more processors coupled to the first camera, the one or more processors configured to receive a first set of frames captured by the first camera at a first rate, process the first set of frames to generate video content for display, receive a selection to zoom in or out, receive a second set of frames captured by the first camera at a second rate less than the first rate during the zoom in or out to a zoom threshold for the first camera, process the second set of frames to generate video content for display, and generate video content for playback including the first set of frames and the second set of frames at a third rate less than the first rate.

Description

Claim of Priority

[0001]

[0001] This application claims the priority of U.S. Application No. 17 / 156,219, filed on January 22, 2021, the entire content of which is incorporated herein by reference.

Technical Field

[0002]

[0002] This disclosure relates to image processing.

Background Art

[0003]

[0003] A camera device includes one or more cameras that capture frames (e.g., images). Examples of camera devices include stand-alone digital cameras or digital video camcorders, wireless communication device handsets with cameras such as mobile phones having one or more cameras, cellular or satellite radiotelephones, personal digital assistants (PDAs) with cameras including so-called "webcams", panels or tablets, gaming devices, computer devices, or any device having digital imaging or video capabilities.

[0004]

[0004] The camera device processes the captured frames and outputs the frames for display. In some examples, the camera device captures frames at a first rate and displays the frames at a second rate that is less than the first rate. The result of displaying the frames at a second rate that is less than the first rate at which the frames are captured is that the image content appears to move in slow motion.

Summary of the Invention

[0005]

[0005] Generally speaking, this disclosure describes techniques for slow-motion video capture using zoom-in or zoom-out capabilities in a multi-camera device. Each camera on the camera device may be associated with a zoom coefficient range between its respective zoom thresholds. For example, the zoom coefficient for a first camera may be a magnification between 0.5x and 1x of the captured image content. In this example, the zoom coefficient range is from 0.5x to 1x, the first zoom threshold is 0.5x, and the second zoom threshold is 1x. Other cameras on the device may have other zoom coefficient ranges (e.g., magnification from 1x to 2x). Also, each camera may be configured to capture frames according to its respective camera parameters. Regarding zooming in (for example, increasing the magnification) or zooming out (for example, decreasing the magnification), the camera processor processes the frames captured by the first camera based on the camera parameters for the first camera until a zoom threshold for the first camera is reached, and at the point when the zoom threshold for the first camera is reached, the camera processor processes the frames captured by the second camera (for example, the camera processor switches from the wide-angle camera to the telephoto camera).

[0006]

[0006] In one or more examples, during zooming in or zooming out (for example, during the time it takes for a frame captured by the first camera to zoom in or zoom out from the current zoom coefficient to a zoom threshold), the first camera may be configured to capture frames at a lower frame rate than the rate at which the first camera can capture frames when it is not zooming in or zooming out. As will be described in more detail, by reducing the capture rate during zooming in or zooming out, the exemplary technique can reduce the number of inferior quality frames that are captured and displayed by the first camera before frames captured by the second camera are displayed.

[0007]

[0007] For example, in slow motion, captured frames are displayed at a lower rate than the rate at which the frames were captured. Also, camera parameters may not be optimized between zooming in or zooming out, which results in lower image quality for frames captured during the time it takes to zoom in or zoom out from the current zoom coefficient to the zoom threshold. In slow motion, if there are many frames with lower image quality, the viewer experience is negatively affected. By reducing the number of lower-quality frames that are captured and then displayed, exemplary techniques can improve the overall operation of the camera device, and thus the resulting device can provide the ability to zoom in or zoom out even when slow-motion video capture is enabled.

[0008]

[0008] In one example, the present disclosure describes a device for generating video content, the device comprising a first camera and one or more processors coupled to the first camera, the one or more processors being configured to receive a first set of frames captured by the first camera at a first rate, process the first set of frames to generate video content for display, receive a choice to zoom in or zoom out, receive a second set of frames captured by the first camera at a second rate less than the first rate during the zoom in or zoom out to a zoom threshold for the first camera, process the second set of frames to generate video content for display, and generate video content for playback at a third rate less than the first rate, including the first set of frames and the second set of frames.

[0009]

[0009] In one example, the present disclosure describes a method for generating video content, the method comprising: using one or more processors to receive a first set of frames captured by a first camera at a first rate; using one or more processors to process the first set of frames to generate video content for display; using one or more processors to receive a selection to zoom in or zoom out; using one or more processors to receive a second set of frames captured by the first camera at a second rate lower than the first rate during the zoom in or zoom out to a zoom threshold for the first camera; using one or more processors to process the second set of frames to generate video content for display; and using one or more processors to generate video content for playback, including the first set of frames and the second set of frames, at a third rate lower than the first rate.

[0010]

[0010] In one example, the present disclosure describes a computer-readable storage medium storing instructions, which, when executed, causes one or more processors to receive a first set of frames captured by a first camera at a first rate; process the first set of frames to generate video content for display; receive a choice to zoom in or zoom out; receive a second set of frames captured by the first camera at a second rate less than the first rate while zooming in or zooming out to a zoom threshold for the first camera; process the second set of frames to generate video content for display; and generate video content for playback at a third rate less than the first rate, including the first set of frames and the second set of frames.

[0011]

[0011] In one example, the present disclosure describes a device for generating video content, the device comprising means for receiving a first set of frames captured by a first camera at a first rate; means for processing the first set of frames to generate video content for display; means for receiving a choice to zoom in or zoom out; means for receiving a second set of frames captured by the first camera at a second rate less than the first rate during zoom in or zoom out to a zoom threshold for the first camera; means for processing the second set of frames to generate video content for display; and means for generating video content for playback, including the first set of frames and the second set of frames, at a third rate less than the first rate.

[0012]

[0012] Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0013] [Figure 1]

[0013] A block diagram of a device configured to perform one or more of the exemplary techniques described herein. [Figure 2]

[0014] A block diagram showing the camera processor of the device in Figure 1 in more detail. [Figure 3]

[0015] A conceptual diagram illustrating an exemplary method for enabling zoom in or zoom out. [Figure 4]

[0016] A timing diagram illustrating an example of zooming in or zooming out. [Figure 5]

[0017] A flowchart illustrating an exemplary method of operation by one or more examples described in this disclosure. [Modes for carrying out the invention]

[0014]

[0018] The exemplary techniques described herein relate to slow-motion video capture using zoom-in or zoom-out capabilities in a multi-camera device. The camera device may include multiple cameras to support zooming in or zooming out of video content. For example, the camera device may include an ultrawide camera (configured to capture a field of view greater than 120°, e.g., 120° to 180°), a wide camera (configured to capture a field of view greater than 60°, e.g., 60° to 120°), and a telephoto camera (configured to capture a field of view less than 60°, e.g., 10° to 60°). Each of the multiple cameras may be associated with its own zoom coefficient range and zoom threshold. For example, the ultrawide camera may provide a zoom coefficient range of 0.5x to 1x, where 0.5x and 1x are zoom thresholds. The wide camera may provide a zoom coefficient range of 1x to 2x, where 1x and 2x are zoom thresholds. Telephoto cameras can offer zoom factors greater than 2x, where 2x is the zoom threshold. Zoom factors of 0.5x, 1x, and 2x can refer to the reproduction ratio of objects on the respective camera's sensor (for example, a 0.5x zoom factor for an ultrawide camera means a 1 / 2-inch object is reproduced as 0.25 times the object's size on the ultrawide camera's sensor).

[0015]

[0019] Each camera may be associated with a respective zoom coefficient based on its optical zoom capability and / or digital zoom capability. That is, each camera may offer zooming in or zooming out over a zoom coefficient range based on one or a combination of optical zoom and digital zoom. Examples of optical zoom and digital zoom are described below.

[0016]

[0020] In the example above, there is no overlap in the zoom coefficient ranges for each camera. However, in some examples, there may be overlap in the zoom coefficient ranges. For example, the zoom coefficient range for an ultrawide camera may be 0.5x to 1.3x, the zoom coefficient range for a wide camera may be 1x to 2.3x, and the zoom coefficient range for a telephoto camera may be greater than 2x. In this example, the zoom thresholds for the ultrawide camera are 0.5x and 1.3x, the zoom thresholds for the wide camera are 1x and 2.3x, and the zoom threshold for the telephoto camera is 2x.

[0017]

[0021] The zoom coefficient ranges and zoom thresholds are provided as examples only and should not be considered limiting. The exemplary techniques described herein are not limited to any particular zoom coefficient range or zoom threshold.

[0018]

[0022] In some examples, when zooming in or zooming out, there is a handoff between different cameras once the respective zoom threshold is reached. For example, a user of a camera device may first capture video content using an ultrawide camera with a zoom factor of 0.5x, and the camera device's camera processor may process the frames captured by the ultrawide camera. The user may then zoom in (for example, using a button or pinch motion). In this example, during the zoom-in, the camera processor may process the frames captured by the ultrawide camera until the ultrawide camera's zoom threshold is reached (for example, 1x). The camera processor may then process the frames captured by the wide camera.

[0019]

[0023] One exemplary method for maintaining the quality of captured video content is for each camera to determine its own camera parameters before video capture. Examples of camera parameters include the 3A parameters (autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters), as will be explained in more detail. The camera parameters are already known even if the camera processor is not processing frames from a particular camera. In this case, when the camera processor should switch the processing of frames from the current camera to the subsequent camera, there is no delay in determining the camera parameters for the subsequent camera because the camera parameters for the subsequent camera have been predetermined. In this way, the handoff from the current camera to the subsequent camera (e.g., switching in processing frames) can be performed with limited impact on quality and limited delay.

[0020]

[0024] In addition to including multiple cameras to support zooming in or out, a camera device also offers an option for slow-motion video capture. In slow-motion video capture, the camera device's cameras capture frames at a first rate greater than a second rate at which the captured frames are displayed. For example, a camera might capture 10 seconds of video content at a capture rate of 120 frames per second (fps). In this example, there are 1200 frames (e.g., 120 fps * 10 seconds). The camera device, or any other device, might display these 1200 frames at a display rate of 30 fps. In this example, the 1200 frames are displayed over 40 seconds (e.g., 1200 frames / 30 fps). Therefore, in this example, 10 seconds of video content is displayed over 40 seconds, and the viewer perceives the video content as being in slow motion (e.g., the rate of motion of objects in the video content is slower than the rate at which objects actually move).

[0021]

[0025] There may be technical difficulties in enabling zooming in or out while slow - motion video capture is enabled. For example, in response to a request to zoom in or out, the camera processor performs digital zooming in or out on the frames captured by the current camera until a zoom threshold is reached. As an example, the current camera may be an ultra - wide camera, and in response to a zoom - in, the camera processor may perform digital zooming in on the frame capture by the current camera until a 1x zoom threshold is reached.

[0022]

[0026] Digital zooming in or out involves cropping and resizing of frames during a linear increase or decrease of the zoom factor. For example, the camera processor may receive a frame from an ultra - wide camera, and to perform a 0.6x zoom, the camera processor may crop 10% of the perimeter of the frame and resize the result of the cropping. The camera processor can perform digital zooming in or out by increasing the zoom factor over a period of time until a zoom threshold is reached (e.g., from 0.6x, 0.7x, 0.8x, 0.9x to 1x). The example is described with respect to an ultra - wide camera, but exemplary techniques for zooming in or out may be similar for other cameras.

[0023]

[0027] In some examples, digital zooming in or out causes a reduction in image quality due to resizing. For example, camera parameters may be inaccurate for frames captured while the camera processor is performing a digital zoom in or out. For example, camera parameters can be set based on a particular zoom factor (e.g., 0.5x for an ultra-wide camera, 1x for a wide camera, and 2x for a telephoto camera). During digital zooming in or out, the captured frames can be considered as frames captured with different video content compared to video content without digital zooming in or out. Camera parameters such as auto focus (AF) parameters, auto white balance (AWB) parameters, and auto exposure control (AEC) parameters determined for a particular zoom factor may be inaccurate for frames captured with different video content, where the frames captured during digital zooming in or out effectively have different video content.

[0024]

[0028] The above describes an example of digital zoom. However, for capturing using optical zoom, there is a similar problem of reduction in image quality. In optical zoom, the camera's lens and / or sensor move to provide a zoom in or out. Camera parameters can be set for the lens or sensor at a particular location, and the movement of the lens or sensor can cause the camera parameters to be inaccurate.

[0025]

[0029] After the zoom threshold for the current camera is reached, the camera processor outputs the frame captured by the subsequent camera. In this case, the frame captured by the subsequent camera is captured using the already determined camera parameters for the subsequent camera. Therefore, the image quality is relatively high for the frame captured by the subsequent camera. The exemplary process described above, which produces a frame with lower image quality, is repeated if there is another request for zooming in or zooming out using the subsequent camera.

[0026]

[0030] As explained above, in slow-motion video capture, the rate at which frames are captured is greater than the rate at which frames are displayed to produce the slow-motion effect. Also, as explained, the image quality of captured frames may be relatively low during zooming in or out. In some examples, these frames with lower image quality may be displayed for a longer period of time than the amount of time it took to capture them, so there may be undesirablely long periods of time during which low-quality frames are displayed. For example, the current camera may be capturing frames at 120fps, and the amount of time to zoom in or out to the zoom threshold may be 0.5 seconds. In this example, during zooming in or out, the current camera may capture 60 frames (e.g., 120fps * 0.5 seconds), and the image quality of these 60 frames may be worse than without zooming in or out. The display rate may be 30fps. Therefore, the 60 frames may be displayed over 2 seconds (e.g., 60 frames / 30fps). In this example, there might only be a frame of inferior quality equivalent to 0.5 seconds, but it could take 2 seconds to display that inferior quality frame, which can be undesirable for the viewer.

[0027]

[0031] According to one or more examples described herein, during normal operation with slow motion enabled (e.g., without zooming in or out), the camera may capture a first set of frames at a first rate (e.g., 120 fps or higher), and during zooming in or out with slow motion enabled, the camera may capture a second set of frames at a second rate lower than the first rate (e.g., 30 fps or lower). The image quality of the second set of frames may be inferior to that of the first set of frames.

[0028]

[0032] A camera device or any other device may display a first set of frames and a second set of frames at a third rate (e.g., 30fps). In this example, if 9.5 seconds of video content is captured by the first set of frames and 0.5 seconds of video content is captured by the second set of frames, there will be 1140 frames with relatively high-quality image content (e.g., 9.5 seconds * 120fps) and 15 frames with relatively low-quality image content (e.g., 0.5 seconds * 30fps). In that case, during display, there will be 38 seconds of playback of the frames with relatively high-quality image content (e.g., the 9.5 seconds of video content is stretched to 38 seconds to create a slow-motion effect), and 0.5 seconds of playback of the frames with relatively low-quality image content (e.g., 0.5 seconds of video content is displayed within 0.5 seconds). The amount of time spent on frames with relatively low-quality image content is reduced from 2 seconds to 0.5 seconds when the capture rate is 120fps, so the impact on the viewer experience may be minimal.

[0029]

[0033] Figure 1 is a block diagram of a device configured to implement one or more of the exemplary techniques described herein. Examples of camera devices 10 include a standalone digital camera or digital video camcorder, a mobile phone with multiple cameras, a camera-equipped wireless communication device handset such as a cellular or satellite radiophone, a camera-equipped personal digital assistant (PDA), a panel or tablet, a gaming device, a computer device, or any device having digital imaging or video capabilities, including a camera such as a so-called “webcam”.

[0030]

[0034] As shown in the example in Figure 1, the camera device 10 includes cameras 12A to 12C (each having an image sensor and a lens, for example), a camera processor 14 and its local memory 20, a central processing unit (CPU) 16, a graphical processing unit (GPU) 18, a user interface 22, a memory controller 24 providing access to system memory 30, and a display interface 26 that outputs signals for displaying graphical data on a display 28. The example in Figure 1 shows one camera processor 14, but in some examples there may be multiple camera processors (for example, one for each of the cameras 12A to 12C). The techniques described herein with respect to the camera processor 14 may include examples in which one or more camera processors implement exemplary techniques.

[0031]

[0035] Furthermore, although various components are shown as separate components, in some examples, components can be combined to form a system-on-a-chip (SoC). For example, the camera processor 14, CPU 16, GPU 18, and display interface 26 can be formed on a common integrated circuit (IC) chip. In some examples, one or more of the camera processor 14, CPU 16, GPU 18, and display interface 26 may be in separate IC chips. Various other substitutions and combinations are possible, and this technique should not be considered limited to the example shown in Figure 1.

[0032]

[0036] The various components shown in Figure 1 (whether formed on one device or on different devices) may be formed as at least one of fixed-function or programmable circuits, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuits, such as those in one or more microprocessors. Examples of local memory 20 and system memory 30 include one or more volatile or non-volatile memory or storage devices, such as random-access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM®), flash memory, magnetic data media, or optical storage media.

[0033]

[0037] The various units shown in Figure 1 communicate with each other using bus 32. Bus 32 can be any of various bus structures, such as a third-generation bus (e.g., HyperTransport bus or InfiniBand bus), a second-generation bus (e.g., Advanced Graphics Port bus, Peripheral Component Interconnection (PCI) Express bus, or Advanced Extensible Interface (AXI) bus), or another type of bus or device interconnect. The specific configuration of buses and communication interfaces between different components shown in Figure 1 is merely an example, and other configurations of camera devices and / or other image processing systems with the same or different components may be used to implement the techniques of this disclosure.

[0034]

[0038] The camera device 10 includes cameras 12A to 12C. Each of cameras 12A to 12C may be configured to capture frames having image content according to their respective camera parameters, as described below. Cameras 12A to 12C may output their captured frames to a camera processor 14 for image processing. In one or more examples, as described in more detail below, the camera processor 14 may control the rate at which cameras 12A to 12C capture frames, such as during zooming in or out, to reduce the number of frames displayed that have relatively lower quality.

[0035]

[0039] The camera processor 14 outputs the obtained frames, which have image content (for example, pixel values ​​for each image pixel), to the system memory 30 via the memory controller 24. That is, the camera processor 14 can generate video content for playback, which includes the obtained frames. In one or more examples described herein, the frames may be further processed to generate one or more frames for display. In some examples, instead of the camera processor 14 performing the blending, the GPU 18 or some other circuitry in the camera device 10 may be configured to perform the blending.

[0036]

[0040] This disclosure describes exemplary techniques as being implemented by the camera processor 14. However, the exemplary techniques should not be considered as being limited to the camera processor 14 implementing the exemplary techniques. For example, the camera processor 14 may be configured in combination with the CPU 16, GPU 18, and / or the display interface 26 to implement the exemplary techniques described herein. In some examples, each of the cameras 12A to 12C may include processing circuits. Therefore, an example of a processor capable of implementing the exemplary techniques includes processing circuits that are part of the cameras 12A to 12C. For example, one or more processors may be configured to implement the exemplary techniques described herein. An example of one or more processors includes the camera processor 14, the CPU 16, the GPU 18, the display interface 26, and one or more processing circuits from the cameras 12A to 12C, or any combination of one or more of the camera processor 14, the CPU 16, the GPU 18, the display interface 26, and one or more processing circuits from the cameras 12A to 12C.

[0037]

[0041] The CPU 16 may comprise a general-purpose processor or a dedicated processor that controls the operation of the camera device 10. The user may provide input to the camera device 10 to cause the CPU 16 to run one or more software applications. Software applications running on the CPU 16 may include, for example, a media player application, a video game application, a graphical user interface application, or another program. The user may also provide input to the camera device 10 via one or more input devices (not shown), such as a keyboard, mouse, microphone, touchpad, or another input device, which are connected to the camera device 10 via the user interface 22.

[0038]

[0042] An example of a software application is a camera application. The CPU 16 runs the camera application, and in response, the camera application causes the CPU 16 to generate content to be output by the display 28. The GPU 18 may be configured to process the content generated by the CPU 16 for rendering on the display 28. For example, the display 28 may output information such as light intensity, whether the flash is enabled, and other such information. The user of the camera device 10 may interface with the display 28 to configure the manner in which the image is generated (e.g., whether or not to use the flash, focus settings, exposure settings, and other parameters). As an example, the user of the camera device 10 may choose to take multiple frames (e.g., multiple pictures), where two or more of the multiple frames are blended together to produce one or more output frames (e.g., to reduce blur). However, taking multiple frames to be blended together may be a default option (e.g., no user selection is required). The camera application also causes the CPU 16 to instruct the camera processor 14 to capture and process frames of image content captured by the camera 12 in a user-defined manner.

[0039]

[0043] The memory controller 24 enables the transfer of data into and out of the system memory 30. For example, the memory controller 24 may receive memory read and write commands and service such commands relating to the memory 30 in order to provide memory services to the components in the camera device 10. The memory controller 24 is communicatively coupled to the system memory 30. In the example of the camera device 10 in Figure 1, the memory controller 24 is shown as a processing circuit separate from both the CPU 16 and the system memory 30, but in other examples, some or all of the functions of the memory controller 24 may be implemented on one or both of the CPU 16 and the system memory 30.

[0040]

[0044] The system memory 30 may store program modules and / or instructions and / or data accessible by the camera processor 14, CPU 16, and GPU 18. For example, the system memory 30 may store user applications (e.g., instructions for camera applications), frames obtained from the camera processor 14, and so on. The system memory 30 may further store information for use by other components of the camera device 10, and / or information generated by other components. For example, the system memory 30 may function as device memory for the camera processor 14.

[0041]

[0045] As shown in the figure, the camera device 10 includes cameras 12A to 12C. Each of cameras 12A to 12C may have a different zoom coefficient and may capture different amounts of field of view. In some examples, each of cameras 12A to 12C may have a different lens geometry and different levels of optical zoom (for example, different zoom coefficient ranges based on how much the lenses or sensors of cameras 12A to 12C can move). The field of view may be the field of view (FOV). In some examples, cameras 12A to 12C with the same zoom coefficient may capture the same FOV.

[0042]

[0046] Camera 12A may be an ultrawide camera (configured to capture a field of view of 120° to 180°, for example), camera 12B may be a wide camera (configured to capture a field of view of 60° to 120°, for example), and camera 12C may be a telephoto camera (configured to capture a field of view of 10° to 60°). Camera 12A may be configured to provide zoom coefficients from a first zoom threshold of 0.5X to a second zoom threshold of 1X. Camera 12B may be configured to provide zoom coefficients from a third zoom threshold of 1X to a fourth zoom threshold of 2X. Camera 12C may be configured to provide zoom coefficients greater than a fifth zoom threshold of 2X.

[0043]

[0047] In the example above, the second and third zoom thresholds are the same, and the fourth and fifth zoom thresholds are the same. However, this technique is not limited in this way. In some examples, there may be overlap in the zoom coefficient ranges of cameras 12A to 12C. For example, the second zoom threshold for camera 12A may be 1.3X, and the third zoom threshold for camera 12B may be 1X. In this example, both cameras 12A and 12B can provide such zoom coefficients from 1X to 1.3X zoom.

[0044]

[0048] The number of cameras 12 on the camera device 10, the field of view, and the zoom coefficient range and zoom threshold are provided only to aid understanding. The exemplary techniques should not be considered limiting to such examples. There may be more or fewer cameras 12, and the field of view may differ, and the zoom coefficient range and threshold may differ.

[0045]

[0049] The zoom coefficient range associated with each of the cameras 12A-12C does not necessarily mean that the sensors or lenses of cameras 12A-12C move to provide the desired zoom, but it is possible. That is, optical zoom is one method of providing zoom, but this technique is not limited to optical zoom. In some examples, the zoom coefficient range associated with cameras 12A-12C may indicate from which of the cameras 12A-12C the camera processor 14 should receive frames for performing digital zoom in or zoom out. Thus, the configuration of cameras 12A-12C for a particular zoom coefficient range may include examples where the sensors or lenses of cameras 12A-12C move to provide optical zoom in or zoom out, and / or examples where the camera processor 14 receives frames from them for performing digital zoom in or zoom out.

[0046]

[0050] The camera processor 14 is configured to receive image frames from cameras 12A-12C and process the image frames to generate output frames for display (for example, to generate video content for playback). The CPU 16, GPU 18, camera processor 14, or any other circuitry may be configured to process the output frames containing the image content generated by the camera processor 14 into images for display on the display 28. In some examples, the GPU 18 may be further configured to render graphics content on the display 28.

[0047]

[0051] Cameras 12A to 12C may be configured to capture frames according to their respective camera parameters, which may be determined by the camera processor 14 and / or CPU 16. For example, before the user of camera device 10 captures a frame, cameras 12A to 12C may determine camera parameters called 3A parameters (autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters).

[0048]

[0052] In autofocus, the CPU 16 can determine the area of ​​the image to focus on, and therefore the image content within the focused area is in focus. Other areas may be in focus, but not necessarily, depending on the aperture size of cameras 12A-12C. As an example, the CPU 16 may utilize a phase detection autofocus (PDAF) technique. The CPU 16 can then control cameras 12A-12C based on the area to focus on (for example, by adjusting the focal length by adjusting the lens location of cameras 12A-12C). In some examples, the CPU 16 may determine an object in the foreground and control cameras 12A-12C to focus on the foreground object. As another example, the CPU 16 may utilize an object detection technique, such as face detection, to determine the area to focus on.

[0049]

[0053] In automatic exposure control, the CPU 16 may determine the aperture and shutter speed of cameras 12A-12C based on factors such as external lighting conditions. For example, the camera processor 14 may determine information indicating light and adjust the aperture and shutter speed accordingly to maintain a constant brightness in the image (for example, keeping the brightness of a desired area of ​​the frame in the mid-range of brightness values). For example, if there is too much light (i.e., overexposure), details in the image may be lost, and if there is too little light (i.e., underexposure), the image may be too dark and details may not be visible. There are various ways in which exposure is controlled using “metering,” such as a center-weighted metering map, where brightness is kept higher at the center of the image and progressively decreases near the edges of the image. In some examples, center-weighted metering ensures that the center of the image has values ​​within the mid-range of possible brightness levels (i.e., midtones). The outer areas may be brighter, darker, or the same, depending on the actual content of the scene being captured. The techniques described herein are not limited to the example of a center-weighted photometric map (for example, not limited to a photometric map biased toward exposure control for the center of the image).

[0050]

[0054] In automatic white balance, CPU16 can determine the "color temperature" of the light source, where color temperature refers to the relative warmth or coolness of white. In some cases, the captured image may have an unrealistic color cast. In some examples, the automatic white balance algorithm analyzes the ratio between the red, green, and blue components and applies a heuristic that outputs suggested red and blue gain levels. When the R and B channels are multiplied by their respective multipliers, the image will appear more balanced, gray areas will appear gray, and there will be no unnatural color cast. However, in the case of multi-lighting scenes (e.g., with shadows), determining the multipliers can be more difficult, as different multipliers are ideal for different areas.

[0051]

[0055] In one or more examples, the camera processor 14 may store the respective camera parameters for each of the cameras 12A-12C (e.g., 3A camera parameters). Cameras 12A-12C may utilize AF parameters for adjusting the focal length and AEC parameters to cause cameras 12A-12C to operate according to the shutter and aperture size. The camera processor may utilize AWB as part of processing the frames. In some examples, each of the cameras 12A-12C may capture frames according to their respective camera parameters. However, the camera processor 14 may process frames from only one of the cameras 12A-12C, and the remaining frames may be "flushed" (e.g., not processed or stored for display). As described below, in some examples, frames from two different cameras 12A-12C may be processed, but generally, frames from only one of the cameras 12A-12C are processed by the camera processor 14.

[0052]

[0056] In some examples, the camera processor 14 may be configured as an image processing pipeline. For example, the camera processor 14 may include a camera interface that interfaces between the camera 12 and the camera processor 14. The camera processor 14 may include additional circuitry for processing image content.

[0053]

[0057] In some examples, the camera processor 14 may be configured to perform digital zoom in or digital zoom out. While the camera processor 14 has been described as performing digital zoom in or digital zoom out, in some examples, the cameras 12A to 12C may be configured to perform digital zoom in or digital zoom out. For example, each of the cameras 12A to 12C may include processing circuitry for performing digital zoom in or zoom out.

[0054]

[0058] To zoom in, the camera processor 14 may perform cropping and resizing of the frame received from one of the cameras 12A to 12C. For example, in response to a request to zoom in from 1X to 2X, the camera processor 14 may crop 50% of the outer edge of the frame received from camera 12B and resize the cropped frame. In this example, the central 50% of the frame is now resized to the full size of the frame, and the image content appears to be twice as large.

[0055]

[0059] To avoid abrupt jumps when zooming, the camera processor 14 and / or cameras 12A-12C may gradually increase or decrease the zoom coefficient when zooming from one zoom threshold to another (for example, from 1X to 2X). For example, if the zoom coefficient changes abruptly from 1X to 2X, the viewer experience may be undesirable. Therefore, the zoom coefficient may change gradually. As an example, the zoom coefficient (by camera processor 14 and / or cameras 12A-12C) may increase or decrease linearly, so the zoom coefficient may, as an example, be 1X, then 1.1X, then 1.2X, and so on up to 2X. Both the linear increase or decrease of the zoom coefficient and the step size of 0.1X are provided as examples and should not be considered limiting. Furthermore, as described above, there are at least two ways in which the zoom coefficient can be changed: (1) based on digital zoom, in which the camera processor 14 crops and resizes; and (2) based on optical zoom, in which the sensors or lenses of cameras 12A to 12C move. These two methods can be used individually or in combination.

[0056]

[0060] Again, the user of camera device 10 can zoom in or zoom out of the video content being recorded by camera device 10. In response to a request to zoom in or zoom out, camera processor 14 may process the frames captured by one of cameras 12A to 12C until the zoom coefficient reaches a zoom threshold for one of cameras 12A to 12C. Since the zoom coefficient can change gradually from one zoom threshold to another, there can be a certain amount of time that elapses for the zoom coefficient to reach a zoom threshold. For example, it may take about 0.5 seconds to reach a zoom threshold. In some examples, the range of time required to reach a zoom threshold is between about 0.2 seconds and 0.7 seconds.

[0057]

[0061] Therefore, the camera processor 14 may process frames captured by the first camera among the cameras 12A to 12C based on camera parameters for the first camera among the cameras 12A to 12C. In response to a request to zoom in or zoom out, the camera processor 14 may process frames captured by the first camera during zoom in or zoom out to a zoom threshold for the first camera among the cameras 12A to 12C. For example, during zoom in or zoom out to a zoom threshold (e.g., 0.5 seconds) for the first camera, the first camera captures frames based on initially determined camera parameters. For example, the zoom coefficient for the first camera among the cameras 12A to 12C may initially be 1X, and camera parameters (e.g., 3A parameters for AF, AWB, and AEC) may be determined when the zoom coefficient for the first camera is 1X. The zoom coefficient for the second camera among cameras 12A to 12C can initially be 2X, and the camera parameters (e.g., 3A parameters) may be determined when the zoom coefficient for the second camera is 2X.

[0058]

[0062] In that case, during the time (e.g., 0.5 seconds) that the zoom coefficient changes from 1X to a 2X zoom threshold, the first camera may continue to capture frames based on the camera parameters determined at a 1X zoom coefficient. That is, the zoom coefficient may change (e.g., 1.1X, 1.2X, 1.3X, and so on up to 2X), but the first camera may continue to capture frames based on the camera parameters determined at a 1X zoom coefficient. After reaching a 2X zoom coefficient, the camera processor 14 may begin processing the frames captured by the second camera among the cameras 12A to 12C using the camera parameters for the second camera among the cameras 12A to 12C, which were determined based on the 2X zoom coefficient for the second camera among the cameras 12A to 12C.

[0059]

[0063] In the example above, during the time it takes for the zoom coefficient to change from a 1X zoom threshold to a 2X zoom threshold (e.g., 0.5 seconds), the first camera may continue to capture frames based on the camera parameters determined at a 1X zoom coefficient. Generally, the camera parameters for each of cameras 12A to 12C may be determined based on each of cameras 12A to 12C being at its respective first zoom threshold. During the time it takes for the zoom coefficient to change from the first zoom threshold to the second zoom threshold for each of cameras 12A to 12C, each of cameras 12A to 12C may continue to capture frames based on the camera parameters determined at the first threshold.

[0060]

[0064] In some cases, the quality of captured frames may be lower during the time it takes for the zoom coefficient to change from a first zoom threshold to a second zoom threshold than the quality of captured frames when the zoom coefficient is not changing. For example, since the camera parameters for the first camera among cameras 12A-12C are based on a first zoom threshold for the first camera, the camera parameters may not be precisely accurate for the first camera if the zoom coefficient is not the first zoom threshold. As an example, the camera parameters for the first camera may be set based on a zoom coefficient of 1X. The camera parameters may not be accurate for a zoom coefficient of 1.2X. Therefore, a frame captured at a zoom coefficient of 1.2X using camera parameters determined for a zoom coefficient of 1X may be of lower quality compared to a frame captured at a zoom coefficient of 1X.

[0061]

[0065] For example, whenever there is zoom, the field of view (FOV) may remain stable until the zoom ends (for example, until the zoom threshold is reached). Camera parameters during zooming are not reset or recalculated, but the FOV may change from zooming in or out. Therefore, the camera parameters used for capture and processing while zooming are the same parameters that were already set before zooming began. This results in the zooming FOV not having precise camera parameters. For example, if the object to be focused (e.g., AF) is somewhere other than the central FOV, zooming will result in an off-focus preview because the object to be focused on is no longer within the FOV. The same applies to exposure (e.g., AEC) and white balance (AWB). For example, if the zoom is from a well-lit scene to a shadowed scene, the zooming FOV will remain poorly lit because it uses the exposure parameters per zoom (i.e., pre-zoom AEC). Therefore, images captured during zooming in or out tend to be of lower quality than when not zooming in or out.

[0062]

[0066] The quality of frames captured during the time the zoom coefficient changes from the first zoom threshold to the second zoom threshold may be relatively low, but in most cases, this relatively low quality of the frames does not affect the viewer experience. For example, each of cameras 12A to 12C may be configured to capture frames at a specific frame rate (for example, the capture rate is 30 frames per second (fps)). In some cases, camera device 10 displays frames at a display rate that is the same as the capture rate (for example, the display rate is 30fps and the capture rate is 30fps).

[0063]

[0067] As explained above, it can take 0.5 seconds to zoom from one zoom threshold to another. Therefore, during this 0.5-second period, assuming a capture rate of 30fps, there may be approximately 15 frames captured at a relatively low quality (e.g., 30fps * 0.5 seconds = 15 frames). If the display rate is also 30fps, it will take 0.5 seconds to display these frames of relatively low quality. Viewers may not perceive, or may not be affected by, the display of 0.5 seconds worth of inferior quality frames.

[0064]

[0068] However, in some cases, having frames of lower quality can affect the viewer experience. For example, a user might configure camera device 10 to capture frames for slow motion. In slow motion, the frame capture rate is higher than the frame display rate. Therefore, it takes more time to display a frame than it took to capture it. This results in the viewer perceiving slow motion because the image content appears to be moving more slowly.

[0065]

[0069] As an example, let's assume the capture rate is 240fps and the display rate is 30fps. In this example, one second of video capture will produce 240 frames. At a display rate of 30fps, it will take 8 seconds (for example, 240 frames / 30fps) to display that one second of video. In this example, from the viewer's point of view, objects in the captured frames will appear to be moving at 1 / 8 the rate at which the objects are actually moving, thus creating the perception of slow motion.

[0066]

[0070] In some examples, the user of camera device 10 may start video capture in normal mode. The user may then cause camera device 10 to start capturing in slow motion. After a while, the user may cause camera device 10 to return to capturing in normal mode. For example, during normal mode, cameras 12A-12C may capture frames at 30fps. During slow-motion mode, cameras 12A-12C may capture frames at 120fps or higher.

[0067]

[0071] For a better visual experience, the user can zoom in or out during slow motion. For example, the first camera among cameras 12A-12C may be set to a zoom factor of 0.5X, where the zoom threshold for the first camera is 0.5X-1X. Initially, the first camera may capture at 30fps with a zoom factor of 0.5X. The user can then cause the camera device 10 to capture in slow motion mode. In this example, the CPU 16 and / or camera processor 14 may cause the first camera to capture at 240fps. While the first camera is capturing frames, the user can zoom in, and thus the zoom factor increases from 0.5X to 1X (for example, over about 0.5 seconds). The camera processor 14 can then begin processing the frames captured by the second camera among cameras 12A-12C, which has a zoom factor range of 1X-2X. Since slow motion is still enabled, the CPU 16 and / or camera processor 14 may cause the second camera to capture at 240fps with a zoom factor of 1X. At some point later, the user may cause the camera device 10 to end slow motion, and the camera processor 14 may process the frames captured by the second camera at 30fps.

[0068]

[0072] In this way, there is a handoff and switching between the first and second cameras to enable slow-motion video capture using zoom in or zoom out. For example, camera processor 14 processes frames from the first camera among cameras 12A-12C until a zoom threshold for the first camera is reached, and then camera processor 14 processes frames from the second camera among cameras 12A-12C. The capture rates of cameras 12A-12C can be based on whether slow motion is enabled or not (e.g., a higher capture rate for slow motion and a lower capture rate for normal capture).

[0069]

[0073] However, there can be some issues with enabling zoom in or zoom out with slow motion. As explained above, frames captured during zoom in or zoom out tend to be of lower quality because the camera parameters are not precise with respect to the zoom factor, but the amount of time required to view these lower-quality frames is not long (e.g., 0.5 seconds), so the impact is low. With slow-motion video capture enabled, the amount of time required to view these lower-quality frames during zoom in or zoom out is relatively long.

[0070]

[0074] For example, if the capture rate is 240fps, the number of frames captured in the 0.5 seconds it takes to move from the first zoom threshold to the second zoom threshold of one of the cameras 12A-12C is 120 frames (e.g., 240fps * 0.5 seconds). If the display rate is 30fps, it takes 4 seconds to display 120 frames (e.g., 120 frames / 30fps). Therefore, in this example, there are 4 seconds during which frames of relatively lower quality are displayed. As explained above, in normal mode (e.g., not slow motion), there are 0.5 seconds during which frames of relatively lower quality are displayed. Therefore, in slow motion with zooming in or out, if the capture rate is 240fps, the amount of time during which frames of lower quality are displayed can be eight times longer than in normal mode with zooming in or out. At higher capture rates, the amount of time that frames with inferior quality are displayed can be much greater (for example, 960fps / 30fps * 0.5 seconds equals 16 seconds, so 960fps will result in 16 seconds of inferior quality).

[0071]

[0075] This disclosure describes exemplary techniques for enabling zooming in or zooming out while slow motion is enabled. For example, while zooming in or zooming out to a zoom threshold of one of the cameras 12A to 12C while slow motion is enabled, the camera processor 14 may receive frames at a lower rate than at the rate at which it receives frames while slow motion is enabled but without zooming in or out.

[0072]

[0076] For example, the camera processor 14 may receive a first set of frames captured by the first camera among cameras 12A to 12C at a first rate (for example, the first rate is 120fps or higher) with slow-motion video capture enabled. The camera processor 14 may process the first set of frames to generate video content for display. The CPU 16 and / or the camera processor 14 may receive a choice to zoom in or zoom out. While zooming in or out to a zoom threshold for the first camera, the camera processor 14 may receive a second set of frames captured by the first camera at a second rate lower than the first rate (for example, the second rate is 30fps). The camera processor 14 may process the second set of frames to generate video content for display. For example, the camera processor 14 may generate video content for playback at a third rate lower than the first rate (for example, 30fps), including the first set of frames and the second set of frames. For example, the generation of video content for playback by the camera processor 14 may refer to an example in which the camera processor 14 provides a first set of frames and a second set of frames so that the display interface 26 can output them for display.

[0073]

[0077] In this example, the display interface 26 may output generated video content for display, which includes a first set of frames and a second set of frames at a third rate lower than the first rate (for example, the third rate is 30 fps). In some examples, the display interface 26 may be configured to generate video content for playback. For example, to generate video content for playback, the display interface 26 may receive a first set of frames and a second set of frames, output the first set of frames, and then output the second set of frames in a manner that allows the viewer to perceive video playback of the frames. Generally, when the camera processor 14, the display interface 26, or any other component generates video content for playback, it may mean that the camera processor 14, the display interface 26, or any other component configures, receives, or processes a set of frames (for example, a first and second set of frames) in a manner that allows the set of frames to be output for display.

[0074]

[0078] In an exemplary example, in normal mode, the camera processor 14 may receive frames at a rate of 30fps from camera 12A, which is currently at a zoom factor of 0.5X. Thus, camera 12A is configured to capture frames at 30fps. In this example, we assume that camera 12A captures 8 seconds of frames at a capture rate of 30fps, and therefore captures 240 frames.

[0075]

[0079] Next, the user may enable slow motion. With slow motion enabled, the camera processor 14 may receive frames at a rate of 120fps from camera 12A, which is currently at a zoom factor of 0.5X. In this example, camera 12A is configured to capture frames at 120fps when slow motion is enabled. Assume that camera 12A captures 4 seconds of frames at a capture rate of 120fps, and therefore captures 480 frames.

[0076]

[0080] The user can then enable zoom-in (or, in the case of object or face tracking, there may be auto-zoom), so the zoom coefficient changes from 0.5X to 1X (for example, the zoom threshold for camera 12A). Instead of capturing frames at 120fps during the 0.5 seconds it takes to go from a zoom coefficient of 0.5X to 1X (for example, due to a change in lens / sensor position or progressive zoom), camera 12A may capture frames at 30fps to reduce the number of lower-quality frames captured during the time it takes to go from a zoom coefficient of 0.5X to 1X, and camera processor 14 may receive frames at 30fps. As described above, frames captured during the 0.5 seconds it takes for the zoom coefficient to reach the zoom threshold tend to be of lower quality. Thus, there will be 15 frames of lower quality.

[0077]

[0081] After reaching a zoom factor of 1X, the camera processor 14 can receive frames from camera 12B. Since slow motion is still enabled and there is no further zooming in, the camera processor 14 can receive frames from camera 12B at a rate of 120fps. Assume there is a 4-second frame with a capture rate of 120fps, and therefore camera 12B has captured 480 frames.

[0078]

[0082] Next, we assume the user turns off slow motion. We also assume there are another 10 seconds of frames captured by camera 12B at 30fps, resulting in 300 frames.

[0079]

[0083] In this example, if the display rate is 30fps, the amount of time required to display the frames is as follows: First, there are 8 seconds of capture at a 30fps capture rate, which generates 240 frames. Displaying 240 frames takes 8 seconds at 30fps. Next, there are 4 seconds of frames at a 120fps capture rate, which generates 480 frames. Displaying 480 frames takes 16 seconds at 30fps.

[0080]

[0084] Next, zooming in is enabled, and here, a 0.5-second frame is captured at 30fps, resulting in 15 frames. It takes 0.5 seconds at 30fps to display these 15 frames. As explained above, the quality of these 15 frames may be relatively poor due to the imprecision of camera parameters during zooming in. By reducing the capture rate to 30fps, there is only a 0.5-second period of time to display the inferior quality frames, compared to 2 seconds if the capture rate were maintained at 120fps.

[0081]

[0085] Next, there are 4 seconds of frames captured at 120fps, resulting in 480 frames. It takes 16 seconds to display these 480 frames at 30fps. Then, slow motion is turned off, and 10 seconds of frames are captured at 30fps, resulting in 300 frames. It takes 10 seconds to display these 300 frames at 30fps.

[0082]

[0086] In the example above, each of cameras 12A through 12C is configured to capture frames at 120fps during slow motion or at 30fps during normal video. However, in some examples, the capture rates for two or more of cameras 12A through 12C may differ. For example, during slow motion, camera 12A may capture at 960fps, camera 12B at 480fps, and camera 12C at 120fps. The capture rates for each of cameras 12A through 12C may be user-configurable options.

[0083]

[0087] The camera processor 14 may perform some initial image processing on multiple frames, but such initial image processing is not necessary in all examples. The camera processor 14 may output multiple frames to system memory 30 for storage. In some examples, instead of outputting multiple frames to system memory 30, or in addition to that, the camera processor 14 may output multiple frames to local memory 20. In another example, the camera processor 14 may store each of the multiple frames in local memory 20 for temporary storage when each frame is captured, and then move the multiple frames from local memory 20 to system memory 30.

[0084]

[0088] Figure 2 is a block diagram showing the camera processor of the device in Figure 1 in more detail. As shown, cameras 12A to 12C can each capture frames that are output to the camera processor 14. As described above, each of cameras 12A to 12C can be associated with different field of view angles and different ranges for zoom coefficients between their respective zoom thresholds. In this way, there are parallel streams output to the camera processor 14.

[0085]

[0089] In one or more examples, frame capture may be preceded by, or possibly the first few captured frames, which are used to determine camera parameters. For example, the camera processor 14 and / or CPU 16 may determine, for each of the cameras 12A-12C, autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters, referred to as 3A parameters. As shown in the figure, camera parameter 34A may be a camera parameter for camera 12A, camera parameter 34B may be a camera parameter for camera 12B, and 3A parameter 3A may be a camera parameter for camera 12C. For ease of explanation, camera parameters 34A-34C are stored locally in cameras 12A-12C, respectively. However, camera parameters 34A-34C may be stored elsewhere, such as in local memory 20 and / or system memory 30, as two examples.

[0086]

[0090] In some cases, once camera parameters 34A-34C are determined, they may remain stable for the remainder of the recording. In some cases, camera parameters 34A-34C may not be recalculated because such recalculation would be time-consuming and could result in delays or inferior quality. The camera processor 14 and / or CPU 16 may determine the camera parameters 34A-34C individually (e.g., different parameters for each), or the camera processor 14 and / or CPU 16 may determine one of the camera parameters 34A-34C and apply the determined one to the others (e.g., the same values ​​for focus blur distance, lumens index, and color temperature for camera parameters 34A-34C).

[0087]

[0091] However, in some cases, it may be possible to recalculate camera parameters 34A to 34C for cameras 12A to 12C whose frames have not been processed. For example, if frames from camera 12A have been processed, frames captured by cameras 12B and 12C can be used to update camera parameters 34B and 34C, respectively. Such calculations may allow for compensation of field of view (FOV) changes when switching between cameras 12A to 12C.

[0088]

[0092] In Figure 2, the camera processor 14 includes a switch 36 configured to selectively select a stream from one of the cameras 12A-12C. In some cases, as described below, the switch 36 may be configured to select two of the three streams. However, generally, the switch 36 can select one of the streams from cameras 12A-12C. The unselected stream can be bypassed from all processing and go directly to the sink / flash.

[0089]

[0093] As shown in the figure, the camera processor 14 includes a rate controller 38. The rate controller 38 may be configured to select the rate at which the camera processor 14's circuitry receives frames. For example, the rate controller 38 may be configured to set the capture rate at which cameras 12A-12C capture frames. For example, the rate controller 38 may cause cameras 12A-12C to capture frames at 30fps, 120fps, 240fps, 960fps, etc. In such an example, the sensors of cameras 12A-12C may detect light based on the capture rate (for example, 30 times per second at 30fps) and output a current indicating the light and its color. The rate controller 38 may set the rates of cameras 12A-12C to be the same or different.

[0090]

[0094] There may be other ways in which the rate controller 38 can select the rate at which the camera processor 14's circuitry receives frames. For example, cameras 12A to 12C may each capture frames at a relatively high rate (e.g., 960fps), and the rate controller 38 may be configured to drop frames to achieve the desired rate. For example, the rate controller may drop every frame, expecting every eight frames, resulting in an effective rate of 120fps (e.g., 960fps / 8=120fps).

[0091]

[0095] In this disclosure, the camera processor 14 receiving frames at a specific capture rate refers to an example where the capture rate of cameras 12A-12C is set to a specific rate, and an example where frames captured by cameras 12A-12C are dropped in order to effectively achieve the specific capture rate. There may be other ways in which the camera processor 14 can receive frames at a specific rate (e.g., the capture rate), and the techniques described in this disclosure are not limited to set techniques.

[0092]

[0096] The camera processor 14 includes a digital zoom circuit 40A, a processing circuit 42A, and a buffer 44A. The camera processor 14 may optionally include a digital zoom circuit 40B, a processing circuit 42B, and a buffer 44B. An exemplary use of the digital zoom circuit 40B, the processing circuit 42B, and the buffer 44B will be described in more detail below. For ease of understanding, the exemplary techniques will be described in relation to the digital zoom circuit 40A, the processing circuit 42A, and the buffer 44A.

[0093]

[0097] The digital zoom circuit 40A or 40B may be part of the camera processor 14, or in addition to it, part of the cameras 12A-12C. Furthermore, the digital zoom circuit 40A and / or 40B are not necessarily required in all examples. For example, in some examples, optical zoom may suffice. In some examples, zooming in or out may be based on both optical zoom and digital zoom provided by the digital zoom circuit 40A and / or 40B. To implement optical zoom, the camera processor 14 may include circuitry that sets the sensor and / or lens locations of the cameras 12A-12C to achieve a desired zoom factor.

[0094]

[0098] The digital zoom circuit 40A receives a frame from one of the selected cameras 12A to 12C, and if zoom is not enabled, it may pass the frame to the processing circuit 42A. The processing circuit 42A may be configured to perform any processing, such as filtering or any other processing required to prepare the output frame to be output. Examples of processing include, among others, spatial filtering (e.g., filtering across image content within the same frame) or edge enhancement. The buffer 44A may be configured to store those frames until they are output to the system memory 30. The display interface 26 may then output the frames for display. Thus, the camera processor 14, the buffer 44A, and the display interface 26 can be considered to generate video content for playback, including the frames that are then output for display.

[0095]

[0099] In one or more examples, as part of capturing video content, the user may enable slow-motion mode (e.g., video capture with slow-motion enabled). Processing circuit 42A may receive a first set of frames captured by the first camera 12A at a first rate. The first rate may be set by the rate controller 38 and may be relatively high (e.g., at least 120 fps). Processing circuit 42A may process the first set of frames to generate video content for display (e.g., store the frames in buffer 44A). The first camera 12A may capture a first set of frames based on camera parameters 34A.

[0096]

[0100] The CPU 16 may receive a selection to zoom in or zoom out. For example, the user may utilize the display 28 to provide input indicating that the user wishes to zoom in or zoom out on the captured image content. The CPU 16 may then output information indicating the selection to zoom in or zoom out to the camera processor 14 and / or cameras 12A-12C.

[0097]

[0101] In response to a selection to zoom in or zoom out, the camera processor 14 may enable the implementation of zooming in or zooming out. For example, since the camera processor 14 has received a frame from the first camera 12A, the camera processor 14 may cause the position of the lens and / or sensor of the camera 12A to move in order to change the focus and provide zooming in or zooming out. As another example, a digital zoom circuit 40A may perform a digital zoom on the frame captured by the camera 12A.

[0098]

[0102] During zooming in or out of the first camera 12A up to a zoom threshold, the processing circuit 42A may receive a second set of frames captured by the first camera 12A at a second rate lower than the first rate. For example, the rate controller 38 may cause the first camera 12A to capture frames at a second rate (e.g., 30 fps). Alternatively, the rate controller 38 may drop frames from the first camera 12A to achieve a valid second rate.

[0099]

[0103] A second set of frames is captured during zoom-in or zoom-out, and since camera parameter 34A is determined without zooming in or out, the image quality of the second set of frames is inferior to that of the first set of frames. The number of frames with inferior quality can be minimized by reducing the rate at which the second set of frames is captured (for example, by setting the capture rate to achieve a valid capture rate or by dropping frames).

[0100]

[0104] The camera processor 14 may process a second set of frames to generate video content for display. For example, the digital zoom circuit 40A may perform digital zoom in or digital zoom out until a zoom threshold for the first camera 12A is reached (for example, from a 0.5X to a 1X zoom factor). For example, performing digital zoom in or digital zoom out until a zoom threshold for the first camera 12A is reached includes the digital zoom circuit 40A performing cropping and resizing during the increase or decrease of the zoom factor for the second set of captured frames.

[0101]

[0105] Buffer 44A may store a first set of frames and a second set of frames. System memory 30 may receive a first set of frames and a second set of frames. Display interface 26 may retrieve a first set of frames and a second set of frames and output them for display on display 28 at a third rate (e.g., 30fps) lower than the first rate. For example, the camera processor 14, buffer 44A, and display interface 26 may be considered, either individually or in combination, to generate video content for playback, including a first set of frames and a second set of frames at a third rate (e.g., 30fps) lower than the first rate.

[0102]

[0106] In the example above, the selection to zoom in or zoom out is received while slow-motion video capture is enabled. The camera processor 14 may also receive a second set of frames while slow-motion video capture is enabled.

[0103]

[0107] After the zoom threshold for the first camera 12A is reached, the camera processor 14 may receive frames from another of the cameras 12. Switching from one camera to another is called a hard switch. For example, switch 36 may switch from the first camera 12A to the second camera 12B. In this example, the camera processor 14 may receive a third set of frames captured by the second camera 12B of the multiple cameras 12 at a fourth rate that is higher than the second rate. For example, the fourth rate may be 120fps or higher, and higher than the second rate which may be 30fps. As above, the rate controller 38 may set the capture rate for the second camera 12B or drop frames to achieve that capture rate. The camera processor 14 may process the third frames (for example, via processing circuit 42A, since there may be no zoom) to generate video content for display. The display interface 26 may output the third set of frames at the third rate for display on the display 28. For example, the camera processor 14 and the display interface 26 can be considered to generate video content for playback at a third rate, including a first set of frames, a second set of frames, and a third set of frames. In this example, since slow-motion video capture is still enabled, the fourth rate at which the camera processor 14 receives and processes frames is greater than the display rate of the third rate.

[0104]

[0108] The second camera 12B may capture frames based on camera parameters 34B. For example, while the first camera 12A is capturing a first set of frames, or before that, the CPU 16 and / or camera processor 14 may determine one or more of the AF parameters, AWB parameters, and AEC parameters for each of the multiple cameras 12A-12C (for example, determining camera parameters 34A-34C). In such an example, receiving a third set of frames captured by the second camera 12B involves receiving the third set of frames based on one or more of the AF parameters, AWB parameters, and AEC parameters (for example, camera parameters 34B) determined for the second camera 12B while the first camera 12A is capturing a first set of frames, or before that. For example, as described above, camera parameters 34A-34C may be stable or may not be redetermined during frame capture. Therefore, although the 3A parameter 34B was determined when the first set of frames was captured by camera 12A, or before, the camera parameter 34B can still be accurate for camera 12B.

[0105]

[0109] As described above, in some examples, the camera processor 14 may optionally include a digital zoom circuit 40B, a processing circuit 42B, and a buffer 44B. One or more of the digital zoom circuit 40B, processing circuit 42B, and buffer 44B (for example, the processing circuit 42B may be present, but the digital zoom circuit 40B may not). The digital zoom circuit 40B, processing circuit 42B, and buffer 44B may be used in examples where there is overlap in the zoom coefficient ranges for different cameras 12A to 12C. For example, in some examples, the zoom thresholds for camera 12A may be 0.5X and 1X, and the zoom thresholds for camera 12B may be 1X to 2X. However, in some examples, the zoom thresholds for camera 12A may be 0.5X to 1.3X, and the zoom thresholds for camera 12B may be 1X to 2X. In this example, there is overlap in the zoom coefficient ranges from 1X to 1.3X for cameras 12A and 12B.

[0106]

[0110] The use of the digital zoom circuit 40B, processing circuit 42B, and buffer 44B when there is overlap in the zoom coefficient range is an example and should not be considered limiting. The digital zoom circuit 40B, processing circuit 42B, and buffer 44B can still be used even in examples where there is no overlap in the zoom coefficient range.

[0107]

[0111] In such an example, if processing circuit 42A is processing frames from camera 12A in response to zooming in or out, processing circuit 42A may continue processing frames from camera 12A until the zoom threshold for camera 12A is reached. In parallel, processing circuit 42B may begin processing frames from camera 12B. For example, switch 36 may be configured to route the output from camera 12A to processing circuit 42A (e.g., via digital zoom circuit 40A) and the output from camera 12B to processing circuit 42B (e.g., via digital zoom circuit 40B). Processing circuit 42B may output to buffer 44B.

[0108]

[0112] The camera processor 14 may also include a switch 46. The switch 46 may be configured to selectively output from either buffer 44A or buffer 44B. In some examples, as the zoom coefficient for camera 12A passes through 1X and moves towards 1.3X, the switch 46 may be configured to output from buffer 44A until the zoom coefficient reaches 1.3X. In parallel, after the zoom coefficient has passed 1X, the processing circuit 42B may be configured to output frames to buffer 44B. In this case, frames in buffer 44B may be overwritten when buffer 44B is full, from the zoom coefficient from 1X to 1.3X. However, after the zoom coefficient reaches 1.3X, the switch 46 may switch to selectively output from buffer 44B.

[0109]

[0113] Having both buffer 44A and buffer 44B can be beneficial in avoiding gaps in the frames to be displayed when switching from camera 12A to camera 12B. For example, frames ready to be displayed may already be in buffer 44B. Otherwise, there may be a momentary blank frame during the switchover from camera 12A to camera 12B. By pre-filling buffer 44B with frames, there may be no blank frames.

[0110]

[0114] As described above, with slow-motion video capture enabled, the camera processor 14 may be configured to receive a first set of frames captured by the first camera 12 at a first rate, and a second set of frames captured by the first camera 12A at a second rate smaller than the first rate during zoom-in or zoom-out to a zoom threshold for the first camera. In some examples, in parallel with processing the second set of frames, the camera processor 14 may receive a third set of frames captured by the second camera 12B among the multiple cameras 12A-12C. For example, while camera 12A is zoomed in from 0.5X to 1.3X, the camera processor 14 may in parallel receive a third set of frames from camera 12B, such as when the zoom coefficient is between 1X and 1.3X. The camera processor 14 may process the third set of frames.

[0111]

[0115] Switch 46 may be configured to select buffer 44A to output a second set of frames for display until a second zoom threshold (e.g., 1.3X) for the first camera 12A is reached. Switch 46 may then be configured to select buffer 44B to output a third set of frames for display after the second zoom threshold (e.g., 1.3X) for the first camera 12A has been reached. Thus, generating video content for playback may include generating video content for playback that, at a third rate, includes a first set of frames, a second set of frames until the second zoom threshold for the first camera is reached, and a third set of frames after the second zoom threshold for the first camera has been reached. In this example, both the first camera 12A and the second camera 12B are configured to operate at zoom coefficients greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold. For example, both camera 12A and camera 12B are configured to operate at zoom coefficients greater than or equal to 1X and less than or equal to 1.3X.

[0112]

[0116] Figure 3 is a conceptual diagram illustrating an exemplary method for enabling zoom in or zoom out. In the example shown in Figure 3, display 28 may display three icons 48A to 48C, each associated with a different zoom threshold for one of the cameras 12A to 12C. For example, icon 48A may be associated with 0.5X for camera 12A, icon 48B may be associated with 1X for camera 12B, and icon 48C may be associated with 2X for camera 12C. Thus, in some examples, the camera processor 14 and / or CPU 16 may receive information indicating a choice to zoom in or zoom out as a way of receiving a choice to zoom in or zoom out (for example, when slow-motion video capture is enabled).

[0113]

[0117] However, there may be other ways to receive a choice to zoom in or zoom out, such as by receiving information indicating a pinch-to-zoom motion. In a pinch-to-zoom motion, the user may place two fingers on the display 28 and pinch the two fingers together to zoom out, or spread the two fingers apart to zoom in.

[0114]

[0118] In either case (for example, selection or pinch zoom), the camera processor 14 may cause the zoom coefficient of one of the cameras 12A-12C, from which the camera processor 14 is receiving a frame, to increase or decrease until a zoom threshold is reached for one of the cameras 12A-12C. During the zooming in or out to the zoom threshold (for example, about 0.5 seconds, which is the time it takes for the zoom coefficient to reach the zoom threshold), the camera processor 14 may receive and process frames at a lower rate than the rate at which the camera processor 14 can receive and process frames when the user is not zooming in or out.

[0115]

[0119] Figure 4 is a timing diagram illustrating an example of zooming in or zooming out. In Figure 4, before time T0, the camera processor 14 may be configured to receive frames from the camera 12A at 30fps, since zooming is not enabled.

[0116]

[0120] At time T0, the user may enable slow motion (for example, via input through the display 28). From time T0, the camera processor 14 may receive a first set of frame captures by the first camera 12A at a first rate. The first rate may be relatively high, such as 120fps or higher. The processing circuit 42A may be configured to process the first set of frames to generate video content for display. The processing circuit 42A may store the first set of frames in a buffer 44A.

[0117]

[0121] At T1, the CPU 16 and / or camera processor 14 may receive a selection to zoom in or zoom out. For example, receiving a selection to zoom in or zoom out may include receiving information indicating a pinch zoom motion (for example, via the selection of one of icons 48A to 48C) or receiving information indicating a selection to zoom.

[0118]

[0122] From time T1 to T2, the zoom coefficient for camera 12A can change. For example, it may take approximately 0.5 seconds to zoom from 0.5X to 1X. During these 0.5 seconds, the camera processor 14 may receive frames captured by camera 12A. For example, during zooming in or out to a zoom threshold for the first camera 12A, the camera processor 14 may receive a second set of frames captured by the first camera 12A at a second rate smaller than the first rate. Since the second set of frames is captured using camera parameters 34A selected for a zoom coefficient of 0.5X, the image quality of the second set of frames may be inferior to that of the first set of frames. As an example, the second rate may be 30fps.

[0119]

[0123] The processing circuit 42A processes a second set of frames to generate video content for display and may store the second set of frames in buffer 44A (for example, after processing). The camera processor 14 may cause buffer 44A to output the first set of frames and the second set of frames to system memory 30. The display interface 26 receives the first set of frames and the second set of frames from system memory 30 and may output the first set of frames and the second set of frames at a third rate (for example, 30fps) lower than the first rate for display on display 28.

[0120]

[0124] At time T2, the zoom threshold for camera 12A is reached, and the camera processor 14 processes frames from the second camera 12B. For example, switch 36 switches to routing the output from the second camera 12B to the processing circuit 42A (for example, via the digital zoom circuit 40A if necessary). For example, after the zoom threshold for the first camera 12A is reached, the camera processor 14 may be configured to receive a third set of frames captured by the second camera 12B of the multiple cameras 12A-12C at a fourth rate (for example, 120fps or higher) that is greater than the second rate (for example, the second rate is 30fps). The processing circuit 42A processes the third set of frames to generate video content for display and may store the third set of frames in buffer 44A. Buffer 44A may output the third set of frames to system memory 30. The display interface 26 may take a third set of frames and output the third set of frames for display at a third rate (e.g., 30fps).

[0121]

[0125] At time T3, the user may disable video capture with slow motion. In this case, the camera processor 14 may continue to receive frames from the second camera 12B. However, the rate at which the camera processor 14 receives frames from the second camera 12B may be relatively low (e.g., 30fps) because slow motion has been disabled. In this case as well, with slow motion, frames are captured at a high rate and displayed at a low rate. When slow motion is disabled, the rate at which frames are captured may be approximately the same as the rate at which frames are displayed.

[0122]

[0126] Figure 5 is a flowchart illustrating an exemplary method of operation by one or more examples described in this disclosure. The example in Figure 5 describes one or more processors and a first camera, wherein the example of one or more processors includes one or more of a camera processor 14, a CPU 16, and a display interface 26, and the example of the first camera includes one of cameras 12A to 12C.

[0123]

[0127] One or more processors may receive a first set of frames captured by a first camera at a first rate (50). For example, slow-motion video capture may be enabled. In such a case, the first rate may be relatively high, such as 120 fps or higher.

[0124]

[0128] One or more processors may process a first set of frames to generate video for display (52). For example, a processing circuit 42A may be configured to process a first set of frames to generate video content and to store the resulting first set of frames in a buffer 44A.

[0125]

[0129] At some point in time, one or more processors may receive a choice to zoom in or zoom out (54). For example, receiving a choice to zoom in or zoom out may include receiving a choice to zoom in or zoom out while slow-motion video capture is enabled.

[0126]

[0130] During the zoom-in or zoom-out of the first camera to a zoom threshold, one or more processors may receive a second set of frames captured by the first camera at a second rate smaller than the first rate (56). In some examples, receiving a second set of frames includes receiving a second set of frames with slow-motion video capture enabled. As described above, it may take some time to increase or decrease the zoom coefficient for the first camera to reach a zoom threshold for the first camera. During the time the zoom coefficient is increasing or decreasing to the zoom threshold, the captured frames (e.g., a second set of frames) tend to have lower image quality because the camera parameters are not accurate for those zoom coefficients. In some examples, the image quality of the second set of frames is lower than that of the first set of frames.

[0127]

[0131] One or more processors may process a second set of frames to generate video content for display (58). Processing a second set of frames may include performing digital zoom in or digital zoom out until a zoom threshold for the first camera is reached. For example, performing digital zoom in or digital zoom out until a zoom threshold for the first camera is reached may include performing cropping and resizing on the second set of frames while increasing or decreasing the zoom factor.

[0128]

[0132] Buffer 44A can store a first set of frames and a second set of frames. In some examples, one or more processors can generate video content for playback, including the first set of frames and the second set of frames, at a third rate lower than the first rate (60). One or more processors (for example, display interface 26) can output the generated video content, including the first set of frames and the second set of frames, at a third rate lower than the first rate, for display on display 28.

[0129]

[0133] The first camera is one of a plurality of cameras coupled to one or more processors (for example, one of cameras 12A to 12C). After a zoom threshold for the first camera is reached, one or more processors may be configured to receive a third set of frames captured by the second camera of the plurality of cameras at a fourth rate greater than the second rate, process the third set of frames to generate video content for display, and output the third set of frames at the third rate for display. For example, one or more processors may generate video content for playback at the third rate, including the first set of frames, the second set of frames, and the third set of frames, and then output the generated video content for playback for display.

[0130]

[0134] As described above, multiple cameras may capture frames based on their respective camera parameters. For example, while the first camera captures a first set of frames, or before that, one or more processors may be configured to determine one or more of the AF parameters, AWB parameters, and AEC parameters (e.g., camera parameters 34A-34C) for each of the multiple cameras. In such an example, receiving a third set of frames captured by the second camera may involve receiving the third set of frames based on one or more of the AF parameters, AWB parameters, and AEC parameters determined for the second camera while the first camera captures a first set of frames, or before that.

[0131]

[0135] In some examples, the zoom threshold is a first zoom threshold. After the first zoom threshold for the first camera is reached, one or more processors may be configured to receive a third set of frames captured by the second camera of the multiple cameras in parallel with processing a second set of frames (for example, switch 36 routes frames from the first camera to processing circuit 42A and frames from the second camera to processing circuit 42B), and to process the third set of frames (for example, using processing circuit 42B).

[0132]

[0136] One or more processors may be configured to generate video content for playback at a third rate, including a first set of frames, a second set of frames until a second zoom threshold for the first camera is reached, and a third set of frames after the second zoom threshold for the first camera is reached. For example, one or more processors may output the second set of frames for display until the second zoom threshold for the first camera is reached, and then output the third set of frames for display after the second zoom threshold for the first camera is reached. In this example, both the first and second cameras are configured to operate at zoom coefficients greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold.

[0133]

[0137] The following describes some exemplary techniques. These exemplary techniques can be used separately or together in any practical combination.

[0134]

[0138] Clause 1. A device for generating video content, the device comprising a first camera and one or more processors coupled to the first camera, wherein one or more processors are configured to receive a first set of frames captured by the first camera at a first rate, process the first set of frames to generate video content for display, receive a selection to zoom in or zoom out, receive a second set of frames captured by the first camera at a second rate less than the first rate during the zoom in or zoom out to a zoom threshold for the first camera, process the second set of frames to generate video content for display, and generate video content for playback at a third rate less than the first rate, including the first set of frames and the second set of frames.

[0135]

[0139] Clause 2. The device described in Clause 1, which is equipped to receive a selection to zoom in or zoom out with slow-motion video capture enabled, and is equipped to receive a second set of frames with slow-motion video capture enabled.

[0136]

[0140] Clause 3. The device according to either Clause 1 or 2, wherein the first camera is one of a plurality of cameras coupled to one or more processors, wherein the device comprises a second camera of the plurality of cameras, wherein after a zoom threshold for the first camera is reached, one or more processors are configured to receive a third set of frames captured by the second camera of the plurality of cameras at a fourth rate greater than the second rate, and to process the third set of frames to generate video content for display, wherein generating video content for playback comprises generating video content for playback at a third rate, comprising a first set of frames, a second set of frames, and a third set of frames.

[0137]

[0141] Clause 4. The device according to Clause 3, wherein, while the first camera captures a first set of frames, or prior to that, one or more processors are configured to determine one or more of the autofocus (AF) parameters, auto white balance (AWB) parameters, and auto exposure control (AEC) parameters for each of the multiple cameras, and receiving a third set of frames captured by the second camera comprises receiving the third set of frames based on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera captures a first set of frames, or prior to that.

[0138]

[0142] Clause 5. The device according to any one of Clauses 1 to 4, wherein processing a second set of frames comprises performing digital zoom in or digital zoom out until a zoom threshold for the first camera is reached, wherein performing digital zoom in or digital zoom out until a zoom threshold for the first camera is reached comprises performing cropping and resizing on the second set of frames during an increase or decrease in the zoom factor.

[0139]

[0143] Clause 6. The device according to any one of Clauses 1 to 5, wherein the zoom threshold comprises a first zoom threshold, wherein the first camera is one of a plurality of cameras coupled to one or more processors, wherein after the first zoom threshold for the first camera is reached, one or more processors are configured to receive a third set of frames captured by a second camera of the plurality of cameras and to process the third set of frames in parallel with processing a second set of frames, wherein generating video content for playback comprises, at a third rate, a first set of frames, a second set of frames until a second zoom threshold for the first camera is reached, and a third set of frames after the second zoom threshold for the first camera has been reached, wherein both the first camera and the second camera are configured to operate at zoom coefficients greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold.

[0140]

[0144] Clause 7. A device described in any of Clauses 1 through 6, whose first rate is greater than or equal to 120 frames per second.

[0141]

[0145] Clause 8. A device as described in any of Clauses 1 through 7, where the third rate is equal to the second rate.

[0142]

[0146] Clause 9. A device as described in any of Clauses 1 through 8, comprising receiving a choice to zoom in or zoom out, receiving information indicating a pinch-zoom motion, or receiving information indicating a choice to zoom.

[0143]

[0147] Clause 10. A device described in any of Clauses 1 through 9, in which the image quality of the second set of frames is inferior to the image quality of the first set of frames.

[0144]

[0148] Clause 11. A method for generating video content, the method comprising: using one or more processors to receive a first set of frames captured by a first camera at a first rate; using one or more processors to process the first set of frames to generate video content for display; using one or more processors to receive a selection to zoom in or zoom out; using one or more processors to receive a second set of frames captured by the first camera at a second rate less than the first rate during the zoom in or zoom out to a zoom threshold for the first camera; using one or more processors to process the second set of frames to generate video content for display; and using one or more processors to generate video content for playback, including the first set of frames and the second set of frames, at a third rate less than the first rate.

[0145]

[0149] Clause 12. The method according to Clause 11, comprising receiving a choice to zoom in or zoom out with slow-motion video capture enabled, and receiving a second set of frames with slow-motion video capture enabled.

[0146]

[0150] Clause 13. The method according to any one of Clauses 11 and 12, wherein the first camera is one of a plurality of cameras coupled to one or more processors, and the method further comprises receiving a third set of frames captured by a second camera of the plurality of cameras at a fourth rate greater than a second rate after a zoom threshold for the first camera has been reached, and processing the third set of frames to generate video content for display, wherein generating video content for playback comprises generating video content for playback at a third rate including a first set of frames, a second set of frames and a third set of frames.

[0147]

[0151] Clause 14. The method of Clause 13, further comprising determining one or more of the autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters for each of a plurality of cameras while the first camera captures a first set of frames, or prior thereto, wherein receiving a third set of frames captured by a second camera comprises receiving a third set of frames based on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera captures a first set of frames, or prior thereto.

[0148]

[0152] Clause 15. The method according to any one of Clauses 11 to 14, wherein processing a second set of frames comprises performing digital zoom in or digital zoom out until a zoom threshold for the first camera is reached, wherein performing digital zoom in or digital zoom out until a zoom threshold for the first camera is reached comprises performing cropping and resizing on the second set of frames during an increase or decrease in the zoom factor.

[0149]

[0153] Clause 16. The method according to any one of Clauses 11 to 15, wherein the zoom threshold comprises a first zoom threshold, wherein the first camera is one of a plurality of cameras coupled to one or more processors, and the method further comprises receiving a third set of frames captured by a second camera of the plurality of cameras, and processing the third set of frames, in parallel with processing a second set of frames for the first camera after the first zoom threshold for the first camera has been reached, wherein generating video content for playback comprises, at a third rate, a first set of frames, a second set of frames until a second zoom threshold for the first camera has been reached, and a third set of frames after the second zoom threshold for the first camera has been reached, wherein both the first camera and the second camera are configured to operate at zoom coefficients greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold.

[0150]

[0154] Clause 17. The method described in any of Clauses 11 to 16, wherein the first rate is greater than or equal to 120 frames per second.

[0151]

[0155] Clause 18. The method described in any of Clauses 11 to 17, wherein the third rate is equal to the second rate.

[0152]

[0156] Clause 19. The method of any one of Clauses 11 to 18, wherein receiving a choice to zoom in or zoom out is one of receiving information indicating a pinch-zoom motion or receiving information indicating a choice to zoom.

[0153]

[0157] Clause 20. The method described in any of Clauses 11 to 19, wherein the image quality of the second set of frames is inferior to the image quality of the first set of frames.

[0154]

[0158] Clause 21. A computer-readable storage medium storing instructions, wherein, when the instructions are executed, one or more processors cause one or more processors to: receive a first set of frames captured by a first camera at a first rate; process the first set of frames to generate video content for display; receive a choice to zoom in or zoom out; receive a second set of frames captured by the first camera at a second rate less than the first rate during the zoom in or zoom out to a zoom threshold for the first camera; process the second set of frames to generate video content for display; and generate video content for playback at a third rate less than the first rate, including the first set of frames and the second set of frames.

[0155]

[0159] Clause 22. A computer-readable storage medium as described in Clause 21, wherein the instruction for causing one or more processors to receive a choice to zoom in or zoom out is an instruction for causing one or more processors to receive a choice to zoom in or zoom out with slow-motion video capture enabled, and the instruction for causing one or more processors to receive a second set of frames is an instruction for causing one or more processors to receive a second set of frames with slow-motion video capture enabled.

[0156]

[0160] Clause 23. A computer-readable storage medium according to any one of Clauses 21 and 22, wherein the first camera is one of a plurality of cameras, and the instruction further comprises an instruction causing one or more processors to receive a third set of frames captured by the second camera of the plurality of cameras at a fourth rate greater than the second rate after a zoom threshold for the first camera has been reached, and to process the third set of frames to generate video content for display, wherein generating video content for playback comprises generating video content for playback at the third rate, including a first set of frames, a second set of frames, and a third set of frames.

[0157]

[0161] Clause 24. A computer-readable storage medium according to Clause 23, further comprising instructions to cause one or more processors to determine one or more of the autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters for each of a plurality of cameras while the first camera captures a first set of frames, or before thereafter, wherein receiving a third set of frames captured by a second camera comprises receiving a third set of frames based on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera captures a first set of frames, or before thereafter.

[0158]

[0162] Clause 25. A computer-readable storage medium according to any one of Clauses 21 to 24, wherein the zoom threshold comprises a first zoom threshold, wherein the first camera is one of a plurality of cameras, and the instruction further comprises an instruction causing one or more processors to receive a third set of frames captured by a second camera of the plurality of cameras and process the third set of frames in parallel with processing a second set of frames after the first zoom threshold for the first camera has been reached, wherein generating video content for playback comprises, at a third rate, a first set of frames, a second set of frames until a second zoom threshold for the first camera has been reached, and a third set of frames after the second zoom threshold for the first camera has been reached, wherein both the first camera and the second camera are configured to operate at zoom coefficients greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold.

[0159]

[0163] Clause 26. A device for generating video content, the device comprising: means for receiving a first set of frames captured by a first camera at a first rate; means for processing the first set of frames to generate video content for display; means for receiving a choice to zoom in or zoom out; means for receiving a second set of frames captured by the first camera at a second rate less than the first rate during zoom in or zoom out to a zoom threshold for the first camera; means for processing the second set of frames to generate video content for display; and means for generating video content for playback, including the first set of frames and the second set of frames, at a third rate less than the first rate.

[0160]

[0164] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or codes on a computer-readable medium, or transmitted through a computer-readable medium, and executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media. Thus, the computer-readable medium may generally correspond to non-transient tangible computer-readable storage media. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described herein. Computer program products may include computer-readable media.

[0161]

[0165] Such computer-readable storage media may include, but are not limited to, examples, RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. It should be understood that computer-readable storage media and data storage media do not include carrier waves, signals or other temporary media, but instead cover non-temporary tangible storage media. As used herein, disks and discs include compact discs (CDs), laserdiscs (discs), optical discs (discs), digital multipurpose discs (DVDs), floppy disks (discs), and Blu-ray discs (discs), where a disk typically reproduces data magnetically and a disc optically reproduces data by laser. Combinations of the above should also be included within the scope of computer-readable media.

[0162]

[0166] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Therefore, the term “processor” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some embodiments, the functions described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. The techniques can also be adequately implemented in one or more circuits or logic elements.

[0163]

[0167] The techniques of this disclosure can be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). While various components, modules, or units have been described in this disclosure to highlight the functional aspects of devices configured to implement the disclosed techniques, these components, modules, or units do not necessarily require implementation by different hardware units. Rather, as described above, the various units, along with suitable software and / or firmware, may be combined in a codec hardware unit, including one or more processors described above, or provided by a set of interoperable hardware units.

[0164]

[0168] Various examples were described. These and other examples fall within the scope of the following claims. The invention described in the original claims of this application is listed below. [C1] A device for generating video content, wherein the device is The first camera, One or more processors coupled to the first camera, The one or more processors are equipped with Receiving a first set of frames captured by the first camera at a first rate, Processing the first set of frames to generate video content for display, Receiving the option to zoom in or zoom out, Receiving a second set of frames captured by the first camera at a second rate smaller than the first rate during the zoom-in or zoom-out up to the zoom threshold for the first camera, Processing the aforementioned second set of frames to generate video content for display, To generate video content for playback, including the first set of frames and the second set of frames, at a third rate smaller than the first rate; A device configured to perform the following actions. [C2] Receiving the selection to zoom in or zoom out includes receiving the selection to zoom in or zoom out while slow-motion video capture is enabled. Receiving the second set of frames includes receiving the second set of frames with slow-motion video capture enabled. The device described in C1. [C3] The first camera is one of a plurality of cameras coupled to the one or more processors, and the device comprises a second camera among the plurality of cameras. After the zoom threshold for the first camera is reached, the one or more processors: Receiving a third set of frames captured by the second camera among the plurality of cameras at a fourth rate greater than the second rate, Processing the third set of frames to generate video content for display, It is configured to do the following: The device according to C1, wherein generating video content for playback comprises generating video content for playback, which includes, at the third rate, the first set of frames, the second set of frames, and the third set of frames. [C4] While the first camera captures the first set of frames, or before that, the one or more processors are configured to determine, for each of the multiple cameras, one or more of the following parameters: autofocus (AF) parameters, auto white balance (AWB) parameters, and auto exposure control (AEC) parameters. Receiving the third set of frames captured by the second camera comprises receiving the third set of frames based on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera is capturing the first set of frames, or before that. The device described in C3. [C5] Processing the second set of frames comprises performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached. Performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached is: Performing cropping and resizing on the second set of frames during an increase or decrease in the zoom factor, A device as described in C1, comprising the above. [C6] The zoom threshold comprises a first zoom threshold, and the first camera is one of a plurality of cameras coupled to the one or more processors. After the first zoom threshold for the first camera is reached, the one or more processors: In parallel with processing the second set of frames, a third set of frames captured by the second camera among the plurality of cameras is received, Processing the aforementioned third set of frames, It is configured to do the following: Generating video content for playback comprises generating video content for playback that, at the third rate, includes the first set of frames, the second set of frames until a second zoom threshold for the first camera is reached, and the third set of frames after the second zoom threshold for the first camera has been reached. Both the first camera and the second camera are configured to operate at a zoom coefficient that is greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold. The device described in C1. [C7] The device described in C1, wherein the first rate is greater than or equal to 120 frames per second. [C8] The device according to C1, wherein the third rate is equal to the second rate. [C9] Receiving the option to zoom in or zoom out means Receiving information indicating pinch-zoom motion, or Receiving information indicating the option to zoom, A device described in C1, comprising one of the following: [C10] The device according to C1, wherein the image quality of the second set of frames is inferior to the image quality of the first set of frames. [C11] A method for generating video content, wherein the method is Using one or more processors, receive a first set of frames captured by a first camera at a first rate, Using one or more of the above processors, the first set of frames is processed to generate video content for display. Using one or more of the aforementioned processors, receive a selection to zoom in or zoom out, During the zoom-in or zoom-out up to the zoom threshold for the first camera, one or more processors are used to receive a second set of frames captured by the first camera at a second rate smaller than the first rate. Using one or more of the aforementioned processors, the second set of frames is processed to generate video content for display. Using one or more of the aforementioned processors, generate video content for playback, including a first set of frames and a second set of frames, at a third rate smaller than the first rate. A method that includes [a certain feature]. [C12] Receiving the selection to zoom in or zoom out includes receiving the selection to zoom in or zoom out while slow-motion video capture is enabled. Receiving the second set of frames includes receiving the second set of frames with slow-motion video capture enabled. Method described in C11. [C13] The first camera is one of a plurality of cameras coupled to the one or more processors, The method described above is performed after the zoom threshold for the first camera is reached. Receiving a third set of frames captured by the second camera among the plurality of cameras at a fourth rate greater than the second rate, Processing the third set of frames to generate video content for display, Furthermore, The method of C11, wherein generating video content for playback comprises generating video content for playback, which includes, at the third rate, the first set of frames, the second set of frames, and the third set of frames. [C14] While the first camera captures the first set of frames, or before that, determine one or more of the autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters for each of the plurality of cameras. Furthermore, Receiving the third set of frames captured by the second camera comprises receiving the third set of frames based on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera is capturing the first set of frames, or before that. Methods used in C13. [C15] Processing the second set of frames comprises performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached. Performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached is: Performing cropping and resizing on the second set of frames during an increase or decrease in the zoom factor, A method for C11 comprising the following: [C16] The zoom threshold comprises a first zoom threshold, and the first camera is one of a plurality of cameras coupled to the one or more processors. The method described above is performed after the first zoom threshold for the first camera is reached. In parallel with processing the second set of frames, a third set of frames captured by the second camera among the plurality of cameras is received, Processing the aforementioned third set of frames, Furthermore, Generating video content for playback comprises generating video content for playback that, at the third rate, includes the first set of frames, the second set of frames until a second zoom threshold for the first camera is reached, and the third set of frames after the second zoom threshold for the first camera has been reached. Both the first camera and the second camera are configured to operate at a zoom coefficient that is greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold. Method described in C11. [C17] The method according to C11, wherein the first rate is greater than or equal to 120 frames per second. [C18] The method according to C11, wherein the third rate is equal to the second rate. [C19] Receiving the option to zoom in or zoom out means Receiving information indicating pinch-zoom motion, or Receiving information indicating the option to zoom, A method for C11 that includes one of the following. [C20] The method according to C11, wherein the image quality of the second set of frames is inferior to the image quality of the first set of frames. [C21] A computer-readable storage medium storing instructions, wherein, when an instruction is executed, one or more processors, Receiving a first set of frames captured by a first camera at a first rate, Processing the first set of frames to generate video content for display, Receiving the option to zoom in or zoom out, Receiving a second set of frames captured by the first camera at a second rate smaller than the first rate during the zoom-in or zoom-out up to the zoom threshold for the first camera, Processing the aforementioned second set of frames to generate video content for display, To generate video content for playback, including the first set of frames and the second set of frames, at a third rate smaller than the first rate; A computer-readable storage medium that enables the following process. [C22] The instruction causing one or more processors to receive the selection to zoom in or zoom out includes an instruction causing one or more processors to receive the selection to zoom in or zoom out while slow-motion video capture is enabled, The instruction causing one or more processors to receive the second set of frames includes an instruction causing one or more processors to receive the second set of frames while slow-motion video capture is enabled. Computer-readable storage medium as described in C21. [C23] The first camera described above is one of several cameras, The instruction is given to one or more processors after the zoom threshold for the first camera has been reached. Receiving a third set of frames captured by the second camera among the plurality of cameras at a fourth rate greater than the second rate, Processing the third set of frames to generate video content for display, It further includes an order to carry out, Herein, generating video content for playback comprises generating video content for playback, comprising, at the third rate, a first set of frames, a second set of frames, and a third set of frames, as described in C21, for a computer-readable storage medium. [C24] The one or more processors described above: While the first camera captures the first set of frames, or before that, determine one or more of the autofocus (AF) parameters, automatic white balance (AWB) parameters, and automatic exposure control (AEC) parameters for each of the plurality of cameras. It further includes an order to carry out, Receiving the third set of frames captured by the second camera comprises receiving the third set of frames based on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera is capturing the first set of frames, or before that. Computer-readable storage medium as described in C23. [C25] The zoom threshold comprises a first zoom threshold, and the first camera is one of a plurality of cameras. The instruction is given to one or more processors after the first zoom threshold for the first camera has been reached. In parallel with processing the second set of frames, a third set of frames captured by the second camera among the plurality of cameras is received, Processing the aforementioned third set of frames, It further includes an order to carry out, Generating video content for playback comprises generating video content for playback that, at the third rate, includes the first set of frames, the second set of frames until a second zoom threshold for the first camera is reached, and the third set of frames after the second zoom threshold for the first camera has been reached. Both the first camera and the second camera are configured to operate at a zoom coefficient that is greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold. Computer-readable storage medium as described in C21. [C26] A device for generating video content, wherein the device is Means for receiving a first set of frames captured by a first camera at a first rate, Means for processing the first set of frames to generate video content for display, A means for receiving the option to zoom in or zoom out, Means for receiving a second set of frames captured by the first camera at a second rate smaller than the first rate during the zoom-in or zoom-out up to a zoom threshold for the first camera, Means for processing the second set of frames to generate video content for display, Means for generating video content for playback, including the first set of frames and the second set of frames, at a third rate smaller than the first rate; A device equipped with the following features.

Claims

1. A device for generating video content, wherein the device is The first camera and, One or more processors coupled to the first camera, The one or more processors are equipped with With slow-motion video capture enabled, a first set of frames captured by the first camera at a first rate is received, Processing the first set of frames to generate video content for display, With slow-motion video capture enabled, you will receive an option to zoom in or zoom out, With slow-motion video capture enabled, a second set of frames captured by the first camera at a second rate smaller than the first rate is received during the zoom-in or zoom-out to a zoom threshold for the first camera. Processing the second set of frames to generate video content for display, To generate video content for playback, including a first set of frames and a second set of frames, at a third rate smaller than the first rate, wherein the second rate is less than or equal to the third rate. A device configured to perform the following actions.

2. The first camera is one of a plurality of cameras coupled to the one or more processors, and the device comprises a second camera among the plurality of cameras. After the zoom threshold for the first camera is reached, the one or more processors: Receiving a third set of frames captured by the second camera among the plurality of cameras at a fourth rate greater than the second rate, Processing the third set of frames to generate video content for display, It is configured to do the following: Herein, generating video content for playback comprises generating video content for playback, which includes, at the third rate, the first set of frames, the second set of frames, and the third set of frames. While the first camera captures the first set of frames, or before that, the one or more processors are configured to determine, for each of the plurality of cameras, one or more of the following: autofocus (AF) parameters, auto white balance (AWB) parameters, and auto exposure control (AEC) parameters. The device according to claim 1, wherein receiving the third set of frames captured by the second camera comprises receiving the third set of frames based on one or more AF parameters and AEC parameters determined for the second camera while the first camera is capturing the first set of frames or before that.

3. Processing the second set of frames comprises performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached. Performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached is: Performing cropping and resizing on the second set of frames during an increase or decrease in the zoom factor, The device according to claim 1, comprising:

4. The zoom threshold comprises a first zoom threshold, and the first camera is one of a plurality of cameras coupled to the one or more processors. After the first zoom threshold for the first camera is reached, the one or more processors: In parallel with processing the second set of frames, a third set of frames captured by the second camera among the plurality of cameras is received, Processing the third set of frames, It is configured to do the following: Generating video content for playback comprises generating video content for playback that, at the third rate, includes the first set of frames, the second set of frames until a second zoom threshold for the first camera is reached, and the third set of frames after the second zoom threshold for the first camera has been reached. Both the first camera and the second camera are configured to operate at a zoom coefficient that is greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold. The device according to claim 1.

5. The device according to claim 1, wherein the first rate is greater than or equal to 120 frames per second, and / or the third rate is equal to the second rate.

6. Receiving the option to zoom in or zoom out means Receiving information indicating pinch-zoom motion, or Receiving information indicating the option to zoom, The device according to claim 1, comprising one of the following.

7. The device according to claim 1, wherein the image quality of the second set of frames is inferior to the image quality of the first set of frames.

8. A method for generating video content, wherein the method is With slow-motion video capture enabled, one or more processors receive a first set of frames captured by a first camera at a first rate, Using one or more of the above processors, the first set of frames is processed to generate video content for display. With slow-motion video capture enabled, one or more processors receive a selection to zoom in or zoom out, With slow-motion video capture enabled, during the zoom-in or zoom-out to a zoom threshold for the first camera, one or more processors are used to receive a second set of frames captured by the first camera at a second rate smaller than the first rate. Using one or more of the aforementioned processors, the second set of frames is processed to generate video content for display. Using one or more of the aforementioned processors, generate video content for playback, including a first set of frames and a second set of frames, at a third rate smaller than the first rate, wherein the second rate is less than or equal to the third rate. A method that includes [a certain feature].

9. The first camera is one of a plurality of cameras coupled to the one or more processors, The method described above is performed after the zoom threshold for the first camera is reached. Receiving a third set of frames captured by the second camera among the plurality of cameras at a fourth rate greater than the second rate, Processing the third set of frames to generate video content for display, Furthermore, Herein, generating video content for playback comprises generating video content for playback, which includes, at the third rate, the first set of frames, the second set of frames, and the third set of frames. While the first camera captures the first set of frames, or before that, one or more of the autofocus (AF) parameters, auto white balance (AWB) parameters, and auto exposure control (AEC) parameters are determined for each of the plurality of cameras. Furthermore, The method according to claim 8, wherein receiving the third set of frames captured by the second camera is performed on one or more of the AF parameters and AEC parameters determined for the second camera while the first camera is capturing the first set of frames or before that.

10. Processing the second set of frames comprises performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached. Performing digital zoom in or digital zoom out until the zoom threshold for the first camera is reached is: Performing cropping and resizing on the second set of frames during an increase or decrease in the zoom factor, The method according to claim 8, comprising:

11. The zoom threshold comprises a first zoom threshold, and the first camera is one of a plurality of cameras coupled to the one or more processors. The method described above is performed after the first zoom threshold for the first camera is reached. In parallel with processing the second set of frames, a third set of frames captured by the second camera among the plurality of cameras is received, Processing the third set of frames, Furthermore, Generating video content for playback comprises generating video content for playback that, at the third rate, includes the first set of frames, the second set of frames until a second zoom threshold for the first camera is reached, and the third set of frames after the second zoom threshold for the first camera has been reached. Both the first camera and the second camera are configured to operate at a zoom coefficient that is greater than or equal to the first zoom threshold and less than or equal to the second zoom threshold. The method according to claim 8.

12. The method according to claim 8, wherein the first rate is greater than or equal to 120 frames per second, and / or the third rate is equal to the second rate.

13. A computer-readable storage medium storing instructions, wherein, when executed, the instructions cause one or more processors to perform the method according to any one of claims 8 to 12.

Images