Device and Method for Camera Processing and Storage Medium

Abstract

Generally speaking, techniques for initiating recursive blending of multiple frames after a camera transition are described. After transitioning from a first camera mode to a second camera mode, one or more processors may blend one or more pixels of a first frame with one or more pixels of a second frame to produce a first blended frame, where the first frame is captured via a first camera and the second frame is captured via a second camera. Then, one or more processors may output the first blended frame.

Description

[0001] This application claims the priority of U.S. Application No. 17 / 009,285, filed on September 1, 2020, and U.S. Provisional Patent Application No. 63 / 023,610, filed on May 12, 2020, the entire contents of each of which are incorporated herein by reference. U.S. Application No. 17 / 009,285, filed on September 1, 2020, claims the benefit of U.S. Provisional Patent Application No. 63 / 023,610, filed on May 12, 2020. Technical Field

[0002] This disclosure relates to image processing. Background Art

[0003] Image capture devices are commonly incorporated into various devices. In this disclosure, an image capture device refers to any device capable of capturing one or more digital images, including devices capable of capturing still images and devices capable of capturing image sequences to record video. For example, an image capture device may include a standalone digital camera or digital video camcorder, a wireless communication device handset equipped with a camera (such as a mobile phone having one or more cameras), a cellular or satellite radiotelephone, a personal digital assistant (PDA) equipped with a camera, a panel or tablet computer, a gaming device, a computer device including a camera (such as a so-called "webcam"), or any device having digital imaging or video capabilities.

[0004] Certain image capture devices include multiple image sensors that capture image data through one or more lenses and then transmit the image data to a camera processor. An image capture device may include multiple lenses used in conjunction with multiple image sensors to capture images at many different zoom levels. Exemplary lens types include wide-angle lenses, ultra-wide-angle lenses, telephoto lenses, telescopic lenses, periscope zoom lenses, fisheye lenses, macro lenses, fixed-focus lenses, or various combinations thereof. In one example, a dual-camera configuration may include both a wide-angle lens and a telephoto lens simultaneously. Similarly, a triple-camera configuration may include an ultra-wide-angle lens, a wide-angle lens, and a telephoto lens. Summary of the Invention

[0005] Generally speaking, the present disclosure describes image processing techniques related to digital cameras having an image sensor and a camera processor. Specifically, the camera processor may be configured to perform various frame blending operations (e.g., pixel blending) in response to a transition between cameras. The camera may include multiple cameras of a computing device or otherwise be coupled to multiple cameras of a computing device, such as a standalone image capture device, a computer device equipped with a camera, a wireless communication device, etc. The computing device may include one or more camera processors, such as an image signal processor (ISP), and the image signal processor may in turn include one or more image processing engines (IPE) that control various signal processing pipeline operations.

[0006] The camera processor may be configured to obtain input frames (e.g., pixel values) of image data from different cameras and then generate corresponding output frames (e.g., preview display frames, still image captures, video frames, etc.) of the image data. The camera processor may output the frames of the image data to various output devices and / or camera modules for further processing, such as for 3A parameter synchronization, generating a video file via the output frames, configuring frames for display, configuring frames for storage, etc. That is, the camera processor may obtain incoming frames from one or more image sensors each coupled to one or more camera lenses and may then generate and output a stream of output frames to respective output destinations. In these examples, the camera processor may be configured to generate a stream of output frames that can dynamically represent a changed zoom level (e.g., increasing or decreasing the zoom level). In one example, the camera processor may receive an input to change the zoom level based on a pinch-zoom operation, gesture detection, etc.

[0007] To achieve various zoom levels, in some cases, the camera processor may initiate a transition between one camera and another based on a zoom level command. Different cameras may have different lenses, such as a first camera before a camera transition using a wide-angle lens and a second camera after a camera transition using a telephoto lens. The camera processor may initiate the camera transition according to various camera transition operations, including but not limited to lens transition, image sensor transition, moving prism actuation, camera module transition, etc.

[0008] In one example, a camera processor may transition from a first camera to a second camera in response to detecting that a particular zoom level has met a camera transition threshold. When implementing such a camera transition, such as during a zoom operation, the camera processor may effectively obtain a first set of input frames from the first camera prior to the camera transition and effectively obtain a second set of input frames from the other camera after the camera transition. The camera processor may then be configured to produce a single output frame stream based on corresponding output frames associated with each camera. However, due to various differences between the cameras, the frames obtained from each camera may flow together in a manner that is perceptible to the human eye as a transition, such as in the resulting video or preview display. Additionally, when processed as a single output stream according to a particular zoom command, the frames obtained from different cameras may ultimately result in relatively inefficient, imprecise, and / or inaccurate image processing operations, such as in terms of quality differences between the input frames, output frames, and / or the resulting video and / or display frames.

[0009] The above problems and others can be solved by the disclosed techniques for blending pixels of frames obtained from one camera after a camera transition with frames previously obtained from a second camera before the camera transition. That is, the camera processor may blend the input frames before the camera transition with the input frames after the camera transition. The camera processor may perform such blending based on a blending weight that represents a particular pixel contribution level. That is, the blending weight may simultaneously indicate both the proportion that the current frame will contribute to the blended frame and the proportion that the previous frame will contribute to the blended frame.

[0010] In some examples, the camera processor may geometrically distort the last frame obtained from the first camera to anticipate the camera transition. That is, the camera processor may distort the frame obtained from the first camera prior to the camera transition to align the pixels of the distorted frame with the pixel coordinates of a first expected input frame obtained from the second camera after the camera transition. When blending the last frame obtained from the first camera (e.g., the geometrically distorted frame) with subsequent frames obtained from the second camera, the camera processor may blend the pixels of the last frame with the pixels of the subsequent frames obtained after the camera transition. By initiating the blending process in response to the camera transition, the camera processor may advantageously minimize the amount of memory utilized in performing the transition blending process while providing the various quality benefits resulting from pixel blending after the camera transition.

[0011] In one example, in some cases, the first input frame and the second input frame can contribute different but corresponding amounts (e.g., proportional or inverse amounts) to produce a first blended output frame after the camera transformation, such that the output frame represents pixels from each input frame. The input frames can contribute a blend of pixels corresponding to the blending weights. Additionally, the camera processor can blend the first blended frame that enters the output stream after the camera transformation with subsequent input frames obtained from a second camera, and this process can continue recursively for a predetermined amount of time, or in some cases, until a predetermined number of frames after the camera transformation have been obtained, or otherwise, until the pixel contributions from the input frames obtained from the first camera have decreased in the overall recursive blend after the camera transformation to meet a predetermined threshold.

[0012] According to one or more of the various techniques of the present disclosure, differences in various quality metrics (e.g., photometric characteristics, hue, color, contrast, etc.) can be blended between frames obtained between cameras in a relatively efficient manner. This is because for a finite number of frames after the camera transformation, each input frame is recursively blended with the input frame obtained from the first camera. Additionally, although a finite number of input frames obtained from the second camera may be different from the corresponding output frames due to the blending of pixels with the input frame obtained from the first camera, the input frames from the first camera may not be significantly different from the corresponding output frames before the camera transformation. Instead, the camera processor can perform a geometric distortion phase separate from the blending phase, where the blending phase occurs after the camera transformation. In such examples, the blending phase can include pixel-level blending of two input frames, or pixel-level blending of a blended frame and an unblended frame that blends pixels at the local pixel level. Advantageously, the blending phase starts at the moment of transformation between cameras so as to consume less processing power and / or other resources when an appropriate blending transition may occur based on the input frame obtained from the first camera.

[0013] Other advantages of this process include efficiency, precision, and / or accuracy in various video decoding processes, because the frames output to the video decoder before the camera transformation can include pixel values that flow in a relatively predictable manner to the first blended frame after the camera transformation. That is, when, for example, there are no sudden changes in photometric details in the hue, color, and / or contrast information between the pixels of the output frames (such as geometric distortion frames and input frames obtained from the second camera), the video decoder may tend to perform temporal prediction techniques (e.g., compressing or decompressing P-frames and B-frames according to video decoding standards) more accurately. Thus, the last input frame obtained before the camera transformation (e.g., the last geometric distortion frame) can be advantageously blended with the frame input obtained from the second camera after the camera transformation, and recursively with subsequent input frames obtained from the second camera, such that certain quality metrics (e.g., photometric details) can be effectively preserved in response to the camera transformation and such that the occurrence of the camera transformation may not be readily perceivable by the end user.

[0014] In one example, the techniques of the present disclosure relate to an apparatus configured for camera processing, the apparatus including: a memory, and one or more processors in communication with the memory, the one or more processors being configured to: receive a signal to transition from a first mode to a second mode, wherein a first frame is received via a first camera of the apparatus when the apparatus is operating in the first mode, and wherein a second frame is received via a second camera of the apparatus when the apparatus is operating in the second mode; transition from the first mode to the second mode; mix one or more pixels of the first frame with one or more pixels of the second frame after the transition to produce a first mixed frame; and output the first mixed frame after the first frame.

[0015] In another example, the techniques of the present disclosure relate to a method for camera processing, the method including: receiving a signal to transition from a first mode to a second mode, wherein a first frame is received via a first camera configured to operate in the first mode, and wherein a second frame is received via a second camera configured to operate in the second mode; transitioning from the first mode to the second mode; mixing one or more pixels of the first frame with one or more pixels of the second frame after transitioning from the first mode to the second mode to produce one or more mixed pixels of a first mixed frame; and outputting the first mixed frame after the first frame.

[0016] In another example, the techniques of the present disclosure relate to an apparatus configured for camera processing, the apparatus including: means for determining a transition from a first camera mode to a second camera mode, wherein the first camera mode includes a mode in which a first frame is received via a first camera, and wherein the second camera mode includes a mode in which a second frame is received via a second camera; means for transitioning from the first camera mode to the second camera mode; means for mixing a first set of pixels of the first frame with a second set of pixels of the second frame after transitioning from the first camera mode to the second camera mode to produce a first mixed frame; and means for outputting the first mixed frame.

[0017] In another example, the techniques relate to an apparatus configured for camera processing, the apparatus including: means for mixing pixels of a first frame with pixels of a second frame in response to a camera transition from a first camera to a second camera to produce a first mixed frame; means for obtaining a third frame via the second camera; and means for mixing the first mixed frame with the third frame.

[0018] In another example, the techniques of the present disclosure relate to a non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: transition from a first camera mode to a second camera mode, wherein a first frame is received by the one or more processors when the one or more processors operate in the first camera mode, and wherein a second frame and a third frame are received by the one or more processors when the one or more processors operate in the second camera mode, wherein the first camera mode and the second camera mode are different from each other; in response to the transition, mix one or more pixels of the first frame with one or more pixels of the second frame to produce a first mixed frame including one or more mixed pixels; and mix one or more mixed pixels of the first mixed frame with one or more pixels of the third frame to produce a second mixed frame.

[0019] Details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a block diagram of a computing device configured to perform one or more exemplary techniques described in the present disclosure.

[0021] Figure 2 is a block diagram showing exemplary components of a Figure 1 computing device illustrating aspects of the techniques described in the present disclosure.

[0022] Figure 3 is a flowchart showing exemplary methods and techniques for transform mixing illustrating aspects of the techniques described in the present disclosure.

[0023] Figure 4 is a conceptual diagram showing an exemplary camera transition and an exemplary transform mixing mode illustrating aspects of the techniques described in the present disclosure.

[0024] Figure 5 is a flowchart showing exemplary methods and techniques for recursive transform mixing illustrating aspects of the techniques described in the present disclosure.

[0025] Figure 6 is a conceptual diagram showing an example of recursive transform mixing illustrating aspects of the techniques described in the present disclosure.

[0026] Figure 7 is a diagram showing an example of recursive transform mixing illustrating aspects of the techniques described in the present disclosure.

[0027] Figure 8 is a conceptual diagram showing an example of recursive transform mixing illustrating aspects of the techniques described in the present disclosure.

[0028] Figure 9 is a conceptual diagram showing an exemplary camera transformation according to aspects of the techniques described in the present disclosure and various stages before and after the exemplary camera transformation.

[0029] Figure 10 is a conceptual diagram showing an exemplary camera transformation according to aspects of the techniques described in the present disclosure and various stages that occur before and after the exemplary camera transformation.

[0030] Figure 11 is a conceptual diagram showing an exemplary camera transformation according to aspects of the techniques described in the present disclosure and various stages including a pixel blending stage and a geometric distortion stage.

[0031] Figure 12 is a timing diagram showing an exemplary camera transformation technique according to aspects of the techniques described in the present disclosure. DETAILED DESCRIPTION

[0032] A camera (including a camera processor and an image sensor) may capture image data frames using one or more image sensors. The camera may output the captured frames (including the pixel information defining the frames) to the camera processor for processing. Among other processing techniques employed, the camera processor may perform various lens and / or image sensor transformation techniques in various situations when receiving the image data in order to effectively transform between the lenses and sensors of a multi-lens and / or multi-sensor image capture device. In one example, the camera processor may initiate or otherwise cause a transformation from a first camera having a first field of view (FOV) and / or a first focal length to a second different camera having a second FOV and / or a second focal length. In an illustrative example, the first camera may include a wide-angle camera, and the second camera may include a telephoto camera. Although various examples are described herein with reference to specific camera types and / or specific zoom operations (e.g., zoom in), the techniques of the present disclosure are not limited thereto, and these techniques may be applied to any number of different camera combinations (e.g., triple camera configurations) or zoom operations.

[0033] In one example, a camera processor may initiate a transition blending process in response to a camera transition from a first camera to a second camera. Accordingly, the camera processor may blend pixels of a first frame with pixels of a second frame to produce a first blended frame. In this case, the first frame may be obtained from the first camera, while the second frame may be obtained from the second camera. Additionally, the second frame may be obtained after the first frame and after the transition to the second camera. Similarly, the first frame may be obtained before the second frame and before the transition from the first camera to the second camera. In some examples, the first frame may include the last frame of a geometric distortion process immediately before a valid switching point between the cameras. In an illustrative example, the camera processor may geometrically distort the last frame and the penultimate frame obtained from the first camera before transitioning to the second camera.

[0034] In some examples, the camera processor may continuously output the first blended frame with the first frame obtained from the first camera. In such examples, the camera processor may produce an output frame representing the conversion of pixel values across frames, e.g., resulting in an uninterrupted transition between the photometric qualities of the various output frames obtained before and after the camera transition. Otherwise, a sudden transition between the output frames on either side of the camera transition may become apparent to the end user, or in another example, may result in an anomaly during an exemplary video decoding process. That is, when comparing the frame obtained from the first camera immediately before an automatic camera transition with the image obtained from the second camera immediately after the camera transition, the transition between different cameras may result in a mismatch between various quality characteristics.

[0035] In an illustrative and non-limiting example, the camera transition may result in a mismatch between certain photometric qualities of the pixel representations of a first set of output frames capturing a particular scene when compared to the corresponding photometric qualities of the pixel representations of a second set of output frames capturing the same scene. In some cases, such a mismatch caused by the transition between different cameras may be based on differences between image sensors manufactured by different manufacturers, but in any case, may be undesirable for many different reasons. For example, such differences between the output frames may be magnified at increasingly high camera resolutions and / or higher display resolutions. That is, a sudden transition may ultimately result in an overall negative experience of the camera transition initiated in response to a change in the zoom level, where such a zoom level may be desirable from the user's perspective.

[0036] The above problems and other problems can be solved by the disclosed technique for smoothing the transition between camera transitions by recursively blending input frames obtained from a second camera with input frames obtained from a first camera. Specifically, the frame blending operation can be initiated in response to a camera transition. That is, the frame blending operation can be initiated when the camera transitions. The camera processor can generate a blended frame as a first output frame after the camera transition. In one example, the camera processor can obtain the blended frame by blending the pixels of the last input frame obtained before the camera transition with the pixels of the first input frame obtained after the camera transition. In some examples, the input frames can include geometrically distorted frames that the camera processor distorts in anticipation of a transition between cameras in order to align the input frames during the geometric distortion phase before the camera transition. In any case, the camera processor can blend the pixels of the input frame before the camera transition with the input frame after the camera transition. The camera processor can perform this blending based on a pixel contribution level representing a blending weight that determines how much the first input frame and the second input frame will contribute to the first blended frame after the camera transition.

[0037] In another example, the output frame before the camera transition can represent the input frame obtained from the first camera and be geometrically distorted to compensate for the differences between the first camera and the second camera. In some examples, the camera processor can perform the distortion operation by geometrically displacing the individual pixels in the frame in order to properly align with the frame of another camera, for example, when reverse-distorting for display, output to memory, output to a video encoder, when received by a video decoder, and so on. Before or in some cases after outputting the geometrically distorted frame as the output frame, the geometrically distorted frame can be blended with the input frame obtained from the second camera after the camera transition.

[0038] In an illustrative example, the first mixed frame has corresponding contribution levels of pixels from the first frame and pixels from the second frame (e.g., 90% and 10%, 80% and 20%, 70% and 30%, 60% and 40%, 50% and 50%, 40% and 60%, 30% and 70%, 20% and 80%, 10% and 90%, etc.). In some examples, the blending weight coefficient 33 can result in substantially no blending after a camera transition, such as by including a value less than 0.1 (e.g., 0.5%, 0.01%, 0.00%, etc.). In this case, the value 0 (e.g., 0%) can represent a hard transition (e.g., no smooth transition). However, generally speaking, as an example, a blending weight coefficient between 0.5 and 0.8 causes one or more camera processors to perform a blending transition that provides a cascade of pixel values throughout the process, recursively blending with subsequent frames, such that the transition between the first camera (e.g., the first lens and image sensor combination) and the second camera (e.g., the second lens and image sensor combination) is less noticeable or not noticeable at all from the perspective of the end user and / or effectively reduces the consumption of irrelevant power, processing resources, and / or memory resources.

[0039] In addition, the first mixed frame can be blended with subsequent input frames obtained from the second camera, such that the previous frame obtained from the first camera can be recursively blended with each subsequent frame during the process of weakening or attenuating the pixel contribution level. That is, the camera processor can apply a blending weight that determines the contribution level of the mixed frame recursively over time. In one example, the pixels of the first frame can represent a 20% contribution level to the first mixed frame, while the second frame can represent an 80% contribution level to the second mixed frame. For a third input frame, the pixels of the third frame can represent an 80% contribution level to the second mixed frame, while the first mixed frame can represent a 20% contribution level to the second mixed frame. That is, the first frame obtained from the first camera represents a 4% contribution level, which may continue to cascade and further weaken throughout the transition blending process due to the blending function and blending weight.

[0040] Figure 1is a block diagram of a computing device 10 configured to perform one or more of the exemplary techniques described in this disclosure. Examples of the computing device 10 include a computer (e.g., a personal computer, a desktop computer, or a laptop computer), a mobile device (such as a tablet computer), a wireless communication device (e.g., a mobile phone, a cellular phone, a satellite phone, and / or a mobile phone handset), an Internet phone, a digital camera, a digital video recorder, a handheld device (such as a portable video game device or a personal digital assistant (PDA)), a drone device, or any device that may include one or more cameras. In some examples, the computing device 10 may include a central processing unit (CPU) 16, a video encoder / decoder 17, a graphics processing unit (GPU) 18, local memory 20 of the GPU 18, a user interface 22, a memory controller 24 that provides access to system memory 30, and a display interface 26 that outputs a signal to display graphical data on a display 28.

[0041] As Figure 1 shown in the example of, the computing device 10 includes one or more image sensors 12A-N. In some cases herein, the image sensors 12A-N may be simply referred to as "sensors 12", while in other cases, they may be referred to as multiple "sensors 12" as appropriate. The computing device 10 also includes one or more lenses 13A-N and a camera processor 14. Similarly, in some cases herein, the lenses 13A-N may be simply referred to as "lenses 13", while in other cases, they may be referred to as multiple "lenses 13" as appropriate. In some examples, the sensor 12 represents one or more image sensors 12, each of which may include processing circuitry, an array of pixel sensors (e.g., pixels) for capturing a representation of light, a memory (such as a buffer memory or on-chip sensor memory), and the like.

[0042] As Figure 1As shown, the computing device 10 includes a plurality of cameras 15. As used herein, the term "camera" refers to a specific image sensor 12 of the computing device 10 or a plurality of image sensors 12 in the computing device 10, wherein the image sensor 12 is arranged in combination with one or more lenses 13 of the computing device 10. That is, the first camera 15 of the computing device 10 refers to a first set of devices including one or more image sensors 12 and one or more lenses 13, and the second camera 15 separated from the first camera 15 refers to a second set of devices including one or more image sensors 12 and one or more lenses 13. In addition, the camera processor 14 or the CPU 16 can receive image data from the image sensor 12 of a specific camera 15. That is, in some examples, the camera processor 14 or the CPU 16 can receive a first set of image data frames from the first image sensor 12 of the first camera 15 and a second set of image data frames from the second image sensor 12 of the second camera 15.

[0043] In one example, the term "camera" as used herein refers to a combined image sensor 12 and lens 13 coupled together, which are configured to capture at least one frame of image data and transmit the at least one frame of image data to the camera processor 14 and / or the CPU 16. In an illustrative example, the first camera 15 is configured to transmit a first frame of image data to the camera processor 14, and the second camera 15 is configured to transmit a second frame of image data to the camera processor 14, where the two frames are captured by different cameras, which can be demonstrated, for example, by the difference in the field of view between the first frame and the second frame. The difference in the field of view can correspond to the difference in the focal length between the first camera 15 and the second camera 15. In such an example, the camera processor 14 or the CPU 16 can cause a transition from the first camera 15 to the second camera 15 (e.g., in response to a change in the zoom level) by initiating a hybrid conversion process according to one or more of the various techniques of the present disclosure.

[0044] In one example, the camera processor 14 can receive a first frame and a second frame from two different cameras 15, and then can mix the first frame with the second frame to generate a first mixed frame after switching from the first camera 15 to the second camera 15 as the main camera for generating content sent from the camera processor 14 to one or more of the display interface 26, the system memory 30, and / or the video encoder / decoder 17. In some examples, the camera processor 14 can cause a change in power usage between the first camera 15 and the second camera 15 to achieve the switch. That is, the second camera 15 can increase power usage via the combination of the specific lens 13 and image sensor 12 corresponding to the second camera 15, while the first camera 15 can reduce the power usage consumed via the combination of the specific lens 13 and image sensor 12 corresponding to the first camera 15.

[0045] In an illustrative example, a single computing device 10 may include multiple different cameras 15 (e.g., a mobile phone having one or more front cameras and one or more rear cameras). In some examples, a computing device 10 may include a first camera 15 having one or more image sensors 12 and one or more lenses 13 and a second camera 15 having one or more image sensors 12 and one or more lenses 13. The first camera 15 may have a first focal length that results from a particular combination of the lens 13 and sensor 12 of the first camera 15, while the second camera 15 may have a second focal length that results from a particular combination of the lens 13 and sensor 12 of the second camera 15. In one example, the camera processor 14 may switch from the first camera 15 to the second camera 15, and in response to that switch, the camera processor 14 may blend a frame obtained from the first camera 15 with a subsequent frame obtained from the second camera 15.

[0046] Although some exemplary techniques are described herein with respect to a single sensor 12, the exemplary techniques are not limited thereto and may be applicable to various camera types for capturing images / videos, including devices having multiple image sensors and / or multiple lens types. For example, the computing device 10 may include a dual-lens device, a triple-lens device, a 360-degree camera lens device, etc. Thus, each combination of a lens 13 and an image sensor 12 may provide various zoom levels, angle of views (AOV), focal lengths, fields of view (FOV), etc. In some examples, a particular image sensor 12 may be assigned to each lens 13, and vice versa. For example, multiple image sensors 12 may be assigned to different lens types (e.g., wide-angle lens, ultra-wide-angle lens, telephoto lens, and / or periscope lens, etc.) respectively.

[0047] In examples including multiple lenses 13, the CPU 16 and / or the camera processor 14 may activate a particular lens 13 or a combination of lenses 13 in response to receiving a user input (e.g., via the user interface 22). For example, the CPU 16 and / or the camera processor 14 may receive a user input via the user interface 22, including zoom settings (e.g., a user's selection of a particular lens 13 (e.g., a fish-eye lens camera), a zoom command, etc.). In another example, the camera processor 14 may receive zoom settings regardless of the user interface 22, such as from the system memory 30 or the CPU 16. In one example, the camera processor 14 may receive zoom settings via the user interface 22 and may also retrieve pre-programmed zoom settings from the system memory 30. In such an example, the camera processor 14 may use the zoom settings alone, or in some cases, may combine the zoom settings received from different sources.

[0048] In some examples, the CPU 16 can automatically determine which lens 13 to activate and select an initial lens 13 to activate or switch from one lens 13 to another. For example, the CPU 16 and / or the camera processor 14 can determine which lens 13 to activate when detecting operating conditions that meet specific lens selection criteria (e.g., a zoom level that meets a predetermined camera switching threshold, a change in lighting conditions, an input from a user calling for a specific lens 13, etc.). In some examples, multiple cameras can be used in combination with each other to capture a composite image (e.g., a panoramic image).

[0049] In some examples, the camera processor 14 can adjust the effective focal lengths of one or more lenses 13 and one or more image sensors 12. For example, the camera processor 14 can activate a lens actuator that moves the lens 13 towards or away from the corresponding image sensor 12. In another example, the camera processor 14 can activate a lens switch from a first lens 13A to a second lens 13B. In any case, the camera processor 14 can adjust the focal length of the image data frames captured via the lens 13 and the image sensor 12. The camera processor 14 can perform such adjustments in response to zoom settings (e.g., a zoom command, auto-zoom programming, a zoom level input, a zoom transition, etc.). In some examples, the camera processor can perform such adjustments while executing various autofocus techniques.

[0050] In some examples, a single image sensor 12 can correspond to multiple lenses 13. In such examples, an optical waveguide can be used to direct the light incident on the lens 13 to the corresponding image sensor 12. Exemplary optical waveguides can include prisms, moving prisms, mirrors, etc. In this way, the light received from a single lens can be redirected to a specific sensor 12, such as away from one sensor 12 and towards another sensor 12. For example, the camera processor 14 can move a prism and redirect the light incident on one of the lenses 13 so as to effectively change the focal length.

[0051] Although the various structures of the computing device 10 are shown as separate in Figure 1 , the techniques of the present disclosure are not limited thereto, and in some examples, these structures can be combined to form a system-on-chip (SoC). As an example, the camera processor 14, the CPU 16, the GPU 18, and the display interface 26 can be formed on a common integrated circuit (IC) chip. In some examples, one or more of the camera processor 14, the CPU 16, the GPU 18, and the display interface 26 can be formed on separate IC chips. Various other arrangements and combinations are possible, and the techniques of the present disclosure should not be considered limited to Figure 1The example shown. In one example, CPU 16 may include camera processor 14 such that one or more camera processors 14 are part of CPU 16. In such an example, CPU 16 may be configured to perform one or more of the various techniques otherwise ascribed to camera processor 14 herein. For purposes of this disclosure, camera processor 14 will be described herein as separate and distinct from CPU 16, although that is not always the case.

[0052] Figure 1 The various structures shown may be configured to communicate with each other using bus 32. Bus 32 may be any of a variety of bus structures, such as a third-generation bus (e.g., HyperTransport bus or Wireless Bandwidth bus), a second-generation bus (such as an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI) Express bus, or an Advanced eXtensible Interface (AXI) bus), or other types of buses or device interconnects. It should be noted that Figure 1 The specific configuration of the bus and communication interfaces between the different structures shown is merely exemplary, and other configurations of computing devices and / or other image processing systems having the same or different structures may also be used to implement the techniques of this disclosure.

[0053] In addition, Figure 1 The various components shown (whether formed on one device or on different devices) (including sensor 12 and camera processor 14) may be formed as at least one of a fixed-function or programmable circuit or a combination of both, such as in one or more microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), or other equivalent integrated or discrete logic circuits. In addition, examples of local memory 20 include one or more volatile or non-volatile memory or storage devices, such as random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, magnetic data media, or optical storage media.

[0054] In some examples, memory controller 24 may facilitate data transfer to and from system memory 30. For example, memory controller 24 may receive memory read and write commands and service those commands for memory 30 to provide memory services to the various components of computing device 10. In these examples, memory controller 24 may be communicatively coupled to system memory 30. Although memory controller 24 is shown in Figure 1In an example of computing device 10, it is shown as a processing circuit separate from both CPU 16 and system memory 30. However, in some examples, some or all of the functions of memory controller 24 may be implemented on one or more of CPU 16, system memory 30, camera processor 14, video encoder / decoder 17, and / or GPU 18.

[0055] System memory 30 may store program modules and / or instructions and / or data that can be accessed by camera processor 14, CPU 16, and / or GPU 18. For example, system memory 30 may store user applications (e.g., instructions of a camera application), the resulting images from camera processor 14, etc. System memory 30 may additionally store information used and / or generated by other components of computing device 10. For example, system memory 30 may serve as the device memory of camera processor 14. System memory 30 may include one or more volatile or non-volatile memories or storage devices, such as RAM, SRAM, DRAM, ROM, EPROM, EEPROM, flash memory, magnetic data media, or optical storage media. Additionally, system memory 30 may store image data (e.g., video data frames, encoded video data, zoom settings, 3A parameters, etc.). In some examples, system memory 30 or local memory 20 may store image data into on-chip memory, such as into a memory buffer of system memory 30 or local memory 20. In another example, system memory 30 or local memory 20 may output image data for storage external to the memory or buffer of the chip, such as storage to a secure digital (SD TM ) card of the camera device or in some cases to another internal storage device of the camera device. In an illustrative example, system memory 30 or local memory 20 may be implemented as a buffer memory on camera processor 14 chip, GPU 18 chip, or both (where a single chip includes two processing circuits).

[0056] In some examples, system memory 30 may include instructions that cause camera processor 14, CPU 16, GPU 18, and / or display interface 26 to perform the functions ascribed to these components in this disclosure. Thus, system memory 30 may be a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors (e.g., camera processor 14, CPU 16, GPU 18, and display interface 26) to perform the various techniques of this disclosure.

[0057] In some examples, the system memory 30 is a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is not embodied in a carrier wave or a propagated signal. However, the term "non-transitory" should not be construed to mean that the system memory 30 is immovable or that its contents are static. As an example, the system memory 30 can be removed from the computing device 10 and moved to another device. As another example, a memory substantially similar to the system memory 30 can be inserted into the computing device 10. In certain examples, the non-transitory storage medium can store data that can change over time (e.g., in RAM).

[0058] In addition, the camera processor 14, the CPU 16, and the GPU 18 can store image data, user interface data, etc. in corresponding buffers allocated within the system memory 30. The display interface 26 can retrieve data from the system memory 30 and configure the display 28 to display an image represented by the image data on a screen via the user interface 22, for example. In some examples, the display interface 26 can include a digital-to-analog converter (DAC) that is configured to convert digital values retrieved from the system memory 30 into analog signals consumable by the display 28. In other examples, the display interface 26 can pass the digital values directly to the display 28 for processing.

[0059] In some examples, the camera processor 14 can include an image signal processor (ISP). For example, the camera processor 14 can include a camera interface that interfaces between the sensor 12 and the camera processor 14. The camera processor 14 can include additional circuitry for processing image content. The camera processor 14 can be configured to perform various operations on the image data captured by the sensor 12, including automatic white balance, color correction, or other post-processing operations. In some examples, the camera processor 14 can execute a "3A" algorithm. Such algorithms can include autofocus (AF), automatic exposure control (AEC), and automatic white balance (AWB) techniques. In such examples, 3A can represent the functionality of a statistical algorithm processing engine, where one of the camera processors 14 can implement and operate such a processing engine.

[0060] In some examples, the camera processor 14 is configured to receive image frames (e.g., pixel data) from the sensor 12 and process the image frames to generate image and / or video content. For example, the image sensor 12 can be configured to capture individual frames, frame bursts, sequences of frames for generating video content, photo still images captured during video recording, preview frames, or motion photos before and / or after capturing a still photo. The CPU 16, GPU 18, camera processor 14, or some other circuitry can be configured to process the image and / or video content captured by the sensor 12 into an image or video for display on the display 28. An image frame can generally refer to a data frame of a still image or a video data frame or a combination thereof, such as in the case of a motion photo. The camera processor 14 can receive pixel data of image frames in any format from the sensor 12. For example, the pixel data can include different color formats, such as RGB, YCbCr, YUV, etc. In any case, the camera processor 14 can receive multi-frame image data from the image sensor 12.

[0061] In examples that include multiple camera processors 14, the camera processors 14 can share the sensor 12, where each camera processor 14 can interface with each sensor 12. In any case, the camera processor 14 can initiate the capture of a video or an image of a scene using multiple pixel sensors of the sensor 12. In some examples, the video can include a sequence of individual frames. Thus, the camera processor 14 causes the sensor 12 to capture an image using multiple pixel sensors. The sensor 12 can then output pixel information (e.g., pixel values, luminance values, color values, charge values, analog-to-digital unit (ADU) values, etc.) to the camera processor 14, which represents the captured image or the sequence of captured images. In some examples, the camera processor 14 can process monochrome and / or color images to obtain an enhanced color image of the scene. In some examples, the camera processor 14 can determine a general mixing weight coefficient 33 for different types of pixel mixing, or can determine different mixing weight coefficients 33 for mixing different types of pixels that make up a pixel frame (e.g., a first mixing weight coefficient 33 for mixing pixels obtained via a monochrome sensor of a first camera 15 and pixels obtained via a monochrome sensor of a second camera 15, a second mixing weight coefficient 33 for mixing pixels obtained via a Bayer sensor of a first camera 15 and pixels obtained via a Bayer sensor of a second camera 15, etc.).

[0062] The computing device 10 may include a video encoder and / or a video decoder 17, either of which may be integrated as part of a combined video encoder / decoder (CODEC) (e.g., a video coder-decoder). The video encoder / decoder 17 may include a video encoder that encodes video captured by one or more cameras 15, or a decoder that decodes compressed or encoded video data. In some cases, the CPU 16 and / or the camera processor 14 may be configured to encode and / or decode video data, in which case the CPU 16 and / or the camera processor 14 may include the video encoder / decoder 17. In any case, the video encoder / decoder 17 may be configured to compress or decompress all or at least a portion of the potential FOV frames (e.g., multiple captured frames) of the image data received from the image sensor 12. Similarly, the video encoder / decoder 17 may be configured to compress or decompress output frames (e.g., mixed frames, unmixed frames, etc.). In some examples, the video encoder / decoder 17 may be further configured to compress or decompress preview frames representing zoom settings (e.g., a first set of zoom frames generated according to a first set of zoom settings), zoom setting information, and the like.

[0063] The CPU 16 may include a general-purpose or special-purpose processor that controls the operation of the computing device 10. A user may provide input to the computing device 10 to cause the CPU 16 to execute one or more software applications. Software applications executed on the CPU 16 may include, for example, a camera application, a graphics editing application, a media player application, a video game application, a graphical user interface application, or other programs. For example, the camera application may allow the user to control various settings of the camera 15. The user may provide input to the computing device 10 via one or more input devices (not shown), such as a keyboard, a mouse, a microphone, a touchpad, or another input device coupled to the computing device 10 via the user interface 22. In some examples, the user interface 22 may provide various methods for adjusting the blending weights for the mixed frames.

[0064] An exemplary software application is a camera application. The CPU 16 executes the camera application, and in response, the camera application causes the CPU 16 to generate the content output by the display 28. For example, the display 28 can output information such as light intensity, whether the flash is enabled, and other such information. The camera application can also cause the CPU 16 to instruct the camera processor 14 to process the image captured by the sensor 12 in a user-defined manner. A user of the computing device 10 can interact with the display 28 (e.g., via the user interface 22) to configure the manner in which the image is generated (e.g., apply zoom settings, use or not use the flash, focus settings, exposure settings, video or still image, and other parameters). For example, the CPU 16 can receive a zoom setting via the user interface 22 that commands an increase in the zoom level to a camera transition threshold. Additionally, the CPU 16 can receive values for a blending weight coefficient via the user interface 22, and the blending module 19 can receive these values and use them to blend the pixels of the frame, such as recursively after a camera transition. Additionally, via the IPE 42, the camera processor 14 can perform spatial denoising, temporal denoising, edge enhancement, sharpening, zooming (e.g., upscaling, downscaling), etc.

[0065] In some examples, the first image sensor 12 corresponding to the first camera can be the same image sensor 12 corresponding to the second camera. That is, the image sensor 12 corresponding to the second camera can be the same first image sensor corresponding to the first camera. The first camera and the second camera can provide different effective focal lengths, for example, due to the use of different lenses, activation of a motion prism, etc. In some examples, the camera processor 14 can move the lens 13A or the lens 13B according to an autofocus process and / or in response to a zoom setting. When performing an autofocus process or applying various zoom settings, the camera processor 14 can cause the lens 13A to move to achieve a specific focal length. The autofocus process can include focusing and subsequent defocusing, and vice versa.

[0066] The display 28 may include a monitor, a television, a projection device, a liquid crystal display (LCD), a plasma display panel, a light emitting diode (LED) array, an organic LED (OLED), an electronic paper, a surface conduction electron emission display (SED), a laser television display, a nanocrystal display, or other types of display units. The display 28 may be integrated within the computing device 10. For example, the display 28 may be the screen of a mobile phone handset, a tablet computer, or a laptop computer. Alternatively, the display 28 may be a stand-alone device coupled to the computing device 10 via a wired or wireless communication link. For example, the display 28 may be a computer monitor or a flat panel display connected to a personal computer via a cable or a wireless link. The display 28 may provide a preview frame that a user can view to see the content being stored or what the content might look like if the camera 15 were to actually take a photo or start recording a video photo.

[0067] In some zoom operations, the camera processor 14 may utilize such techniques in cases where pixel binning techniques are used to accommodate an increased or decreased zoom level, as described in U.S. Patent Application No. 16 / 667,662, titled "IMAGE CAPTURE MODE ADAPTATION," filed on October 29, 2019, by Liu et al., which is hereby incorporated by reference in its entirety. Generally, the image sensor 12 performs pixel binning (e.g., 4×4 binning, 3×3 binning, 2×2 binning, horizontal binning, vertical binning, etc.) by combining multiple pixels of the image sensor 12 into fewer pixels for output to the camera processor. The binning techniques improve the signal-to-noise ratio (SNR) by allowing the image sensor 12 (or, in some cases, the camera processor 14) to combine pixels together through various combination schemes (including averaging or summing multiple pixels for each output pixel). In accordance with this technique, the image sensor 12 may output fewer pixels during a binning level transition and / or a camera transition. Among other things, in these examples, the camera processor 14 may align the pixels of a first frame obtained before the camera transition and in accordance with a first binning mode (e.g., binning at a first binning level, such as an un-binned level) with the pixels of a second frame obtained after the camera transition and in accordance with a second binning mode (i.e., binning at a second binning level different from the first binning level). In an illustrative example, the camera processor 14 may obtain a first frame before the camera transition, the first frame including pixels binned at a first binning level and including a reduced number of pixels compared to the number of pixels available via the first image sensor 12. In response to the camera transition, the camera processor 14 may obtain a next frame including pixels binned at a second level different from the first binning level. The camera processor 14 may align the first frame with the second frame (e.g., via geometric warping), and thereby the camera processor 14 may blend the pixels of the first frame and the second frame to produce a first blended frame.

[0068] In addition, the obtained frame may include a composite frame created from multiple frames of the same camera. In one example, a first frame obtained from a first camera 15 before a camera transition may be a frame composed of multiple frames (e.g., multiple defocused frames), and the camera processor 14 may combine these frames to form a single focused frame for output via an output stream. In such a case, the first frame may be aligned with a frame obtained from a second camera 15, and thereby the camera processor 14 may blend the pixels of the first frame and the second frame to produce a first blended frame.

[0069] In some examples, the camera processor 14 may output a frame stream to the memory controller 24 to store the output frames as a video file. In some examples, the CPU 16, the video encoder / decoder 17, and / or the camera processor 14 may output frames to be stored as a video file. In some examples, the memory controller 24 may generate and / or store the output frames in any suitable video file format. In some examples, the video encoder / decoder 17 may encode the output frames before the CPU 16, the video encoder / decoder 17, and / or the camera processor 14 store the output frames as encoded video. The encoder / decoder 17 may encode the frames of the image data using various encoding techniques, including those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H 265 / High Efficiency Video Coding (HEVC), Versatile Video Coding (VCC), etc. and their extensions. In a non-limiting example, the CPU 16, the video encoder / decoder 17, and / or the camera processor 14 may cause the output frames (e.g., geometrically distorted frames, blended frames, unblended frames, etc.) to be stored using a Moving Picture Experts Group (MPEG) video file format.

[0070] When implementing such a camera transition, such as during a zoom operation, the camera processor 14 may effectively obtain a first set of input frames from the first camera 15 before the camera transition and a second set of input frames from another camera 15 after the camera transition. The camera processor 14 may then be configured to generate a single output frame stream based on the respective output frames associated with each camera 15. In these examples, the camera processor 14 may blend the currently captured frame obtained from one camera 15 with a previous output frame obtained from another camera 15 to produce a first blended frame that follows the previous output frame in the output frame stream. The camera processor 14 may be configured to perform various techniques when combining the respective output frames such that, for example, the end user will not perceive the occurrence of the camera transition, or will perceive the transition only to a particular minimum extent acceptable to the user. However, due to various differences between the cameras 15, such as differences between lens shapes, lens sizes, sensor algorithms, sensor sizes, sensor types, etc., the frames obtained from each camera 15 may flow together in a way that the human eye perceives as a stationary transition, such as in the resulting video or preview display. In another example, the camera processor may output frames to the video encoder / decoder 17. The video encoder / decoder 17 may be configured to produce a video file, such as a compressed video file. Thus, in some cases, the video encoder / decoder 17 may produce the video file based on an analysis of the various differences between the output frames of the output stream.

[0071] To bridge sudden transitions between cameras 15 and potentially produce a seamless stream of output frames, the camera processor 14 may employ additional preprocessing and / or postprocessing operations based on, for example, camera specifications, the context of the scene being captured, and / or offline processing that may not occur until a complete set of frames has been captured. However, certain operations, when implemented or if implemented improperly, may result in a relatively inefficient consumption of power, memory, and / or processing resources, or may involve potentially extraneous or imprecise camera operations. In any case, imprecise or extraneous camera operations for bridging sudden camera transitions may ultimately have the unintended effect of introducing unwanted quality differences between one or more input frames and one or more corresponding output frames. That is, certain operations may not only introduce such quality differences between frames immediately following a camera transition, but may also introduce such differences extraneously over a longer period of time following the camera transition and / or during a period of time prior to the camera transition.

[0072] Accordingly, in response to a camera transition from a first camera 15 to a second camera 15, the camera processor 14 may blend the pixels of a first frame with the pixels of a second frame to produce a first blended frame. In this case, the first frame is captured via the first camera 15, and the second frame is captured via the second camera 15. Additionally, the camera processor 14 may obtain a third frame captured via the second camera 15. In this case, the camera processor 14 may blend the pixels of the first blended frame with the pixels of the third frame to produce a second blended frame. That is, in response to a camera transition from a first camera 15 to a second camera 15, the camera processor 14 may initiate a blending process. During the blending process, the camera processor 14 may blend the pixels of the first frame with the pixels of the second frame to produce a first blended frame, where the first blended frame includes corresponding contribution levels from the first frame and the second frame.

[0073] Figure 2FIG. 0 is a block diagram showing exemplary components of a computing device 10 in accordance with one or more of the various techniques of the present disclosure. In some examples, the camera processor 14 may include an Image Front End (IFE) 40. In some examples, the IFE 40 interfaces between the image sensor 12 and one or more Image Processing Engines (IPE) 42. In some examples, one of the IPEs 42 may perform real-time processing of image data. In some examples, one or more of the IPEs 42 may be configured to perform techniques such as spatial processing, temporal processing, sharpening, denoising, etc. Additionally, the camera processor 14 may be configured to perform various operations on the image data captured by the sensor 12, including blending of inter-frame pixels, 3A synchronization, etc., where the 3A algorithm represents the functionality of the statistical algorithm processing engine of the camera processor 14. The 3A algorithm may include Auto Focus (AF), Auto Exposure Control (AEC), and Auto White Balance (AWB). When performing the 3A algorithm, the IPE 42 may be configured to perform one or more of the AF, AEC, and / or AWB processing to control the functionality of the image sensor 12.

[0074] In some examples, the camera processor 14 may include the IPE 42 representing the image processing engine internal to the camera processor 14. The IPE 42 may be configured to process various real-time image processing techniques (e.g., image processing that occurs at the same speed as the throughput of the image sensor 12). In some examples, the IPE 42 may be configured to perform processing techniques for video and image preview, such as 3A processing. In some examples, a “processing engine” may refer to hardware that can be used to process data. Exemplary hardware may include fixed function or programmable processing engines in the camera processor 14, GPU 18, CPU 16, DSP, or any other processing circuitry available on the computing device 10.

[0075] In some examples, the IPE 42 may operate the blending module 19, which is configured to perform pixel blending according to one or more of the various techniques of the present disclosure. In one example, the blending module 19 may implement one or more blending weight coefficients 33 to blend the pixels of the frame. The blending weight coefficients 33 may be adjusted automatically or by manual input. The blending weight coefficients 33 may include adjustable blending weights that can be automatically adjusted by the camera processor 14 or, in some cases, adjusted through a manual process. In one example, the camera processor 14 may adjust the blending weight coefficient 33 from a first value to a second value, such as based on the amount of detail captured in the frame. The camera processor 14 may apply a high-pass filter or another filter to the pixels of the incoming frame (e.g., the first frame 404, the second frame 406, or the third frame after the second frame 406) to determine the amount of detail captured in the incoming frame. In another example, the camera processor 14 may adjust the blending weight coefficient 33 based on the characteristic differences between the sensors 12 of the camera conversion, the 3A synchronization method employed by the camera processor 14 and the results of such 3A synchronization methods, the frame rate (e.g., frames per second), the geometric alignment (e.g., whether the frames are well-aligned, misaligned, misaligned by a specific amount, aligned by a specific amount, etc.), the camera mode, and the like.

[0076] In an illustrative example, the camera processor 14 may determine a first blending weight coefficient 33 to be applied during a camera mode involving slow-motion capture of a frame and a camera conversion resulting from a change in the zoom setting during the slow-motion capture. In another example, the camera processor 14 may determine a second blending weight coefficient 33 different from the first blending weight coefficient 33 during the normal capture mode (e.g., non-slow-motion) of the frame. In this illustrative example, the second blending weight coefficient 33 may be lower than the first blending weight coefficient 33.

[0077] In such examples, better processing, quality, resolution, etc. may be achieved for frames obtained from different cameras 15 before and after the camera conversion. Additionally, in specific implementations that allow high processing capabilities, selectively allocating processing resources (e.g., blending conversion processing, geometric distortion), such as based on the blending coefficients, the timing of when to start and stop blending (e.g., adjustment parameters), whether to invoke conversion blending, etc., may also reduce the power consumption of the computing device 10. That is, the camera processor 14 may generate a blended frame after the camera conversion such that the frame after the camera conversion is blended with the last frame obtained from the previous camera, making the transition between the frames relatively seamless in a visual and / or encoding sense and saving processing resources simultaneously.

[0078] In an illustrative example, the image sensor 12 may capture an image data frame and output the image data frame to the IFE 40. In some examples, the IFE 40 or the IPE 42 may apply a warping operation to the image data frame to produce a geometrically warped frame of the image data.

[0079] Figure 3 is a flowchart showing exemplary methods and techniques for conversion blending in accordance with various aspects of the techniques described in the present disclosure. In one example, the camera processor 14 may receive an image data frame from the image sensor 12. Additionally, the camera processor 14 may apply a zoom setting to the frame. In response to a particular zoom level, the camera processor 14 may initiate a camera transition from a first camera 15 (e.g., a first combination of the lens 13 and the image sensor 12) to a second camera 15 (e.g., a second combination of the lens 13 and the image sensor 12). In response to the camera transition, the camera processor 14 may initiate the application of a blending function that causes the blending of pixels from two input frames to produce a blended output frame.

[0080] The camera transition may include a lens 13 transition and / or an image sensor 12 transition. In some cases, the camera processor 14 may initiate the camera transition by deactivating the first camera 15 before blending a first frame obtained from the first camera 15 with another frame obtained via the second camera 15. In one example, the first camera 15 may have a first FOV, and the second camera may have a second FOV. In a non-limiting example, the first FOV may be set within the second FOV, and vice versa. In another example, the first camera 15 may have a first focal length, and the second camera 15 may have a second focal length. In various examples, the first camera 15 may be configured to capture image data of a scene according to the first focal length of the first camera 15 (such as the effective focal length corresponding to the camera processor 14 sending a signal to define the first combination of the lens 13 and the image sensor 12 of the first camera 15). Similarly, the second camera 15 may be configured to capture image data of the scene according to the second focal length (such as the effective focal length corresponding to the camera processor 14 sending a signal to define the second combination of the lens 13 and the image sensor 12 of the second camera 15), where the second focal length of the second camera 15 is different from the first focal length of the first camera 15.

[0081] In some examples, the camera processor 14 may cause an automatic transition from the first camera 15 to the second camera 15 to achieve a particular zoom level. In another example, the camera processor may cause a transition from the second camera to the third camera 15 in response to detecting another zoom level. It should be understood that various other zoom operations may be implemented based on similar camera transition techniques (e.g., transitioning from the third camera to the first camera in response to a decrease in the zoom level, transitioning from the first camera to the second camera in response to an increase in the zoom level, transitioning to the third and / or fourth cameras in response to an increase in the zoom level, etc.).

[0082] In some examples, the camera processor 14 may obtain a first frame via the first lens 13 and / or the first image sensor 12. The camera processor 14 may output the first frame prior to the camera transition, such as by outputting a geometrically distorted first frame. Similarly, the camera processor 14 may obtain a second frame via the second lens 13 and / or the second image sensor 12. In this case, prior to the transition blend, the first frame is aligned with the second frame before blending the pixels of the first frame and the pixels of the second frame.

[0083] In such an example, the camera processor 14 may initiate a blend transition (302) in response to a transition of the camera from a first camera mode to a second camera mode (such as by transitioning from the first camera 15 to the second camera 15). The camera processor 14 may initiate a camera transition in response to a received signal indicating that the zoom level is changing (e.g., a first zoom level corresponding to the first camera changes to a second zoom level that is higher relative to the first zoom level). Thus, the camera processor 14 may receive the signal as a transition signal indicating a transition from the first camera mode to the second camera mode to the camera processor 14. The transition from the first camera mode to the second camera mode may include a transition in power state with respect to a combination of a first camera lens 13 and an image sensor 12 having a first effective focal length (e.g., the first camera 15) and a second camera lens 13 and an image sensor 12 having a second effective focal length (e.g., the second camera 15), the second effective focal length being different from the first effective focal length of the combination of the first camera lens 13 and the image sensor 12.

[0084] In some examples, the camera processor 14 may disable the first camera 15 to a first power state and may enable (e.g., boost) the second camera 15 to a second power state that is higher than the first power state. In this way, the camera processor 14 may emphasize one camera over the other in terms of the power state assigned to each camera, where both the first camera 15 and the second camera 15 may remain active even if the first camera 15 is disabled to a lower power state relative to the power state of the second camera 15. That is, by changing the power state of the second camera 15 to a higher power state relative to the power state of the first camera 15 (e.g., full power capacity usage) and enabling the second camera 15 in response to a transition from a first camera mode to a second camera mode, the first camera may have previously remained at a higher power state when the first camera 15 was active as the primary camera. In another example, the transition from the first camera mode to the second camera mode may include a transition from a geometric distortion process to a transform blend process, where the transform blend process is separated from the geometric distortion process and is separated by a predetermined scaling level threshold.

[0085] In such an example, the camera processor 14 may transition from the first camera mode to the second camera mode in response to receiving a transition signal. After transitioning from the first camera mode to the second camera mode, the camera processor 14 may blend a first frame with a second frame to produce a first blended frame (304). In some examples, the camera processor 14 may blend the pixels of the first frame with the pixels of the second frame to produce the first blended frame. In one example, the camera processor 14 may blend one or more pixels of the first frame with the corresponding one or more pixels of the second frame to produce the first blended frame (e.g., a first frame including one or more blended pixels). Before blending the pixels, the camera processor 14 may spatially align the pixels of the second frame with the pixels of the first frame such that a particular pixel of one frame is aligned with a specific pixel of a subsequent frame to enable proper blending of the pixels, thereby producing a frame including one or more blended pixels. Additionally, the camera processor 14 may blend a first portion of the pixels of the first frame with a second portion of the pixels of the second frame, such as in a case where other portions of the frame include a blending weight of 0. In one example, the frame may include a blending weight of 0 for the edges or corners of the frame. In some examples, the camera processor 14 may blend the pixels associated with the first and second portions to provide an improved SNR with respect to the first blended frame, where the first blended frame may include other portions that do not include blended pixels (e.g., pixels corresponding to edges, pixels corresponding to corners, etc.).

[0086] In an illustrative example, a first frame is captured via a first camera 15 in a first mode, and a second frame is captured via a second camera 15 in a second mode. The second frame may be captured continuously with respect to the first frame. In one example, the first camera 15 may include a primary camera, such as a main camera or a default camera. In a non-limiting example, the first camera 15 may include a fixed focal length lens, a wide-angle lens, an ultra-wide-angle lens. Similarly, the second camera 15 may include a telephoto lens. In any case, the first camera 15 and the second camera 15 may be configured to image a scene using different effective focal lengths and / or at different zoom levels based on their respective different lenses 13 and / or image sensors 12. In one example, the camera processor 14 may utilize the first camera 15 as an effective primary camera to capture image data for generating a preview stream (e.g., for providing a preview via a display 28), video recording, video encoding, etc.

[0087] As used herein, when two frames are captured "continuously", this generally refers to frames that are substantially continuous, where in some cases there may still be some temporal overlap between consecutive frames. In an illustrative example involving the second camera 15 corresponding to a telephoto lens, the camera processor 14 may cause a longer exposure during the transition from the first mode to the second mode. Thus, the camera processor 14 may initiate the transition from the first mode to the second mode while the first camera 15 is capturing the first frame in order to provide a seamless output frame rate during the mode transition. That is, in cases where the exposure times are different due to the pixel sizes of the first camera 15 and the second camera 15, there may be some temporal overlap even though the second frame is captured continuously with respect to the first frame via the second camera 15.

[0088] In some examples, a first frame may be captured at a first time (t1), and a second frame may be captured at a second time (t2), where t1 and t2 are temporally subsequent (e.g., temporally consecutive), and the first camera 15 or the second camera 15 does not capture additional frames during this period. However, in some cases, the first camera 15 may capture multiple frames at the first time, and / or the second camera 15 may capture multiple frames at the second time, where, in the case where the first camera 15 or the second camera 15 includes a lens 13, the multiple frames may be combined to account for lens distortion, using frame fusion to correct frames having certain areas in focus and other areas out of focus due to the specific geometry of the lens 13. In any case, the camera processor 14 may obtain a first frame (e.g., a first fused frame, a geometrically distorted frame, etc.) via the first camera 15, and may obtain consecutive second frames (e.g., a first frame obtained during a second camera mode, a first fused frame obtained during a second camera mode) via the second camera 15, and blend the first frame obtained during the first mode via the first camera 15 with the second frame obtained via the second camera 15. That is, this technique can be performed even when various other camera processing techniques (such as frame fusion techniques) are used, or when the exposure time is advantageously changed during the transition from the first mode to the second mode (which may in turn result in partial time overlap between consecutive frames connecting the transition from the first mode using the first camera 15 to the second mode using the second camera 15).

[0089] The camera processor 14 may utilize the first camera 15 as the primary camera based on the FOV and / or based on user preferences. In one example, the camera processor 14 may receive user input indicating a specific lens 13 and image sensor 12 combination that the camera processor 14 will set as the first primary camera, where the first primary camera (e.g., the default camera) is the initial camera that is launched as the primary camera when the camera software application is launched via the computing device 10. The computing device 10 (e.g., the camera processor 14 and / or the CPU 16) may transition from a first mode to a second mode, where in the second mode, the computing device 10 causes the second camera 15 to take over as the primary camera while the first camera 15 is deactivated as the primary camera (e.g., is no longer emphasized, is powered from a higher power state relative to another camera 15 to a lower power state, etc.). Accordingly, the camera processor 14 may blend at least one pixel (e.g., a first pixel) of a first frame with at least one corresponding pixel of a second frame to produce a first blended frame (e.g., a blended pixel for the first blended frame). In one example, pixels of the second frame may be mapped to specific pixels of the first frame, which in some cases may be in accordance with a spatial alignment between the first frame and the second frame such that the pixels of the second frame and the corresponding pixels of the first frame represent temporally or spatially contiguous pixel representations of a scene to be captured via the first camera mode and the second camera mode. In some examples, the first blended frame includes at least one blended pixel that includes a first contribution level from at least one pixel of the first frame and a second corresponding contribution level from at least two pixels of the second frame.

[0090] In some examples, the camera processor 14 may blend one or more pixels of the first frame with one or more pixels of the second frame by applying a blending function that determines the pixel contribution levels as follows:

[0091] Y 1 ’ = α(X M *)+(1 – α)Y 1 ,

[0092] where

[0093] Y 1 ’ represents the first blended frame,

[0094] X M * represents the first frame with geometric distortion,

[0095] Y 1 represents the second frame, and

[0096] α represents a blending weight as follows: 0 ≤ α ≤ 1.

[0097] In some examples, the camera processor 14 can determine the value to be assigned to the blending weight automatically, based on a manual input, or both (e.g., between 0 and 1, between 0 and 100%, etc.). In such examples, the camera processor 14 can geometrically distort the first frame captured by the first camera 15 (X M *) before a camera transition (e.g., a transition that signals a switch from a first camera mode to a second camera mode to initiate a blending transition process). That is, the camera processor 14 can blend frames at a local pixel level for each pair of frames being blended. In some examples, the camera processor 14 can discard one or more pixels of the first frame, such as one or more edge pixels (e.g., corner pixels), while blending one or more other pixels of the first frame with one or more corresponding pixels of the second frame. In such examples, the first blended frame generated from the pixels of the first frame and the pixels of the second frame can include a contribution from the first frame and a mutual contribution from the second frame that is reduced relative to the first blended frame, based on the blending weight or another equivalent method of discarding pixels in a particular region of the blended frame during the blending transition process (e.g., during the second camera mode).

[0098] In some examples, the camera processor 14 can output the first blended frame (306). In some examples, the camera processor 14 can output the first blended frame after the first frame. The first frame can in some cases include a geometrically distorted frame. In such a case, the blended frame produced after the camera transition can have a different FOV than the FOV of the unblended frames obtained during a period before the camera transition (such as the distorted period before the camera transition). In some examples, the camera processor 14 can perform a global 3A synchronization process on frames including the first frame obtained from the first camera 15 and the first blended frame. In some examples, the camera processor 14 can output the first frame and the first blended frame to the video encoder / decoder 17, the system memory 30, and / or the display interface 26.

[0099] In some examples, the camera processor 14 may output a first blended frame (306). The camera processor 14 may output the first blended frame continuously with the first frame. Additionally, when outputting the first frame, the camera processor 14 causes geometric distortion of the first frame in accordance with a first distortion period that continues until the camera transition; and outputs the first frame including the geometric distortion. In some examples, the camera processor 14 may output the first blended frame to the encoder / decoder 17, the display interface 26 (e.g., for display via the display 28), or the system memory 30 (e.g., DRAM). In some examples, the camera processor 14 may output the first blended frame for additional processing via the IPE 42. In another example, the IPE 42 may output the first blended frame to another one in the IPE 42. In one example, one of the IPE 42s may output the distorted frame to the system memory 30 for subsequent access by the same IPE 42. In another example, the IPE 42 may output the first blended frame to another component or logical node (such as a 3A node) of the computing device 10 for further processing. In some examples, the camera processor 14 may output the first blended frame to the GPU 18. The GPU 18 may store the first blended frame in the local memory 20 for subsequent access and processing. For brevity, not all possible blended frame output results may be listed, but it should be understood that there may be more output results for the first blended frame in the context of the computing device 10 using the camera processor 14.

[0100] In some examples, the camera processor 14 may detect a change in the zoom direction after the camera transition. In such a case, the camera processor 14 may output a plurality of subsequent frames in accordance with a second camera process in response to detecting the change, where the second camera process is separate and different from the blending initiated at the camera transition. The second camera process may include a geometric distortion process or another process, such as a normal output stream process where the output frame is perpendicular to the input frame.

[0101] Figure 4 is a conceptual diagram showing an exemplary camera transition 402 and an exemplary transition blending mode in accordance with various aspects of the techniques described in the present disclosure. The camera processor 14 may obtain a first input frame corresponding to a first output frame 404 from a first camera 15. The camera processor 14 may obtain a second frame corresponding to a first blended frame 406 of the transition blending mode from a second camera 15. The camera processor 14 may perform to generate the first blended frame 406 based on the blending of pixels corresponding to the frame 404 (e.g., pixels of the frame 404, pixels of the input frame corresponding to the frame 404). Additionally, the camera processor 14 may blend the pixels of the blended frame 406 with a second input frame obtained from the second camera 15 to generate a second blended frame Y 2’。In some examples, the transition blend mode can continue until the pixel contribution from frame 404 drops below a predefined threshold, or until a predefined number of frames have been obtained from the second camera 15 and / or output according to the transition blend mode, or until a predefined amount of time has elapsed since the camera transition 402 (e.g., the start of the transition blend mode).

[0102] Figure 5 is a flowchart showing exemplary methods and techniques for recursive transition blending in accordance with aspects of the techniques described in this disclosure. In some examples, the camera processor 14 can perform pixel blending after a camera transition, e.g., based on a geometrically distorted frame (e.g., the first frame) obtained prior to the camera transition. The camera processor 14 can obtain a second frame after the first frame, i.e., the first frame after the camera transition. In such an example, the camera processor 14 can obtain a third frame, which can be the second frame after the camera transition (502). The third frame can be after the second frame obtained from the second camera 15. In such an example, the camera processor 14 can blend the pixels of the first blended frame with the pixels of the third frame to produce a second blended frame (504).

[0103] In such an example, the camera processor 14 can blend the first pixel of the first blended frame with the corresponding first pixel of the third frame to produce the second blended frame. In some examples, the camera processor 14 can align the first blended frame with the third frame before blending the pixels of the first blended frame with the pixels of the third frame. In one example, the camera processor 14 can distort the first blended frame to align with the corresponding pixels of the third frame before blending the first blended frame with the third frame to produce the second blended frame. Similar to outputting the first blended frame, the camera processor 14 can continuously output the second blended frame with the first blended frame (506).

[0104] Similar to producing the first blended frame, the camera processor 14 can blend the pixels of the first blended frame with the pixels of multiple consecutive frames based on the following recursive blending function:

[0105] Y 2 ’ = α(Y 1 ’)+(1 – α)Y 2 ,

[0106] where

[0107] Y 2 ’ represents the second blended frame,

[0108] Y 1 ’ represents the first blended frame, and

[0109] Y 2 represents the incoming frame (e.g., the third frame).

[0110] The recursive blending function can continue for each successive frame until a specific trigger frame is received. In one example, the trigger frame can be the nth frame input or the output of blending the nth frame. The camera processor 14 can output the nth frame as the final blended frame of the blending conversion process. In such an example, the camera processor 14 can determine the nth frame trigger by determining that a predetermined number of frames have been obtained or that a predetermined duration has elapsed since the start of the blending conversion process, or by determining that the pixel contributions of the pixels from the first frame include values that meet a predetermined pixel contribution threshold (below the contribution threshold).

[0111] In an illustrative example, the camera processor 14 can obtain the nth frame after the second frame and can output the nth frame as the final blended frame of the blending conversion process. In this case, the final blended frame of the blending conversion process includes a lower pixel contribution of the first frame pixels relative to the blended output frames before the final blended frame in the blending conversion process. In such an example, where the lower pixel contribution includes cascaded pixel contributions, the cascaded pixel contributions cascade pixel contributions throughout successive blended frames in the blending conversion process.

[0112] In addition, the camera processor 14 can obtain a frame after the nth frame (e.g., as part of a stage after the blending stage). In accordance with a camera mode initiated after the blending stage, the camera processor 14 can output the frame after the nth frame as an unblended frame.

[0113] Figure 6 is a conceptual diagram showing an example of recursive transform blending after a camera transform from a first camera 15(X) to a second camera 15(Y) in accordance with aspects of the techniques described in the present disclosure. Although a specific sequence of operations is described, the techniques of the present disclosure are not limited thereto. Thus, reference Figure 6 The example described is intended to illustrate an exemplary sequence of operations involving transform blending operations. An exemplary sequence is shown with reference to a conceptual blending of a first frame 404 and a second frame 408 to produce a blended frame 406.

[0114] The mixing module 19 can mix the pixels of the first frame 404 with the pixels of the second frame 408 to produce an output frame 406. Additionally, in some examples, the alignment module 34 can align the frames for mixing. That is, adding a scaling operation may cause the two frames to be misaligned for mixing. In such examples, the alignment module 34 can perform a second geometric distortion, which can be different from the first geometric distortion performed during a stage prior to the camera transformation. However, in some examples, the alignment module may not perform alignment of the frames, and thus, the mixed frames can be routed to the mixing module 19. Although shown as returning to the mixing module 19, it should be understood that this is for illustrative purposes, and the mixing module 19 can alternatively retain a copy of the mixed frames (such as in a buffer (e.g., a first-in-first-out buffer)), which can be used for mixing with the input frames.

[0115] In another example, the last frame of the geometric distortion process (e.g., frame 404) can be not output from or produced by the mixing module 19, but instead can be routed to the mixing module 19 from the IPE 42 or another module of the camera processor 14 (such as an IFE configured to perform frame distortion operations). In some cases, the camera processor 14 can determine that the camera processor 14 has not successfully aligned (e.g., distorted) the first frame 404 with the second frame 408. In such a case, the camera processor 14 can determine not to invoke the mixing module 19 of the IPE 42, but instead pass the input frame 408 and the first frame 404 as output frames without mixing. Otherwise, when the camera processor 14 determines an appropriate pre-transformation alignment (e.g., geometric distortion) between the first frame 404 obtained from the first camera 15 and the input frame 408 obtained from the second camera 15, the camera processor 14 can continue to invoke the mixing transformation process by utilizing the mixing module 19. As described herein, in some cases, the camera processor 14 can utilize the alignment module 34 to further align the pixels from (e.g.) the first frame 404 and the second frame 408, which might otherwise be misaligned due to differences in the scaling settings applied by the camera processor 14 to the two frames (e.g., 2.99x scaling and 3.01x scaling, where the camera transformation can be configured to occur at 3.00x scaling).

[0116] As Figure 6 shown, the last geometric distortion frame can enter the alignment module 34. In examples where the second geometric distortion is not implemented, the IPE 42 can route the frame 404 to the mixing module 19 and bypass the alignment module 34. Additionally, the IPE 42 can output the frame 404 ( Figure 6(not shown in the figure). The mixing module 19 can obtain the frame 408. The mixing module 19 can mix the pixels of the frame 404 with the pixels of the frame 408 to generate a mixed frame 406. The mixing module 19 can output the mixed frame 406 to continue the mixing process until the final frame of the mixing conversion process is mixed. The IPE 42 can interrupt routing the frame to the mixing module 19 after the mixing process ends, and generally, when not called, the IPE 42 can completely skip the recursive mixing mode, such as before the camera conversion and during the geometric distortion phase.

[0117] Figure 7 FIG. is an example showing recursive conversion mixing according to various aspects of the techniques described in the present disclosure. In one example, the camera processor 14 can cause a conversion between camera X (e.g., the first camera 15) and camera Y (e.g., the second camera 15). The frame X 702a can be a frame generated from camera X (e.g., a wide-angle camera, a telephoto camera, etc.). The frame Y 702b can be a frame generated from camera Y (e.g., a telephoto camera, a wide-angle camera, etc.). The output frame 708 can be a frame output to the display interface 26, sent to another device, and / or stored in the system memory 30. The frame number 704 can be used to indicate the frame corresponding to a specific time period (e.g., the frame X 702a, the frame Y 702b, and / or the output frame 708). Some configurations of the systems and methods disclosed herein can include conversion mixing initiated by the camera processor 14 in response to a camera conversion from camera X to camera Y.

[0118] As Figure 7 shown, the output frame 708 can be converted from the frame X 702a to the frame Y 702b. When converting from the first camera mode (e.g., the first camera X) to the second camera mode (e.g., the second camera Y), the first camera 15 can be deactivated, and the second camera 15 can be activated. In this way, the first camera 15 can enter a low-power sleep state, while the second camera 15 switches to a higher power state relative to the low-power sleep state. As described above, the camera conversion between frames from different cameras 15 can occur during the zooming process. For example, when zooming in, the output frame 708 can be converted from a wide-angle lens to a telephoto lens. Similarly, when zooming out, the output frame 708 can be converted from a telephoto lens (e.g., a telephoto camera) to a wide-angle lens (e.g., a wide-angle camera). The camera processor 14 can generate the output frame 708.

[0119] The camera processor 14 can mix the frame M with the frame 1 after the camera conversion. Figure 7Four hybrid frames 706 (for frames 1–N) are shown. The number of hybrid frames after camera transition can be 3 frames, 8 frames, 25 frames, 50 frames, 100 frames, or any other predetermined number of frames. Additionally or alternatively, the hybrid transition can occur within a particular time period (e.g., 0.5 seconds, 1 second, etc.). In such an example, the camera processor 14 can improve efficiency and conserve camera processing, power, or memory resources by not having concurrent frames from both cameras 15 during an extended camera transition and / or by performing transition mixing in response to the camera transition rather than before the camera transition.

[0120] In some examples, the camera processor can determine an adjustment parameter that defines the maximum number of hybrid frames after a camera transition from a first camera 15 to a second camera 15. In some examples, the camera processor 14 can determine the adjustment parameter based on the degree to which the photometric characteristics between the frames obtained from the first camera 15 and the second camera 15 differ from each other. If the degree is small enough (such as when compared to a predetermined threshold), the camera processor 14 can determine that the photometric characteristics between subsequent frames are consistent, or at least only determine a partial mismatch between the photometric characteristics between subsequent frames.

[0121] In an illustrative and non-limiting example, the camera processor 14 can determine that the hue, color, and / or contrast (TCC) difference between a first frame obtained from the first camera 15 and a second frame obtained from the second camera 15 meets a predetermined threshold. In such a case, the predetermined threshold can indicate that the TCC is consistent or at least only indicate a partial mismatch between the photometric characteristics between the frames. In such a case, the camera processor 14 can determine an adjustment parameter that is configured to invoke an instantaneous mixing of one input frame after the camera transition, or signal the IPE 42 to skip the transition mixing of the frames otherwise generated by the camera transition.

[0122] Furthermore, the adjustment parameter can define the number of frames to be mixed after a given camera transition. The adjustment parameter can be a variable that depends on the circumstances of the camera transition (e.g., the difference in photometric quality generated between the frames of a particular camera 15) or can have a value that is constant for transitions between various cameras 15. In one example, the camera processor 14 can employ the same or different adjustment parameters for a camera transition between the first camera 15 and the second camera 15, just as the camera processor 14 can employ the same or the same adjustment parameters for a camera transition between the second camera 15 and the third camera 15, the third camera 15 and the second camera 15, the third camera 15 and the first camera 15, etc.

[0123] In an illustrative example, the camera processor 14 may determine a mismatch between photometric characteristics (e.g., TCC characteristics) that meet a first predetermined threshold for a first camera transition (e.g., a lower mismatch between the TCC characteristics of frames obtained from either camera 15). In such a case, the camera processor 14 may determine and adopt a first adjustment parameter that results in a first number of blended frames after the first camera transition. In another example, the camera processor 14 may determine a mismatch between photometric characteristics that meet a higher second predetermined threshold different from the first predetermined threshold for a second camera transition. In such a case, meeting the second predetermined threshold may indicate a relatively higher mismatch between the photometric characteristics compared to the photometric characteristics of the first camera transition. Thus, the camera processor 14 may determine and adopt a second adjustment parameter that results in a second number of blended frames after the second camera transition. In such a case, the second adjustment parameter may result in a second number of blended frames that include more blended frames compared to the first camera transition (e.g., the first number of blended frames).

[0124] In some cases, after determining the adjustment parameter, the camera processor 14 may determine the blending weight coefficient 33 (e.g., α) based on the adjustment parameter. In one example, when determining an adjustment parameter that will result in a higher number of blended frames (e.g., eight blended frames after a camera transition), the camera processor 14 may determine a relatively low blending weight coefficient 33 (e.g., α equal to 0.2, or 20% of the previous frame and 80% of the current frame). Similarly, when determining an adjustment parameter that will result in a lower number of blended frames (e.g., three blended frames after a camera transition), the camera processor 14 may determine a higher blending weight coefficient 33 (e.g., α equal to 0.5, or 50% of the previous frame and 50% of the current frame).

[0125] According to one or more different techniques of the present disclosure, the camera processor 14 can process the frame M obtained from camera X and the frame 1 obtained from camera Y in a single pass, potentially increasing efficiency compared to techniques that might blend frame M with a previous frame, for example, before a camera transition. That is, the camera processor 14 can obtain each frame once to achieve transition blending and can reduce the amount of time each frame is held or stored in a memory device, such as a buffer memory or on-chip memory. In one example, when blending occurs immediately before a camera transition, the camera processor 14 can be configured to hold and preprocess frame 1 before blending it with frame M, which in turn might result in consuming limited bandwidth, power, load, etc., such that frame 1 can then be blended with frame M. However, as described herein, the camera processor 14 can blend frame 1 with frame M to produce a first blended frame without preprocessing frame 1 for blending and before outputting the first blended frame via output frame 708. When concurrent frame processing is not employed after a camera transition, the camera processor 14 can achieve similar efficiency savings because the camera processor 14 will then be responsible for processing at least one additional frame after the camera transition as part of receiving concurrent frames, which in some cases might result in a perceivable or noticeable pause in the user zoom experience.

[0126] Figure 8 is a conceptual diagram showing an example of recursive transition blending after a camera transition from a first camera 15 to a second camera 15 in accordance with aspects of the techniques described in the present disclosure. Although a particular sequence of operations is described, the techniques of the present disclosure are not limited thereto. Thus, reference Figure 8 to the example described is intended to illustrate an exemplary sequence of operations involving transition blending operations. The conceptual flow from the illustrated first frame (a) to the last frame (h 1 ) shows an exemplary sequence. The blending module 19 can blend the pixels of at least two frames to produce an output frame. Additionally, in some examples, the alignment module 34 can align the frames for blending. That is, increasing the zoom operation might cause the two frames to be misaligned during blending. In such an example, the alignment module 34 can perform a second geometric distortion that can be different from the first geometric distortion performed during a stage before the camera transition.

[0127] As Figure 8As shown, the final geometric distortion frame can enter the alignment module 34. In an example where the second geometric distortion is not implemented, the IPE 42 can route frame (a) to the blending module 19 and bypass the alignment module 34. Additionally, the IPE 42 can output frame (a), as shown in (b). The blending module 19 can obtain frame (c). The blending module 19 can blend the pixels of frame (c) with the pixels of frame (d), which in some cases represent the same pixels of frame (a), but the pixels have a geometric displacement. The blending module 19 can output the blended frame (e 1 ), and can additionally provide frame (e 2 ) to the alignment module 34 to align frame (e 1 ) with the input frame (f). The alignment module 34 can generate frame (g) by aligning frame (e 2 ) used for pixel blending with frame (f), such as by shifting the pixels of frame (e 2 ) to align with the corresponding pixels of frame (f). The blending module 19 can blend the pixels of frame (g) with the input frame (f) to generate the output frame (h 1 ). The blending module 19 can further provide the pixel information of frame (h 1 ), such as in the form of frame (h 2 ), to continue the blending process until the final frame of the blending conversion process is blended.

[0128] Figure 9 is a conceptual diagram showing an exemplary camera conversion 902 according to various aspects of the techniques described in the present disclosure and various stages before and after the camera conversion 902. Although a specific sequence of operations is described, including various stages before and after 902, the techniques of the present disclosure are not limited thereto. Thus, the examples described with reference to Figure 9 are intended to illustrate an exemplary sequence of operations involving conversion blending operations that can occur before and after the camera conversion 902. In the example shown, the first camera 15 can obtain an input frame corresponding to the output frame represented by "W", while the second camera 15 can obtain an input frame corresponding to the output frame represented by "T". Thus, the camera processor 14 can initiate the camera conversion 902 in response to detecting a specific zoom level. In the example shown, the camera processor 14 can perform geometric distortion during the first stage and can perform conversion blending during the second stage, where the two stages are defined by the camera conversion 902. Additionally, the camera processor 14 can separate the two stages. In one illustrative and non-limiting example, the camera processor 14 can perform a pre-geometric distortion stage, followed by a geometric distortion stage, followed by a conversion blending stage, followed by a post-conversion blending process, such as a normal outflow process.

[0129] Figure 10is a conceptual diagram showing an exemplary camera transition 1002 in accordance with aspects of the techniques described in the present disclosure, and various stages before and after the camera transition 1002. Although a particular sequence of operations is described, including the various stages before and after the camera transition 1002, the techniques of the present disclosure are not limited thereto. Thus, reference Figure 10 The example described is intended to illustrate an exemplary sequence of operations involving a pixel blending operation that can occur before and after the camera transition 1002. In the example shown, a first camera 15 can obtain an input frame corresponding to an output frame represented by "X", while a second camera 15 can obtain an input frame corresponding to an output frame represented by "Y". Thus, the camera processor 14 can initiate the camera transition 1002 in response to detecting a particular zoom level.

[0130] The camera processor 14 can obtain a first set of images from the first camera 15 as part of a first optional stage. In some examples, the first optional stage can include a normal operation stage, a geometric distortion stage, a pixel unmerge stage, etc. In examples including the first optional stage, the camera processor 14 can obtain a first frame X M . The camera processor 14 can perform operations in the first stage, including operations on frame X M in anticipation of the camera transition 1002. The camera processor 14 can then initiate the camera transition 1002 that effectively causes a switch to the second camera 15, which obtains an input frame during the operations of the second stage. The operations of the second stage can include recursive blending of a limited set of frames obtained from the second camera 15 after the camera transition 1002, or can include a stage for transforming photometric characteristics based on frames obtained before the camera transition 1002.

[0131] In one example, the camera processor 14 can begin by blending the pixels of the last frame of the stage before the camera transition 1002 with the pixels of the first input frame (Y 1 ) after the camera transition 1002. That is, the camera processor 14 blends the pixels corresponding to the geometric distortion frame X M * with the pixels from the input frame Y 1 (not shown in the figure) to produce an output frame Y 1 '. In a recursive manner, the camera processor 14 can then blend frame Y 1 ' with the input frame Y 2 (not shown in the figure) to produce Y 2 ', and so on, until reaching Y N (not shown in the figure) and Y N’. In some cases, the camera processor 14 may enter the third stage after a limited number of output frames in the second stage. The third stage may include a normal output stage, where the output frames mirror the input frames, similar to the way the output frames in the first stage mirror the input frames in the first stage. In some examples, the third stage may or may not include geometric distortion. Although the geometrically distorted frames have pixels with geometric displacements, the pixels of the geometrically distorted frames can still mirror the pixels of the corresponding input frames, such as in terms of pixel values, photometric characteristics, etc.

[0132] Figure 11 is a conceptual diagram according to various aspects of the techniques described in the present disclosure, which shows exemplary camera transitions and various stages between, before, and after the first camera transition 1102 and the second camera transition 1104. Although specific sequences of operations are described, including the stages between the first camera transition 1102 and the second camera transition 1104, the techniques of the present disclosure are not limited thereto. Thus, the examples described with reference to Figure 11 are intended to illustrate exemplary sequences of operations involving operations that may be before the camera transition 1102, after the camera transition 1102, and before the camera transition 1104. In the example shown, the first camera 15 may include a telephoto (T) camera, and the second camera 15 may include a wide-angle camera (W). Thus, the camera processor 14 may initiate the camera transition 1102 in response to a decrease in the zoom level. Additionally, the camera processor 14 may initiate the camera transition 1104 in response to a decrease or increase in the zoom level.

[0133] The camera processor 14 may obtain a first set of images from the first camera 15. In one example, the camera processor 14 may obtain a first frame T M . The camera processor 14 may perform geometric distortion on the frames representing the first set of images during a first geometric distortion stage that the camera processor 14 performs before the recursive blending transition stage and before the camera transition 1102. The camera processor 14 may then initiate the camera transition 1102, which effectively switches to obtaining input frames from the second camera 15. In such an example, the camera processor 14 may perform recursive blending on a limited set of frames obtained from the second camera 15 after the camera transition 1102. In one example, the camera processor 14 may start by blending the pixels of the last frame in the stage before the camera transition 1102 with the pixels of the first input frame (W 1 ) after the camera transition 1102. That is, the camera processor 14 blends the pixels corresponding to the geometrically distorted frame T M * with the pixels from the input frame W 1 (not shown in the figure) to produce an output frame W 1 ’. In a recursive manner, the camera processor 14 may then continue to blend the frame W 1 ’ with the input frame W2 (not shown in the figure) are mixed to produce W 2 ’, and so on until W N (not shown in the figure) and W N ’ are reached. As described herein, the camera processor 14 can determine the number of mixed frames that result in and include W N ’ based on adjustment parameters, the blending weight coefficients 33, or a combination of adjustment parameters and the blending weight coefficients 33.

[0134] In one non - limiting and illustrative example, the camera processor 14 can perform recursive blending as part of the TCC transformation phase. Once the recursive blending phase is complete, such as the last frame in the recursive blending phase (W N ’), the camera processor 14 can perform normal output stream operations. If reached before the next phase (such as the geometric distortion phase), the normal output stream operations may occur post - recursive blending. That is, in some cases, based on a change in the scaling direction (e.g., a change from zoom in to zoom out) and based on approaching the next camera transformation, the normal output stream operations may not occur. Instead, the camera processor 14 can enter a second geometric distortion phase. The second geometric distortion phase can occur after the normal output phase (where the camera processor 14 has reached the normal output phase) and before the next recursive blending phase. The next recursive blending phase can start when the second camera transformation 1104 is initiated. That is, the camera processor 14 can perform the second geometric distortion phase in anticipation of the second camera transformation 1104 based on the next camera transformation (e.g., returning to the first camera 15 and / or the next camera 15).

[0135] Figure 12 is a timing diagram showing exemplary camera transformation techniques in accordance with various aspects of the techniques described in the present disclosure. Although the timing diagram shows a particular sequence of operations, various modes (e.g., phases) between camera transformations, and particular outputs from the camera processor 14, the techniques of the present disclosure are not limited thereto. Thus, the examples referred to Figure 12 are intended to illustrate exemplary sequences of operations involving pixel blending operations that can occur after a camera transformation.

[0136] In some examples, the camera processor 14 may obtain a first set of images from the first camera 15. In one example, the sensor 12 may transmit a first set of pixels corresponding to the first set of frames to the camera processor 14. In some cases, the sensor 12 may transmit a certain number of pixels corresponding to the total number of available pixels of the respective image sensor 12 (e.g., 12 million pixels from a 12MP (megapixel) image sensor 12) to the camera processor 14. However, in some cases, the sensor 12 may transmit a reduced number of pixels relative to the total number of available pixels from the respective image sensor 12. For example, the image sensor 12 may combine multiple pixels together to form a reduced number of pixels (e.g., pixel binning) and / or may otherwise transmit only a portion of the total number of available pixels to the camera processor 14. In any case, the camera processor 14 may perform recursive pixel blending based on frames obtained prior to a camera transition (such as a transition from the second camera 15 to the first camera 15). The camera processor 14 may blend the pixels of each subsequent frame (e.g., the pixels received from the sensor 12) in a recursive manner until a change in the zoom direction (e.g., from zoom in to zoom out) occurs or until normal flow operation is initiated at the end of a transitional blending phase.

[0137] Next, the camera processor 14 may obtain a second set of images from the first camera 15. The camera processor 14 may determine that a camera transition from the first camera 15 to the second camera 15 may occur based on the zoom level, e.g., if a particular zoom level continues to increase in a particular direction and / or at a particular rate and reaches a camera transition threshold. Accordingly, the camera processor 14 may perform geometric distortion on the frames representing the second set of images.

[0138] Next, the camera processor 14 may obtain a third set of images from the second camera 15, such as in response to a camera transition from the first camera 15 to the second camera 15. The camera processor 14 may perform recursive blending on the frames representing the third set of images. In one example, the camera processor 14 may utilize the first frame from the second set of images to initiate a recursive blending function such that the pixel contributions from the first frame surge in the recursively blended frames.

[0139] Next, the camera processor 14 can determine a trigger event that is configured to signal the camera processor 14 to stop recursively blending frames, such as the frames represented by the third set of images in this example. Thus, the camera processor 14 can receive a fourth set of images from the second camera 15 and can initiate a normal output stream that outputs frames representing the fourth set of images that were not blended as part of the transform blend as part of the normal output stream. In each case, the camera processor 14 can output the first set of images through the fourth set of images to one or more output destinations (e.g., video encoder / decoder 17, display interface 26, system memory 30, etc.). The camera processor 14 can output the first set of images through the fourth set of images as a set of blended and unblended frames with varying degrees of geometric distortion prior to the camera transform.

[0140] Those skilled in the art will understand that image capture can include snapshot capture, capture for preview display, or video stream capture. For example, one or more cameras 15 of the computing device 10 can perform the blending operations described with reference to when decoding video, taking a still photo after a zoom operation, or when recording video (e.g., picture-in-video), when providing an image preview (e.g., before the user takes a photo or video, while the user is taking a video or photo, etc.), when capturing a motion photo, when the user is taking a video, when storing the video to the system memory 30, etc. Figures 1 to 12 the blending operations.

[0141] Exemplary examples of the present disclosure include:

[0142] Example 1 An apparatus configured for camera processing, the apparatus including: a memory; and one or more processors communicatively coupled to the memory, the one or more processors being configured to: in response to a camera transition from a first camera to a second camera, blend pixels of a first frame with pixels of a second frame to produce a first blended frame, the first frame being captured via the first camera and the second frame being captured via the second camera in sequence with the first frame; and output the first blended frame after the first frame.

[0143] Example 2 The apparatus according to Example 1, wherein to blend the pixels of the first frame with the pixels of the second frame, the one or more processors are configured to: blend a first pixel of the first frame with a corresponding first pixel of the second frame to produce a first blended frame, wherein the first blended frame includes blended pixels that include a first contribution level of the first pixel from the first frame and a second corresponding contribution level of the first pixel from the second frame.

[0144] Example 3. The apparatus according to any one of Examples 1 or 2, wherein one or more processors are further configured to: obtain a third frame after the second frame; mix pixels of the first mixed frame with pixels of the third frame to generate a second mixed frame; and output the second mixed frame continuously with the first mixed frame.

[0145] Example 4. The apparatus according to Example 3, wherein in order to mix pixels of the first mixed frame with pixels of the third frame, one or more processors are configured to: mix a first pixel of the first mixed frame with a corresponding first pixel of the third frame to generate a second mixed frame.

[0146] Example 5. The apparatus according to any one of Examples 3 or 4, wherein one or more processors are further configured to: align the first mixed frame with the third frame before mixing pixels of the first mixed frame with pixels of the third frame.

[0147] Example 6. The apparatus according to Example 5, wherein in order to align the first mixed frame with the third frame, one or more processors are further configured to: warp the first mixed frame to align with corresponding pixels of the third frame.

[0148] Example 7. The apparatus according to any one or more of Examples 1 to 6, wherein one or more processors are further configured to: obtain a first frame via a first lens; output the first frame before a camera transition; obtain a second frame via a second lens; and align the first frame with the second frame before mixing pixels of the first frame with pixels of the second frame.

[0149] Example 8. The apparatus according to Example 7, wherein in order to output the first frame, one or more processors are configured to: cause geometric distortion of the first frame according to a first distortion period that lasts until the camera transition; and output the first frame including the geometric distortion.

[0150] Example 9. The apparatus according to Example 8, wherein a mixed frame generated after the camera transition includes a field of view different from a field of view of an unmixed frame obtained during the first distortion period.

[0151] Example 10. The apparatus according to any one or more of Examples 1 to 9, wherein in order to mix pixels of the first frame with pixels of the second frame, one or more processors are configured to: apply a mixing function for determining a pixel contribution level as follows: Y 1 ’ = α(X M *)+(1–α)Y 1 , where: Y 1 ’ represents the first mixed frame, X M * represents the first frame with geometric distortion, Y 1 represents the second frame, and α represents a mixing weight as follows: 0 ≤ α ≤ 1.

[0152] Example 11. The apparatus according to Example 10, wherein X M * includes a geometrically distorted frame captured by a first camera before a camera transition, wherein the camera transition initiates the application of a blending function that causes blending of pixels from two input frames to produce a blended output frame.

[0153] Example 12. The apparatus according to any one of Examples 10 or 11, wherein the one or more processors are further configured to: blend pixels of a first blended frame with pixels of a plurality of consecutive frames based on the following recursive blending function: Y 2 ’ = α(Y 1 ’)+(1–α)Y 2 where: Y 2 ’ represents a second blended frame, Y 1 ’ represents the first blended frame, and Y 2 represents an incoming frame (e.g., a third frame).

[0154] Example 13. The apparatus according to any one or more of Examples 10 to 12, wherein the one or more processors are further configured to: determine, automatically or based on a manual input or both, a value to assign to a blending weight.

[0155] Example 14. The apparatus according to any one or more of Examples 1 to 13, wherein the one or more processors are further configured to: obtain the nth frame after a second frame; and output the nth frame as a final blended frame of a blending conversion process that is initiated with the blending of the first frame and the second frame, and wherein the final blended frame includes a lower pixel contribution from the first frame relative to the blended output frames prior to the final blended frame in the blending conversion process.

[0156] Example 15. The apparatus according to Example 14, wherein the lower pixel contribution includes a cascaded pixel contribution that cascades pixel contributions throughout consecutive blended frames in the blending conversion process.

[0157] Example 16. The apparatus according to any one of Examples 14 or 15, wherein, in order to output the nth frame as a final blended frame of a blending conversion process, the one or more processors are configured to: determine that a predetermined number of frames have been obtained or that a predetermined duration has elapsed since the blending conversion process was initiated, or determine that the lower pixel contribution from the first frame includes a value that meets a predetermined pixel contribution threshold.

[0158] Example 17. The apparatus according to any one or more of Examples 14 to 16, wherein the one or more processors are further configured to: obtain a frame after the nth frame; and output the frame after the nth frame as an unblended frame according to a camera mode that is initiated after the blending of the nth frame.

[0159] Example 18 The apparatus according to any one or more of Examples 1 to 17, wherein one or more processors are further configured to: perform a global 3A synchronization process on each of a plurality of frames, the plurality of frames including a first frame and a first mixed frame, wherein mixing the plurality of frames includes pixel-level mixing at a local pixel level for each pair of frames to be mixed with each other among the plurality of frames.

[0160] Example 19 The apparatus according to any one or more of Examples 1 to 18, wherein, in order to output the first mixed frame, one or more processors are configured to: output the first frame and the first mixed frame to a video encoder / decoder.

[0161] Example 20 The apparatus according to any one or more of Examples 1 to 19, wherein, in order to output the first mixed frame, one or more processors are configured to: output the first mixed frame to a system memory or a display device.

[0162] Example 21 The apparatus according to any one or more of Examples 1 to 20, wherein one or more processors are further configured to: detect a change in a zoom direction after a camera switch; and in response to detecting the change, output a plurality of subsequent frames according to a second camera process, the second camera process being separate and different from the mixing initiated at the camera switch.

[0163] Example 22 The apparatus according to any one or more of Examples 1 to 21, wherein the first camera includes a first field of view, and wherein the second camera includes a second field of view disposed within the first field of view.

[0164] Example 23 The apparatus according to any one or more of Examples 1 to 22, wherein the first camera includes a first focal length, and wherein the second camera includes a second focal length.

[0165] Example 24 The apparatus according to any one or more of Examples 1 to 23, wherein the camera switch includes a lens switch or an image sensor switch.

[0166] Example 25 The apparatus according to any one or more of Examples 1 to 24, wherein one or more processors are further configured to: cause a camera switch by deactivating the first camera before mixing the second frame with the first frame.

[0167] Example 26 A method for camera processing, the method including: in response to a camera switch from a first camera to a second camera, mixing pixels of a first frame with pixels of a second frame to produce a first mixed frame, the first frame being captured via the first camera, and the second frame being captured continuously with the first frame via the second camera; and outputting the first mixed frame after the first frame.

[0168] Example 27. The method according to Example 26, wherein mixing the pixels of the first frame with the pixels of the second frame includes: mixing a first pixel of the first frame with a corresponding first pixel of the second frame to generate a first mixed frame, wherein the first mixed frame includes mixed pixels, and the mixed pixels include a first contribution level of the first pixel from the first frame and a second corresponding contribution level of the first pixel from the second frame.

[0169] Example 28. The method according to any one of Examples 26 or 27, further comprising: obtaining a third frame after the second frame via a second camera; mixing the pixels of the first mixed frame with the pixels of the third frame to generate a second mixed frame; and outputting the second mixed frame continuously with the first mixed frame.

[0170] Example 29. The method according to Example 28, wherein mixing the pixels of the first mixed frame with the pixels of the third frame includes: mixing a first pixel of the first mixed frame with a corresponding first pixel of the third frame to generate a second mixed frame.

[0171] Example 30. The method according to any one of Examples 28 or 29, further comprising: aligning the first mixed frame with the third frame before mixing the pixels of the first mixed frame with the pixels of the third frame.

[0172] Example 31. The method according to Example 30, wherein aligning the first mixed frame with the third frame includes: warping the first mixed frame to align with the corresponding pixels of the third frame.

[0173] Example 32. The method according to any one or more of Examples 26 to 31, further comprising: obtaining a first frame via a first camera; outputting the first frame before the camera conversion; obtaining a second frame via a second camera; and aligning the first frame with the second frame before mixing the pixels of the first frame with the pixels of the second frame.

[0174] Example 33. The method according to Example 32, wherein outputting the first frame further includes: causing a geometric distortion of the first frame according to a first distortion mode, and the first distortion mode continues until the camera conversion; and outputting the first frame including the geometric distortion.

[0175] Example 34. The method according to Example 33, wherein the mixed frame generated after the camera conversion includes a field of view different from the field of view of the unmixed frame obtained via the first camera during the first distortion mode.

[0176] Example 35. The method according to any one or more of Examples 26 to 34, wherein mixing the pixels of the first frame with the pixels of the second frame includes: applying a mixing function for determining pixel contribution levels: Y 1 ’ = α(X M *)+(1–α)Y 1 where: Y 1' represents the first mixed frame, X M * represents the first frame with geometric distortion, Y 1 represents the second frame, and α represents the following mixing weight: 0 ≤ α ≤ 1.

[0177] Example 36 The method according to Example 35, wherein X M * includes the geometrically distorted frame captured by the first camera before the camera transformation, wherein the camera transformation initiates the application of a mixing function that causes the mixing of pixels from two input frames to produce a single mixed output frame.

[0178] Example 37 The method according to any one of Examples 35 or 36, further comprising: mixing the first mixed frame with successive frames obtained via a second camera by applying a recursive mixing function as follows: Y 2 ' = α(Y 1 ')+(1–α)Y 2 , wherein: Y 2 ' represents the second mixed frame, Y 1 ' represents the first mixed frame, and Y 2 represents the first successive frame obtained via the second camera, the first successive frame being after the second frame.

[0179] Example 38 The method according to any one or more of Examples 35 to 37, further comprising: determining the value of the mixing weight.

[0180] Example 39 The method according to any one or more of Examples 26 to 38, further comprising: obtaining the nth frame after the second frame; and outputting the nth frame as the final mixed frame of the mixing transformation process, wherein the mixing transformation process is initiated with the mixing of the first frame and the second frame, and wherein the final mixed frame includes a lower pixel contribution from the pixels of the first frame relative to the subsequent frames of the mixing transformation process.

[0181] Example 40 The method according to Example 39, wherein the lower pixel contribution includes cascaded pixel contributions cascaded through the mixing transformation process.

[0182] Example 41 The method according to any one of Examples 39 or 40, wherein outputting the nth frame as the final mixed frame of the mixing transformation process includes: determining that a predetermined number of frames have been obtained or a predetermined duration has elapsed since the mixing transformation process was initiated, or determining that the lower pixel contribution from the pixels of the first frame includes a value that meets a predetermined pixel contribution threshold.

[0183] Example 42 The method according to any one or more of Examples 39 to 41, further comprising: obtaining the (n + 1)th frame via the second camera; and outputting the (n + 1)th frame as an unmixed frame consecutive to the final mixed frame according to the camera mode initiated after the mixing of the nth frame.

[0184] Example 43 The method according to any one or more of Examples 26 to 42 further comprises: performing a global 3A synchronization process on each of a plurality of frames, the plurality of frames including a first frame and a first mixed frame, wherein mixing the plurality of frames comprises pixel-level mixing at a local pixel level for each pair of frames to be mixed with each other among the plurality of frames.

[0185] Example 44 The method according to any one or more of Examples 26 to 43, wherein outputting the first mixed frame comprises: outputting the first frame and the first mixed frame to a video encoder / decoder.

[0186] Example 45 The method according to any one or more of Examples 26 to 44, wherein by geometric distortion, the first frame is output for display as a first output frame, and after the output of the first frame, the first mixed frame is output for display as a second output frame consecutive to the first output frame.

[0187] Example 46 The method according to any one or more of Examples 26 to 45 further comprises: detecting a change in the zoom direction after a camera transition; and outputting a plurality of subsequent frames according to a second camera process, the second camera process being separate and different from frame mixing.

[0188] Example 47 The method according to any one or more of Examples 26 to 46, wherein the first camera includes a first field of view, and wherein the second camera includes a second field of view disposed within the first field of view.

[0189] Example 48 The method according to any one or more of Examples 26 to 47, wherein the first camera includes a first focal length, and wherein the second camera includes a second focal length.

[0190] Example 49 The method according to any one or more of Examples 26 to 48, wherein the camera transition includes a transition from a first lens to a second lens.

[0191] Example 50 The method according to any one or more of Examples 26 to 49 further comprises: causing a camera transition by deactivating the first camera.

[0192] Example 51 An apparatus configured for camera processing, the apparatus comprising: means for mixing a first set of pixels of a first frame with a second set of pixels of a second frame to produce a first mixed frame in response to a transition from a first camera to a second camera, the first frame being captured via the first camera and the second frame being captured continuously with the first frame via the second camera; and means for outputting the first mixed frame after the first frame.

[0193] Example 52 The apparatus according to Example 51, wherein the component for mixing the first set of pixels with the second set of pixels comprises: a component for mixing a first pixel of the first set of pixels with a corresponding first pixel of the second set of pixels.

[0194] Example 53 The apparatus according to any one of Examples 51 or 52, further comprising: a component for obtaining a third frame; and a component for mixing the first mixed frame with the third frame to produce a second mixed frame.

[0195] Example 54 The apparatus according to Example 53, wherein the component for mixing the first mixed frame with the third frame comprises: a component for mixing the pixels of the first mixed frame with the corresponding pixels of the third frame.

[0196] Example 55 The apparatus according to any one of Examples 53 or 54, further comprising: a component for aligning the first mixed frame and the third frame with each other.

[0197] Example 56 The apparatus according to Example 55, wherein the component for aligning the first mixed frame and the third frame comprises: a component for warping the first mixed frame.

[0198] Example 57 The apparatus according to any one or more of Examples 51 to 56, further comprising: a component for obtaining a first frame from a first camera; a component for outputting the first frame; a component for obtaining a second frame; and a component for aligning the first frame and the second frame.

[0199] Example 58 The apparatus according to Example 57, wherein the component for outputting the first frame comprises: a component for outputting the first frame including geometric warping.

[0200] Example 59 The apparatus according to Examples 51 to 58, wherein the mixed frame generated after the camera conversion includes a different field of view compared to the unmixed frame obtained before the camera conversion.

[0201] Example 60 The apparatus according to any one or more of Examples 51 to 59, wherein the component for mixing the first set of pixels with the second set of pixels comprises: a component for applying a mixing function that determines the pixel contribution level based on a mixing weight and a corresponding inverse mixing weight, the mixing weight and the corresponding inverse mixing weight summing to the maximum pixel contribution of a third set of pixels of the first mixed frame.

[0202] Example 61 The apparatus according to Example 60, further comprising: a component for adjusting the mixing weight.

[0203] Example 62 The apparatus according to any one or more of Examples 51 to 61, wherein the first frame includes a geometric warping frame, and wherein the camera conversion initiates the mixing of the first frame and the second frame.

[0204] Example 63 The apparatus according to any one or more of Examples 51 to 62 further comprises: means for mixing a first mixed frame with successive frames obtained from a second camera via a recursive mixing function, wherein the first mixed frame and a first successive frame (e.g., a third frame) comprising pixel contributions from the first and second frames are mixed together to produce a second mixed frame, and wherein the second mixed frame is mixed with the next successive frame to produce a third mixed frame.

[0205] Example 64 The apparatus according to any one or more of Examples 51 to 63 further comprises: means for obtaining the nth frame after the second frame; and means for outputting the nth frame as the final mixed frame of the mixing conversion process.

[0206] Example 65 The apparatus according to any one or more of Examples 51 to 64 further comprises: means for cascading pixel contributions across successive mixed frames.

[0207] Example 66 The apparatus according to any one of Examples 64 or 65, wherein the means for outputting the nth frame as the final mixed frame comprises: means for determining that a predetermined number of frames have been obtained or that a predetermined duration has elapsed since the start of the mixing conversion process, or means for determining that the contribution of the first frame has been diluted to a predetermined contribution level since the start of the mixing conversion process.

[0208] Example 67 The apparatus according to any one or more of Examples 64 to 66 further comprises: means for obtaining the n + 1th frame; and means for outputting the n + 1th frame as an unmixed frame consecutive to the final mixed frame.

[0209] Example 68 The apparatus according to any one or more of Examples 51 to 67 further comprises: means for performing a global 3A synchronization process on each of a plurality of frames, the plurality of frames including the first frame and the first mixed frame; and means for mixing the pixels of the plurality of frames at a local pixel level.

[0210] Example 69 The apparatus according to any one or more of Examples 51 to 68, wherein the means for outputting the first mixed frame comprises: means for outputting the first frame and the first mixed frame to a video encoder / decoder.

[0211] Example 70 The apparatus according to any one or more of Examples 51 to 69, wherein the means for outputting the first mixed frame comprises: means for outputting the first mixed frame to a system memory or a display device.

[0212] Example 71. The apparatus according to any one or more of Examples 51 to 70, further comprising: a component for detecting a change in the zoom direction after a camera switch; and a component for outputting a plurality of subsequent frames according to a second camera process in response to detecting the change.

[0213] Example 72. The apparatus according to any one or more of Examples 51 to 71, wherein the first camera includes a first field of view, and wherein the second camera includes a second field of view disposed within the first field of view.

[0214] Example 73. The apparatus according to any one or more of Examples 51 to 72, further comprising: a component for converting from a first focal length to a second focal length.

[0215] Example 74. The apparatus according to any one or more of Examples 51 to 73, wherein the first camera includes a first focal length, and wherein the second camera includes a second focal length.

[0216] Example 75. The apparatus according to any one or more of Examples 51 to 73, further comprising: a component for concurrently deactivating the first camera while activating the second camera or a component for activating the second camera before deactivating the first camera.

[0217] Example 76: A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: in response to a camera switch from a first camera to a second camera, mix the pixels of a first frame with the pixels of a second frame to produce a first mixed frame, the first frame being captured via the first camera and the second frame being captured via the second camera; obtain a third frame captured via the second camera; and mix the pixels of the first mixed frame with the pixels of the third frame to produce a second mixed frame.

[0218] Example 77. The non-transitory computer-readable storage medium according to Example 76, wherein in order to mix the pixels of the first frame with the pixels of the second frame, the one or more processors: mix a first pixel of the first frame with a corresponding first pixel of the second frame to produce a first mixed frame, wherein the first mixed frame includes mixed pixels that include a first contribution level of the first pixel from the first frame and a second corresponding contribution level of the first pixel from the second frame.

[0219] Example 78. The non-transitory computer-readable storage medium according to any one of Examples 76 or 77, wherein the one or more processors are further caused to: output the first mixed frame; and output the second mixed frame after the first mixed frame.

[0220] Example 79. A non-transitory computer-readable storage medium according to any one or more of claims 76 to 78, wherein, to blend pixels of a first blended frame with pixels of a third frame, one or more processors: for each pixel of a first plurality of pixels of the first blended frame and each corresponding pixel of a corresponding second plurality of pixels of the third frame, blend the first plurality of pixels and the corresponding second plurality of pixels.

[0221] Example 80. A non-transitory computer-readable storage medium according to any one or more of examples 76 to 79, wherein the one or more processors are further caused to: align the first blended frame with the third frame before blending the pixels of the first blended frame with the pixels of the third frame.

[0222] Example 81. A non-transitory computer-readable storage medium according to example 80, wherein, to align the first blended frame with the third frame, one or more processors: warp the first blended frame to align with corresponding pixels of the third frame.

[0223] Example 82. A non-transitory computer-readable storage medium according to any one or more of examples 76 to 81, wherein the one or more processors are further caused to: obtain a first frame via a first image sensor; output the first frame before a camera transition; obtain a second frame via a second image sensor; and align the first frame with the second frame before blending the pixels of the first frame with the pixels of the second frame.

[0224] Example 83. A non-transitory computer-readable storage medium according to example 82, wherein, to output the first frame, one or more processors: output the first frame as a geometrically warped frame.

[0225] Example 84. A non-transitory computer-readable storage medium according to any one or more of examples 76 to 83, wherein a blended frame generated after a camera transition includes a field of view different from a field of view of a previously unblended frame.

[0226] Example 85. A non-transitory computer-readable storage medium according to any one or more of examples 76 to 84, wherein, to blend the pixels of the first frame with the pixels of the second frame, one or more processors: apply a blending function that determines a pixel contribution level: Y 1 ’ = α(X M *)+(1–α)Y 1 , where: Y 1 ’ represents the first blended frame, X M * represents the first frame with geometric warping, Y 1 represents the second frame, and α represents a blending weight: 0 ≤ α ≤ 1.

[0227] Example 86. The non - transitory computer - readable storage medium according to Example 85, wherein, in order to apply the blending function, one or more processors: initiate the blending function in response to a camera transition.

[0228] Example 87. The non - transitory computer - readable storage medium according to any one of Examples 85 or 86, wherein further one or more processors: blend a third frame with a first blended frame to obtain a second blended frame as follows: Y 2 ’ = α(Y 1 ’)+(1–α)Y 2 where: Y 2 ’ represents the second blended frame, Y 1 ’ represents the first blended frame, Y 2 represents the third frame, and α represents the blending weight.

[0229] Example 88. The non - transitory computer - readable storage medium according to any one or more of Examples 85 to 87, wherein further one or more processors: automatically determine the value of the blending weight.

[0230] Example 89. The non - transitory computer - readable storage medium according to any one or more of Examples 76 to 88, wherein further one or more processors: obtain the nth frame after the second frame; and output the nth frame as the final blended frame.

[0231] Example 90. The non - transitory computer - readable storage medium according to Example 89, wherein the output of the nth frame includes the last output of the recursive blending mode before the next camera mode that is separated from and different from the recursive blending mode.

[0232] Example 91. The non - transitory computer - readable storage medium according to any one of Examples 89 or 90, wherein, in order to output the nth frame as the final blended frame, one or more processors: determine that a predetermined number of blended frames have been obtained or determine that a predetermined duration has elapsed since the camera transition, or determine that the contribution amount of the first frame to the nth frame satisfies a predetermined contribution threshold.

[0233] Example 92. The non - transitory computer - readable storage medium according to any one or more of Examples 89 to 91, wherein further one or more processors: obtain the (n + 1)th frame; and output the (n + 1)th frame as the unblended frame after the final blended frame.

[0234] Example 93. The non - transitory computer - readable storage medium according to any one or more of Examples 76 to 92, wherein further one or more processors: perform a global 3A synchronization process for each of a plurality of frames, the plurality of frames including the first frame and the first blended frame.

[0235] Example 94. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 93, wherein, in order to output a first mixed frame, one or more processors output the first frame and the first mixed frame to a video encoder / decoder.

[0236] Example 95. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 94, wherein, in order to output a first mixed frame, one or more processors output the first frame and the first mixed frame to a display device.

[0237] Example 96. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 95, wherein the one or more processors are further caused to: detect a change in a scaling direction after obtaining a second frame; and stop generating a mixed frame based on any previously generated mixed frame.

[0238] Example 97. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 96, wherein the first camera includes a first field of view and the second camera includes a second field of view.

[0239] Example 98. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 97, wherein the first camera includes a first focal length and the second camera includes a second focal length.

[0240] Example 99. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 98, wherein the camera conversion includes an image sensor conversion.

[0241] Example 100. The non-transitory computer-readable storage medium according to any one or more of Examples 76 to 99, wherein the one or more processors are further caused to: start the second camera before capturing the first frame.

[0242] It should be recognized that, according to examples, certain actions or events of any of the techniques described herein may be performed in a different order, may be added, combined, or completely omitted (e.g., not all described actions or events are necessary for the practice of the technique). Additionally, in certain examples, the actions or events may be performed concurrently, such as by multithreading, interrupt processing, or multiple processors, rather than sequentially.

[0243] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored as one or more instructions or code on a computer-readable medium or transmitted through a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium. In this way, a computer-readable medium generally may correspond to a tangible computer-readable storage medium (which is non-transitory). A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0244] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, cache memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. It should be understood that the computer-readable storage media and data storage media do not include carrier waves, signals, or other transitory media, but rather refer to non-transitory, tangible storage media. As used herein, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc, where disks typically reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0245] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, as used herein, the term "processor" may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Further, these techniques may be fully implemented in one or more circuits or logic elements.

[0246] The techniques of the present disclosure may be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs) or a set of ICs (e.g., a chipset). Various components, modules or units are described in the present disclosure to emphasize the functional aspects of the devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units may be combined in any codec hardware unit or provided by a collection of interoperating hardware units including one or more processors as described above in combination with suitable software and / or firmware.

[0247] Various examples have been described. These examples, as well as other examples, are within the scope of the following claims.

Claims

1. An apparatus configured for camera processing, the apparatus comprising: a memory; and one or more processors in communication with the memory, the one or more processors being configured to: receive a signal to transition from a first mode to a second mode, wherein a first frame is received via a first camera of the apparatus when the apparatus operates in the first mode, and wherein a second frame is received via a second camera of the apparatus when the apparatus operates in the second mode; receive the first frame via the first camera when the apparatus operates in the first mode; output the first frame; transition from the first mode to the second mode after outputting the first frame; receive the second frame via the second camera when the apparatus operates in the second mode; mix one or more pixels of the first frame with one or more pixels of the second frame after the transition to produce a first mixed frame; output the first mixed frame after the first frame; when the apparatus operates in the second mode, receive a third frame from the second camera after the second frame; mix one or more pixels of the first mixed frame with one or more pixels of the third frame to produce a second mixed frame; and output the second mixed frame after the first mixed frame.

2. The apparatus according to claim 1, wherein in order to mix the pixels of the first frame with the pixels of the second frame, the one or more processors are configured to: mix a first pixel among the one or more pixels of the first frame with a corresponding first pixel among the one or more pixels of the second frame to produce one or more mixed pixels of the first mixed frame.

3. The apparatus according to claim 1, wherein the one or more processors are further configured to: warp the first frame, wherein warping the first frame further provides spatial alignment between the one or more pixels of the first frame and the one or more pixels of the second frame.

4. The apparatus according to claim 1, wherein in order to mix the one or more pixels of the first frame with the one or more pixels of the second frame, the one or more processors are configured to: apply a mixing function that determines a pixel contribution level as follows: Y 1 ’ = α(X M *) + (1 – α)Y 1 , where ： Y 1 ' represents the first mixed frame, X M * indicates the first frame having a geometric distortion, Y 1 represents the second frame, and α represents a mixing weight.

5. The apparatus according to claim 4, wherein XM* includes a geometrically warped frame, and the first camera receives the geometrically warped frame before the second mode after activation of the signal.

6. The apparatus according to claim 4, wherein the one or more processors are further configured to: mix one or more pixels of the first mixed frame with pixels of a plurality of consecutive frames based on the following recursive mixing function: Y 2 ’ = α(Y 1 ’)+(1 – α)Y 2 , where Y 2 ' represents the second mixed frame, Y 1 ' represents the first mixed frame, and Y 2 represents the third frame.

7. The apparatus according to claim 1, wherein the one or more processors are further configured to: receive an nth frame from the second camera, the nth frame being received after the second frame; determine that (i) a predetermined number of frames have been obtained, or (ii) a predetermined duration has elapsed, relative to the transition from the first mode to the second mode; and Output the nth frame as the final blended frame of the blending conversion process, where the blending conversion process starts with the blending of the first frame and the second frame.

8. The apparatus according to claim 1, wherein the first frame represents image data captured at a first zoom level, and wherein the second frame represents image data captured at a second zoom level, where the first zoom level is different from the second zoom level.

9. The apparatus according to claim 1, wherein the first camera is configured to image a scene at a first effective focal length, and wherein the second camera is configured to image the scene at a second effective focal length, where the first effective focal length of the first camera is different from the second effective focal length of the second camera.

10. The apparatus according to claim 1, wherein the one or more processors are further configured to: Before blending the one or more pixels of the first frame with the one or more pixels of the second frame, switch from the first mode to the second mode by changing the power state of the first camera.

11. A method for camera processing, the method comprises: Receiving, by one or more processors, a signal to switch from a first mode to a second mode, where a first frame is received via a first camera configured to operate in the first mode, and where a second frame is received via a second camera configured to operate in the second mode; Receiving, by one or more processors, the first frame via the first camera when operating in the first mode; Outputting, by one or more processors, the first frame; Switching, by one or more processors, from the first mode to the second mode after outputting the first frame; Receiving, by one or more processors, the second frame via the second camera when operating in the second mode; Blending, by one or more processors, one or more pixels of the first frame with one or more pixels of the second frame after switching from the first mode to the second mode to produce one or more blended pixels of a first blended frame; Outputting, by one or more processors, the first blended frame after the first frame; Receiving, by one or more processors, a third frame from the second camera after the second frame when operating in the second mode; Blending, by one or more processors, one or more pixels of the first blended frame with one or more pixels of the third frame to produce a second blended frame; and Outputting, by one or more processors, the second blended frame after the first blended frame.

12. The method according to claim 11, wherein blending the one or more pixels of the first frame with the one or more pixels of the second frame comprises: Blending, by one or more processors, a first pixel among the one or more pixels of the first frame with a corresponding first pixel among the one or more pixels of the second frame to produce a first blended pixel among the one or more blended pixels of the first blended frame.

13. The method according to claim 11, wherein one or more processors mix the one or more pixels of the first mixed frame with the one or more pixels of the third frame comprising: one or more processors mix a first pixel of the one or more mixed pixels of the first mixed frame with a corresponding first pixel of the one or more pixels of the third frame to generate a first mixed pixel of the second mixed frame.

14. The method according to claim 11, further comprising: wherein the first frame represents image data received at a first zoom level, and wherein the second frame represents image data received at a second zoom level, wherein the first zoom level is different from the second zoom level.

15. The method according to claim 11, wherein one or more processors mix the one or more pixels of the first frame with the one or more pixels of the second frame comprising: one or more processors apply a mixing function for determining a pixel contribution level as follows: Y 1 ’ = α(X M *) + (1 – α)Y 1 , where: Y 1 ' represents the first mixed frame, X M * indicates the first frame with geometric distortion, Y 1 represents the second frame, and α represents a mixing weight.

16. The method according to claim 15, wherein X M * includes a geometrically distorted frame generated by the first camera before transitioning from the first mode to the second mode, wherein the transition from the first mode to the second mode initiates the application of the blending function, the blending function causing pixels from two input frames to be blended to produce a single blended output frame.

17. The method according to claim 15, further comprising: mix the first mixed frame with the third frame obtained via the second camera by applying a recursive mixing function as follows: Y 2 ’ = α(Y 1 ’)+(1 – α)Y 2 , where: Y 2 ' represents the second mixed frame, Y 1 ' represents the first mixed frame, and Y 2 represents the third frame.

18. The method according to claim 11, further comprising: one or more processors receive an nth frame from the second camera, the nth frame being received after the second frame; one or more processors determine that (i) a predetermined number of frames have been obtained or (ii) a predetermined duration has elapsed with respect to the transition from the first mode to the second mode; and one or more processors output the nth frame as the final mixed frame of the mixed transition process, wherein the final mixed frame includes a lower pixel contribution from the pixels of the first frame with respect to subsequent frames of the mixed transition process.

19. The method according to claim 11, wherein the first frame represents image data captured at a first zoom level, and wherein the second frame represents image data captured at a second zoom level, wherein the first zoom level is different from the second zoom level.

20. An apparatus configured for camera processing, the apparatus comprising: means for determining a transition from a first camera mode to a second camera mode, wherein the first camera mode includes a mode in which a first frame is received via a first camera, and wherein the second camera mode includes a mode in which a second frame is received via a second camera; means for receiving the first frame via the first camera when operating in the first camera mode; means for outputting the first frame; means for transitioning from the first camera mode to the second camera mode after outputting the first frame; means for receiving the second frame via the second camera when operating in the second camera mode; A component for mixing a first set of pixels of the first frame with a second set of pixels of the second frame to generate a first mixed frame after converting from the first camera mode to the second camera mode; A component for outputting the first mixed frame after the first frame; A component for receiving a third frame from the second camera after the second frame when operating in the second camera mode; A component for mixing one or more pixels of the first mixed frame with one or more pixels of the third frame to generate a second mixed frame; And A component for outputting the second mixed frame after the first mixed frame.

21. The apparatus according to claim 20, wherein the component for mixing the first set of pixels with the second set of pixels Comprises: A component for mixing a first pixel of the first set of pixels with a corresponding first pixel of the second set of pixels.

22. The apparatus according to claim 20, further Comprises: A component for mixing the first mixed frame with consecutive frames received from the second camera.

23. A non-transitory computer-readable storage medium storing instructions that, when executed, cause one or more processors to: Receive a signal indicating a transition from a first camera mode to a second camera mode, wherein the one or more processors receive a first frame when operating in the first camera mode and wherein the one or more processors receive a second frame when operating in the second camera mode, and wherein the first camera mode and the second camera mode are different from each other; Receive the first frame via the first camera when the one or more processors are operating in the first camera mode; Output the first frame; Transition from the first camera mode to the second camera mode after outputting the first frame; Receive the second frame via the second camera when the one or more processors are operating in the second camera mode; After the transition, mix one or more pixels of the first frame with one or more pixels of the second frame to generate a first mixed frame including one or more mixed pixels; Output the first mixed frame after the first frame; When the one or more processors are operating in the second camera mode, receive a third frame from the second camera after the second frame; Mix the one or more mixed pixels of the first mixed frame with one or more pixels of the third frame to generate a second mixed frame; And Output the second mixed frame after the first mixed frame.

24. The non-transitory computer-readable storage medium according to claim 23, wherein in order to mix the one or more pixels of the first frame with the one or more pixels of the second frame, the one or more processors are caused to: Mix a first pixel among the one or more pixels of the first frame with a corresponding first pixel among the one or more pixels of the second frame to generate the one or more mixed pixels of the first mixed frame.

25. The non-transitory computer-readable storage medium according to claim 23, wherein, in order to blend the one or more pixels of the first frame with the one or more pixels of the second frame, the one or more processors: Apply a blending function that determines a pixel contribution level as follows: Y 1 ’ = α(X M *) + (1 – α)Y 1 , Where ： Y 1 ' represents the first mixed frame, X M * indicates the first frame having geometric distortion, Y 1 represents the second frame, and α represents a blending weight.

26. The non-transitory computer-readable storage medium according to claim 23, wherein the one or more processors are further caused to: Receive an nth frame from the second camera, the nth frame being received after the second frame; and Output the nth frame as the final blended frame in the second camera mode.

27. The non-transitory computer-readable storage medium according to claim 26, wherein the one or more processors are further caused to: Start the second camera before receiving the first frame by changing the power state of the second camera during the first camera mode to transfer power usage from the first camera to the second camera so that the second camera serves as the main camera in the second camera mode during the second camera mode.

Images