Fusing optical zoom images to digital zoom images

Abstract

This document describes systems and techniques aimed at fusing an optical zoom image into a digital zoom image. In aspects, a computing device having at least two cameras and an image processing manager is configured to receive a first image at a first optical zoom from the first camera and a second image at a second optical zoom different from the first optical zoom from the second camera. The first camera and the second camera capture the same scene from different fields of view and different viewpoints. The image processing manager receives a desired digital zoom between the first optical zoom and the second optical zoom. Based on the first image and the second image, the image processing manager determines an overlap region of the first image in which the second image overlaps the first image. Further, based on the first image and the second image, the image processing manager applies a higher resolution of the second image than the first image to the overlap region of the first image to determine a fused image of the scene with a desired digital zoom. By applying the disclosed systems and techniques, an image processing manager effectively provides a fused image of a scene having a desired digital zoom and a higher resolution than the first image within at least a portion of the overlap region.

Description

Background Art

[0001] Modern smartphones, especially modern flagship smartphones, may include more than one camera. One of these cameras may be paired with a wide-angle lens with a wide field of view (FOV) and reduced optical zoom or no optical zoom (e.g., 0.7x, 1x). Another of these cameras may be paired with a telephoto lens with a narrow FOV and high optical zoom (e.g., 4x). Although modern smartphones with these camera options provide users with flexibility and choice, these camera options also bring significant challenges.

[0002] For example, a user of a modern smartphone with a 1x optical zoom camera and a 4x optical zoom camera may want to take a photo of a scene at 3x zoom. In this case, the 3x zoom must be achieved digitally. A common way to achieve 3x zoom is to digitally upsample the scene captured by the 1x optical zoom camera. Unfortunately, this approach can suffer from a loss of resolution, resulting in poor photos and impairing the user experience. Summary of the invention

[0003] This document describes systems and techniques for fusing optical zoom images into a digital zoom image. In various aspects, a computing device having at least two cameras and an image processing manager is configured to receive a first image at a first optical zoom from a first camera, and to receive a second image at a second optical zoom different from the first optical zoom from a second camera. The first camera and the second camera capture the same scene from different fields of view and different viewpoints. The image processing manager receives a desired digital zoom between the first optical zoom and the second optical zoom. Based on the first image and the second image, the image processing manager determines an overlap region of the first image in which the second image overlaps the first image. In addition, based on the first image and the second image, the image processing manager applies a higher resolution of the second image to the overlap region of the first image to determine a fused image of the scene with the desired digital zoom. By applying the disclosed systems and techniques, the image processing manager effectively provides a fused image of the scene with the desired digital zoom and a higher resolution than the first image in at least a portion of the overlap region. Thus, aspects of the disclosed systems and techniques can provide image enhancement.

[0004] In various aspects, a method is disclosed, the method comprising: receiving a first image and a second image from a first camera and a second camera, respectively, the first image being captured at a first optical zoom and the second image being captured at a second optical zoom different from the first optical zoom, the first camera and the second camera having different fields of view and different viewpoints, the first image and the second image capturing the same scene with the different fields of view and the different viewpoints; receiving a desired digital zoom, the desired digital zoom being between the first optical zoom and the second optical zoom; determining an overlap area of ​​the first image in which the second image overlaps the first image; determining a fused image of the scene with the desired digital zoom, the determination being based on the first image and the second image, the determination applying a higher resolution of the second image than the first image to the overlap area of ​​the first image; and providing a fused image of the scene with the desired digital zoom, the fused image of the scene having a higher resolution than the first image within at least a portion of the overlap area.

[0005] In various aspects, a computing device is disclosed, comprising: at least two cameras having different optical zooms, different fields of view, and different viewpoints; one or more processors; and a memory storing: instructions that, when executed by the one or more processors, cause the one or more processors to implement an image processing manager to provide image processing using the at least two cameras and the one or more processors by performing the method as described in any of the preceding claims.

[0006] The details of one or more implementations are set forth in the accompanying drawings and in the following detailed description. Other features and advantages will be apparent from the detailed description, drawings, and claims. This Summary is provided to introduce the subject matter that is further described in the detailed description. Therefore, the reader should not regard this Summary as describing the essential features or the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Details of one or more aspects of fusing optical zoom images into a digital zoom image are described in this document with reference to the following figures, where the use of the same numbers in different instances may indicate similar features or components: Figure 1 An example implementation of an example computing device having a first camera, a second camera, a display, and an image processing manager configured to fuse optical zoom images into a digital zoom image is shown; Figure 2 Shown from Figure 1 An example implementation of an example computing device configured to fuse optical zoom images into a digital zoom image; Figure 3 More details are shown from Figure 2 Example implementations of a computing device; Figure 4 An example implementation of an apparatus for training a machine learning model is shown; Figure 5 More details are shown from Figure 4 An example implementation of a device for training a machine learning model; Figure 6 shows an example implementation of a scene of a cube captured by three different cameras for training a machine learning model; Figure 7 An example implementation of a coarse alignment operation for training a machine learning model is shown; Figure 8 An example implementation of a fine alignment operation and an occlusion mask operation for training a machine learning model is shown; Fig. 9 shows example implementations of different depths of field for a camera with high optical zoom and a camera with low optical zoom; and Fig.10 Depicted is an example method for fusing optical zoom images into one digital zoom image. DETAILED DESCRIPTION Overview

[0008] Modern computing devices (e.g., smartphones, tablet computers) typically include more than one camera. Including more than one camera provides options to users of modern computing devices, which is generally expected for a good user experience. The options provided may include low light capability, a wide field of view (FOV) at low optical zoom, a narrow FOV at high optical zoom, a high frame capture rate, and the like. The low light capability option excels in night and twilight photography. A wide FOV at low optical zoom excels in selfie and close-up photography. A narrow FOV at high optical zoom excels in wildlife or other distant object photography. A high frame capture rate facilitates slow motion videography.

[0009] As a specific example, assume that a user of a smartphone with two cameras wants to capture a scene of bright green leaves on a tree branch. The first camera is paired with a lens configured to provide 1x optical zoom and a wide field of view (FOV). The second camera is paired with a lens configured to provide 4x optical zoom and a narrow FOV. Assume also that the background of the scene is a mountain range. The user may capture the scene using the second camera with 4x optical zoom and a narrow FOV. However, the user wants to include more of the mountain range in the scene, so the narrow FOV is not ideal. Alternatively, the user may capture the scene using the first camera with 1x optical zoom and a wide FOV. However, the user wants to include at least some of the finer details of the bright green leaves in the scene, so the 1x optical zoom is not ideal. Instead, the user selects a digital zoom between the first and second optical zooms (e.g., 3x), taps the smartphone's viewfinder to set a focus area around the bright green leaves, and taps the shutter button to capture the scene.

[0010] Because the user selected the desired 3x digital zoom, the first camera cannot natively capture the scene at 1x optical zoom or the second camera at 4x optical zoom. Instead, the first camera can capture the scene at 1x optical zoom, and the resulting image can be digitally magnified to the desired 3x digital zoom. This approach enables the user to capture more of the mountain range in the scene as desired. Unfortunately, however, the image digitally magnified in this manner lacks the finer details of the bright green leaves that the user wants to capture. The missing details of the bright green leaves in the resulting image are an example of a poor user experience. This document describes systems and techniques that are intended to fuse optical zoom images into one digital zoom image to capture both the desired finer details of the mountain range and the leaves. The disclosed systems and techniques can meet the user's desire to obtain an image that represents a wide view of the scene while also containing fine details. The conflict between these requirements can be resolved by the disclosed systems and techniques, which can provide digital zoom images that can be considered enhanced compared to optical zoom images.

[0011] The following discussion describes an operating environment and techniques that may be employed in the operating environment, as well as example methods. Although systems and techniques for fusing optical zoom images into a digital zoom image are described, it should be understood that the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations, and reference to the operating environment is made by way of example only. Operating Environment

[0012] Figure 1An example implementation 100 of an example computing device 102 is shown. As shown, the computing device 102 includes a first camera 104, a second camera 106, an image processing manager 108, and a display 110. The first camera 104 is composed of a first lens configured to provide a first optical zoom of 1x, a wide FOV, and a deep depth of field (DOF). Generally speaking, the first camera 104 with a wide FOV and a deep DOF provides more capabilities that are very suitable for taking selfies with friends or capturing scenes. The second camera 106 is composed of a second lens configured to provide a second optical zoom of 4x, a narrow FOV, and a shallow DOF. The second camera 106 with a larger optical zoom provides features that are very suitable for capturing wildlife or other scenes in the distance. In addition, the first camera 104 and the second camera 106 do not share the same point of view (POV). The image processing manager 108 is configured to fuse the optical zoom images into one digital zoom image.

[0013] In one example, user 110 of computing device 102 wants to take a photo of a scene of bright green leaves on tree branches. A mountain range is in the background of the scene. User 110 wants to capture the scene including portions of the mountain range in the background and at least some of the finer details of the bright green leaves on the tree branches in the foreground. To do so, user 110 frames the scene as shown in display 110-1, selects a desired 3x digital zoom as shown in display 110-2, clicks on a portion of the display to set a focus area 112 around the bright green leaves, and clicks a shutter button 114 to capture the scene. As shown in display 110-2, the scene at the 3x digital zoom is blurry and lacks the finer details of the bright green leaves.

[0014] In response to the user 110 clicking the shutter button 114, the first camera 104 and the second camera 106 may simultaneously capture a first image and a second image. After the images are captured, the image processing manager 108 receives the first image at 1x optical zoom from the first camera 104 and the second image at 4x optical zoom from the second camera 106. Based on the two images, the image processing manager 108 determines an overlapping area of ​​the first image in which the second image overlaps the first image. In this example, because the user adjusted the focus area 112 to surround the bright green leaves, both the first image and the second image are focused on the area surrounding the bright green leaves. The image processing manager 108 may use the focus area 112 surrounding the bright green leaves as the overlapping area. In addition, the image processing manager 108 receives a selection of a desired 3x digital zoom selected by the user 110. Based on the first image and the second image, the image processing manager 108 determines a fused image of the scene with a desired 3x digital zoom. In determining the fused image, the image processing manager 108 may apply a machine learning (ML) model that is configured to compensate for the different FOVs, different POVs, and different DOFs of the two cameras. In some aspects, the ML model or other appropriate systems and techniques may enable the image processing manager 108 to apply a higher resolution of the second image than the first image to the overlap region of the first image. In response to determining the fused image, the image processing manager 108 provides a fused image of the scene with a desired 3x digital zoom and a higher resolution than the first image within at least a portion of the overlap region.

[0015] Figure 2 More details are shown from Figure 1 An example implementation 200 of a computing device 102 configured to fuse optical zoom images into a digital zoom image. The computing device 102 is shown as various example devices. As non-limiting examples, the computing device 102 may be a smartphone 102-1, a tablet computer 102-2, a laptop computer 102-3, a desktop computer 102-4, a smart watch 102-5, a pair of smart glasses 102-6, a game controller 102-7, a smart home speaker 102-8, and a microwave oven 102-9. Although not shown, the computing device 102 may also be implemented as a health monitoring device, a personal media device, a drone, a home appliance, a security system, etc. It should be noted that the computing device 102 may be wearable, non-wearable but mobile, or relatively non-mobile (e.g., desktop computer 102-4, microwave oven 102-9). Furthermore, it should be noted that computing device 102 may be used with or embedded in many computing devices or peripherals, such as in a motor vehicle or as an accessory to a personal computer. Figure 2Additional interfaces and components omitted.

[0016] As shown, computing device 102 includes one or more processors 202 and computer readable medium 204 (CRM204). Processor 202 may include one or more of any suitable processor (e.g., central processing unit). CRM204 includes memory medium 206 and storage medium 208. Computing device 102 also includes operating system 210 (OS 210), application 212, and image processing manager 214 stored as computer readable instructions on CRM 204. Processor 202 may execute computer readable instructions on CRM 204 to provide some or all of the functions described herein. CRM 204 may include one or more non-transitory storage devices, such as random access memory, solid state drive, magnetic rotating drive, or any other type of storage medium suitable for storing electronic instructions, each of which is coupled to a data bus. The term "coupled" may refer to two or more elements that are in direct contact (physical, electrical, optical contact, etc.), or two or more elements that are not in direct contact with each other but still cooperate and interact with each other.

[0017] In some implementations, the image processing manager 214 may include one or more integrated circuits, systems on a chip, secure key storage, hardware embedded with firmware stored on a read-only memory, a printed circuit board having various hardware components, or any combination thereof. As described herein, an image fusion system may include one or more components of the computing device 102 configured to fuse optical zoom images into a digital zoom image, such as Figure 2 In other implementations, the image fusion system may be implemented as the computing device 102 .

[0018] Additionally, computing device 102 includes one or more sensors 216, input / output (I / O) ports 218, and Figure 1display 110. Sensor 216 may be disposed on computing device 102 or anywhere in the computing device. In some examples, sensor 216 may be disposed on or in a peripheral device connected to computing device 102. Sensor 216 may include any of a variety of sensing components, such as an audio sensor (e.g., a microphone), a touch input sensor (e.g., a touch screen), an image sensor (e.g., a camera, a video camera), an ambient light sensor (e.g., a photodetector), an acceleration sensor (e.g., an accelerometer), and the like. Sensor 216 may enable computing device 102 to automatically rotate content displayed by display 110 according to the orientation of computing device 102, measure ambient light to adjust the brightness of display 110, capture images of a scene, and the like. In an implementation, computing device 102 may include more than one sensing component of any one or more of the sensing components to implement various features and functions.

[0019] The I / O ports 218 may enable the computing device 102 to interact with other devices or users through peripheral devices, thereby transmitting any combination of digital and analog signals via wired means (e.g., Ethernet) or wireless means (e.g., radio). The I / O ports 218 may include any combination of internal or external ports, such as Universal Serial Bus (USB) ports, audio ports, video ports, and the like. Various peripheral devices (such as manual input devices, external CRMs, speakers, and displays) may be operatively coupled to the I / O ports 218.

[0020] The display 110 may be or utilize any of a variety of display technologies, including an organic light emitting diode display, a liquid crystal display, an electroluminescent display, etc. The display 110 may be referred to as a screen, so that content may be displayed on the screen. In an example, the content on the screen may be a viewfinder of a camera application.

[0021] Although not shown, the computing device may also include a system bus, interconnect, or other data transfer system that couples the various components of or within the computing device 102. The system bus or interconnect may include any one or combination of various bus structures, such as a memory bus, a peripheral bus, a USB, and a processor or local bus.

[0022] Figure 3A rear view of an example implementation 300 of a computing device 302 (e.g., computing device 102, smartphone 102-1) having a sensor 216 implemented as two separate image sensors (e.g., cameras) is shown. As shown, a first camera 304 and a second camera 306 are disposed on the back of the housing of the computing device 302. As shown, the first camera 304 and the second camera 306 reside in the same XY plane of the back of the housing of the computing device 302. Although not shown, the first camera 304 has a first lens that is configured to provide a first optical zoom of 1x, a wide FOV, and a deep DOF. The second camera 306 has a second lens that is configured to provide a second optical zoom of 4x, a narrow FOV, and a shallow DOF. The first camera 304 has a first POV, and the second camera 306 has a second POV that is different from the first POV. Thus, any time a user (e.g., user 110) captures a scene using a computing device 302 having two cameras, the first camera 304 captures the scene at a wide FOV, a deep DOF, and a first POV, and the second camera 306 captures the scene at a narrow FOV, a shallow DOF, and a second POV that is different from the first POV. To compensate for the different FOVs, DOFs, and POVs of the first and second cameras, the image processing manager 108 may apply the aforementioned ML model.

[0023] Figure 4 A rear view of an example implementation 400 of an example equipment for training an ML model is shown. As shown, the equipment includes a first computing device 402 and a second computing device 404. The first computing device 402 and the second computing device 404 reside in the same XY plane. Figure 3 4. Like computing device 302 of FIG. 4, the computing device has two cameras. First computing device 402 has first training camera 406 and second training camera 408. Similarly, second computing device 404 has first training camera 410 and second training camera 412. Although not shown, first training camera 406 and first training camera 410 each have a lens configured to provide a first training optical zoom of 1x, deep DOF, and wide FOV. Although not shown, second training camera 408 and second training camera 412 each have a lens configured to provide a second training optical zoom of 4x, shallow DOF, and narrow FOV.

[0024] For example, the training of the ML model uses a first set, a second set, and a third set of many (e.g., hundreds, thousands) images. The first set of images is captured by a second training camera 408 of the first computing device 402 at a second training optical zoom of 4x, a shallow DOF, and a narrow FOV. The images in this first set of images can be referred to as reference images (e.g., a first training input). The second set of images is captured by a first training camera 410 of the second computing device 404 at a first training optical zoom of 1x, a deep DOF, and a wide FOV. The images in this second set of images can be referred to as source images (e.g., a second training input). The third set of images is captured by a second training camera 412 of the second computing device 404 at a second training optical zoom of 4x, a shallow DOF, and a narrow FOV. The images in this third set of images can be referred to as target output images (e.g., a third training input). Although the training of the ML model can use multiple sets of many images, only a single image in each set of images is referenced herein.

[0025] Figure 5 More details are shown from Figure 4 5. A top view of an example implementation 500 of a training device for a computer system. As shown, computing device 402 and computing device 404 reside in the same XY plane and face the same scene of cube 502 in the same positive Z direction, with the top surface of the cube shown. Therefore, the three training cameras reside in the same XY plane and face the same scene of cube 502 in the same positive Z direction. A first image (e.g., a reference image) is captured from a first POV 504 by a second training camera 408 of first computing device 402 at a second training optical zoom of 4x, a shallow DOF, and a narrow FOV. A second image (e.g., a source image) is captured from a second POV 506 by a first training camera 410 of second computing device 404 at a first training optical zoom of 1x, a deep DOF, and a wide FOV. A third image (e.g., a target output image) is captured from a third POV 508 by a second training camera 412 of second computing device 404 at a second training optical zoom of 4x, a shallow DOF, and a narrow FOV.

[0026] Figure 6 Shown by Figure 5Detailed view 600-1 shows a target output image of the scene of cube 502 captured by second training camera 412 of second computing device 404. Detailed view 600-2 shows a source image of the scene of cube 502 captured by first training camera 410 of second computing device 404. Detailed view 600-3 shows a reference image of the scene of cube 502 captured by second training camera 408 of first computing device 402. As shown in detailed view 600-1, second training camera 412 captures images of the front side 602 and left side 604 of cube 502 from third POV 508 at 4x training optical zoom. As shown in detailed view 600-2, first training camera 410 captures images of the front side 602 and left side 604 of cube 502 from second POV 506 at 1x training optical zoom. As shown in detail view 600 - 3 , second training camera 408 captures images of the front side 602 and right side 606 of cube 502 from first POV 504 at 4x training optical zoom.

[0027] Figure 6 It is emphasized that different faces (eg, left face 604 , right face 606 ) of cube 502 may be captured depending on the POV (eg, POV 508 , POV 504 ) of a camera (eg, training camera 412 , training camera 408 ). Figure 6 It is also emphasized that the cube 502 may appear large or small depending on the optical zoom (e.g., 1x, 4x) and FOV of the camera. To fuse the optical zoom images into one digital zoom image, the ML model is trained to compensate for the different optical zooms and FOVs through two operations: a coarse alignment operation and a fine alignment operation.

[0028] Figure 7An example implementation 700 of training an ML model to perform a coarse alignment operation is shown. The coarse alignment operation includes warping the source image and the reference image to align with the target output image. Warping includes cropping, magnifying, and rotating the source image and the reference image. Coarse alignment may also include feature matching and global homography, which relates two images of the same scene from different POVs by how features in the first image may be located at different locations in the second image (e.g., how pixels "move" between images). Detailed view 700-1 shows the warping of the source image of cube 502. In this example, the warping simply includes magnifying the source image of cube 502 to match the size of the target output image of cube 502 from detailed view 600-1. Detailed view 700-2 shows the warping of the reference image of cube 502. In this example, no magnification is required because the reference image uses the same 4x training optical zoom as the target output image. However, when compared to detailed view 600-1, detailed view 700-2 of the reference image emphasizes that the left side 604 is not visible. This is a result of the different POVs of the cameras used to capture the images. To address this, the ML model is also trained to use occlusion masking operations to compensate for different POVs.

[0029] Figure 8 An example implementation 800 of training an ML model to perform a fine alignment operation is shown. A detailed view 800-1 shows a reference image of a cube 502 using dashed lines superimposed on a target output image of the cube 502 using solid lines. As shown, the lower left corner 802 of the front face 602 of the cube 502 is not in the same position in the two images. Similarly, the lower right corner 804 of the front face 602 of the cube 502 is not in the same position in the two images. Detailed view 800-2 shows a magnified view of the lower left corner 802 of the front face 602 of the cube 502. The movement 806 of the lower left corner 802 from the reference image (dashed line) to the target output image (solid line) is emphasized. Although only the movement 806 of the lower left corner 802 is shown, the movement of each pixel from the reference image to the target output image can be calculated by an existing convolutional neural network (CNN) (e.g., PWC-Net). The output describing the movement of each pixel can be referred to as an optical flow. The optical flow can be stored as a heat map and can be used to train an ML model to perform a fine alignment operation.

[0030] Figure 8An example of training an ML model to perform an occlusion masking operation is also shown in detail view 800-3. As shown, the right side 606 of the cube 502 from the reference image is filled with black. This is an occlusion mask, which emphasizes the parts of an image (e.g., reference image, source image) that are not visible in another image (e.g., target output image). The occlusion mask can be used to train the ML model to compensate for the different POVs of the camera. In the fusion of the source image with the reference image, details of the reference image that are not covered by the occlusion mask (e.g., not filled with black) can be transferred to the source image.

[0031] The occlusion mask can also be used to train an ML model to transfer the details of a reference image to a source image based on a loss combination. The training of the ML model to transfer these details based on a loss combination is performed using the luminance (e.g., grayscale) channel to avoid color shift. The first detail transfer is the Visual Geometry Group CNN (VGGNet) detail transfer. VGGNet performs well in object recognition (e.g., the group of details that make up an object). Equation 1 defines the VGGNet detail transfer using a fused image (fused), a target output image (target), and an occlusion mask (occ_mask). Equation 1

[0032] The second detail transfer is the least absolute deviation (L1) detail transfer. However, the second detail transfer is not a pure L1 detail transfer, as this may result in large brightness shifts. Therefore, Gaussian blur is applied to the source image and the fused image to avoid large brightness shifts. Equation 2 defines the L1 detail transfer. Equation 2

[0033] The third detail transfer is the context detail transfer. The context detail transfer performs well in further aligning the misaligned regions between the fused image and the target output image. Equation 3 defines the context detail transfer. Equation 3

[0034] Recall that the 1x training optical zoom camera and the 4x training optical zoom camera do not share the same DOF. Fig. 9A top view of an example implementation 900 for training an ML model to compensate for different DOFs is shown. Detailed view 900-1 shows the top surface of cube 502, reference line 902, and reference line 904. Reference line 902 shows the XY plane in which the farthest object in the scene (e.g., cube 502) is in focus. Reference line 904 shows the XY plane in which the closest object in the scene is in focus. DOF 906 between reference lines is the distance between the closest object in focus and the farthest object in the scene when the camera captures the scene at 4x optical zoom. Similarly, detailed view 900-2 shows the top surface of cube 502, reference line 908, and reference line 910. Reference line 908 shows the XY plane in which the farthest object in the scene is in focus. Reference line 910 shows the XY plane in which the closest object in the scene is in focus. DOF 912 between reference lines is the distance between the closest object in focus and the farthest object in the scene when the camera captures the scene at 1x optical zoom. As shown in detail view 900-1, the DOF 906 of the 4x optical zoom camera is shallow, while the DOF 912 of the 1x optical zoom camera shown in detail view 900-2 is deep. The difference in DOF causes different portions of the scene of cube 502 to be in focus in the source image and the reference image.

[0035] When fusing optical zoom images (source and reference images) into a digital zoom image (digitally magnified source image), the ML model is trained not to transfer out-of-focus details. To achieve this, the ML model can make use of a defocus map. The defocus map and its application can be described by two equations. Equation 4 defines the defocus map. Equation 4

[0036] The defocus map (map(x,y)) is a function of the optical flow (flow(x,y)) of the entire image and the optical flow distribution (P(f)) of the focus area of ​​the image (e.g., focus area 112). The focus optical flow (argmax[P(f)]) within the focus area is calculated using k-means clustering. The difference between flow(x,y) and argmax[P(f)] calculates whether a certain area of ​​the image is in focus, and the difference is set as the defocus map. The defocus map can be applied via a defocus mask to transfer details from a reference image at 4x optical zoom to a source image at 1x optical zoom. Equation 5 defines the defocus mask. Equation 5

[0037] The defocus mask (mask(x,y)) is a sigmoid function of the difference between the defocus map and the adjustable parameter (do).

[0038] To avoid color shift in fusing optical zoom images into a digital zoom image, both the reference image and the target output image are set to match the color of the source images. The colors are set using a global mean and standard deviation color matching method.

[0039] In order to avoid poorly fused images in fusing optical zoom images into one digital zoom image, a set of fallback conditions can be set. The fallback conditions can include low light environment, large errors in re-projection of details from the reference image to the source image, large base frame delta, and out-of-focus reference images.

[0040] Although the techniques herein have been described with reference to or for use with a computing device having a first camera paired with a lens capable of 1x optical zoom and a second camera paired with a lens capable of 4x optical zoom, at least some of the techniques described above may also be implemented by other computing devices. For example, a computing device having a first camera paired with a lens capable of 1x optical zoom, a second camera paired with a lens capable of 4x optical zoom, and a third camera paired with a lens capable of 10x optical zoom may implement the techniques described above. For example, the techniques may be applied to fuse images captured by a 4x optical zoom camera and a 10x optical zoom camera into a 6x digital zoom image. Example Method

[0041] Fig.10 A method 1000 is depicted that enables fusion of optical zoom images into a digital zoom image. The method is shown as a collection of boxes that specify operations that are performed but are not necessarily limited to the order or combination shown for performing the operations by the respective boxes. In addition, any one or more of the operations may be repeated, combined, reorganized, or concatenated to provide a number of additional or alternative methods. In portions of the following discussion, reference may be made to Figure 1 Example implementation of Figures 2 to 9 Although the details and examples are described herein, references to them are made only as examples. The techniques are not limited to being performed by one entity or multiple entities operating on one device.

[0042] At 1002, an image processing manager (e.g., image processing manager 108) receives a first image and a second image from a first camera and a second camera, respectively, the first image being captured at a first optical zoom and the second image being captured at a second optical zoom different from the first optical zoom, the first camera and the second camera having different fields of view and different viewpoints, the first image and the second image capturing the same scene with different fields of view and different viewpoints.

[0043] At 1004, the image processing manager receives a desired digital zoom, which is between the first optical zoom and the second optical zoom. For example, the first optical zoom may be 1x or less (e.g., 0.5x, 0.7x), and the second optical zoom may be 4x or greater (e.g., 5x, 6x). Thus, the desired digital zoom may be any value from 1x or less to 4x or greater (e.g., 2x, 3x). Receiving the desired digital zoom may include receiving a user selection of the digital zoom.

[0044] At 1006, the image processing manager determines an overlap region of the first image in which the second image overlaps the first image. In an example, the overlap region may be a focus region selected by a user prior to capturing an image of the scene. The focus region may indicate to the first camera and the second camera an area of ​​the scene that the camera should focus on, for example, using an autofocus optical system.

[0045] At 1008 , the image processing manager determines a fused image of the scene with a desired digital zoom, the determination being based on the first image and the second image, the determination applying a higher resolution of the second image than the first image to an overlapping area of ​​the first image.

[0046] At 1010, the image processing manager provides a fused image of the scene with a desired digital zoom, the fused image of the scene having a higher resolution than the first image within at least a portion of the overlap region. Figure 1 As shown, a higher resolution may be the detail of the leaves captured by the second camera at the second optical zoom of 4x. The fused image of the scene may be provided for display, for example, on the display 110 of the computing device 102. Additional Examples

[0047] In the following sections, additional examples are provided.

[0048] Example 1: A method, comprising: receiving a first image and a second image from a first camera and a second camera, respectively, the first image being captured at a first optical zoom and the second image being captured at a second optical zoom different from the first optical zoom, the first camera and the second camera having different fields of view and different viewpoints, the first image and the second image capturing the same scene with the different fields of view and the different viewpoints; receiving a desired digital zoom, the desired digital zoom being between the first optical zoom and the second optical zoom; determining an overlapping area of ​​the first image, wherein the second image overlaps with the first image in the overlapping area; determining a fused image of the scene with the desired digital zoom, the determination being based on the first image and the second image, the determination applying a higher resolution of the second image than the first image to the overlapping area of ​​the first image; and providing the fused image of the scene with the desired digital zoom, the fused image of the scene having a higher resolution than the first image in at least a portion of the overlapping area.

[0049] Example 2: The method of Example 1, wherein the first camera and the second camera are different cameras in a shared camera array.

[0050] Example 3: The method of Example 1, wherein the first camera and the second camera are separate cameras housed within the same mobile computing device, the separate cameras each having a different lens.

[0051] Example 4: The method of Example 3, wherein the different lenses are configured to provide the first optical zoom and the second optical zoom.

[0052] Example 5: A method as described in Example 4, wherein the first optical zoom is 1x or less, the second optical zoom is 2x or greater, and the desired digital zoom is between the first optical zoom 1x or less and the second optical zoom 2x or greater, excluding endpoints.

[0053] Example 6: The method of Example 1, wherein the first image and the second image are captured simultaneously.

[0054] Example 7: The method of example 1, wherein receiving the desired digital zoom receives a selection of digital zoom by a user of a mobile computing device associated with the first camera and the second camera.

[0055] Example 8: The method as described in any of the preceding examples, further comprising: receiving the desired digital zoom before receiving the first image and the second image, and wherein receiving the first image and the second image comprises: causing the first camera and the second camera to capture the first image and the second image, respectively, in response to receiving the desired digital zoom.

[0056] Example 9: A method as described in any of the preceding examples, wherein determining the fused image of the scene with the desired digital zoom applies a machine learning model, wherein the machine learning model is configured to compensate for the different fields of view of the first image and the second image.

[0057] Example 10: A method as described in any of the preceding examples, wherein determining the fused image of the scene with the desired digital zoom applies a machine learning model, wherein the machine learning model is configured to compensate for the different viewpoints of the first image and the second image.

[0058] Example 11: The method of Example 10, wherein compensating for the different viewpoints generates an occlusion mask that highlights portions of the second image that are not shared by the first image.

[0059] Example 12: The method of Example 11, wherein determining the fused image of the scene with the desired digital zoom copies details not emphasized by the occlusion mask from the second image to the first image.

[0060] Example 13: A method as described in any of the preceding examples, wherein the machine learning model is trained by a first set of images, a second set of images, and a third set of images, wherein the first set of images is captured by a first training camera at a first training optical zoom, the second set of images is captured by a second training camera at a second training optical zoom different from the first training optical zoom, and the third set of images is captured by a third training camera at a training optical zoom that is the same as the first training optical zoom.

[0061] Example 14: The method of Example 13, wherein the first training camera and the second training camera are physically separate cameras located at different locations.

[0062] Example 15: The method of Example 13, wherein the first training camera and the second training camera capture the first set of images and the second set of images while facing in the same direction from the same plane.

[0063] Example 16: The method of Example 13, wherein the first training optical zoom is greater than the second training optical zoom.

[0064] Example 17: The method of Example 13, wherein the first set of images and the second set of images are input images, and the third set of images comprises a target output image.

[0065] Example 18: A computing device comprising: at least two cameras, the at least two cameras having different optical zooms, different fields of view, and different viewpoints; one or more processors; and a memory, the memory storing: instructions, which, when executed by the one or more processors, cause the one or more processors to implement an image processing manager to provide image processing using the at least two cameras and the one or more processors by performing a method as described in any of the preceding examples.

[0066] Example 19: A computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to perform a method as described in any one of Examples 1 to 17. in conclusion

[0067] Unless the context dictates otherwise, the use of the word "or" herein may be viewed as the use of an "inclusive or" or a term that permits the inclusion or application of one or more items connected by the word "or" (e.g., the phrase "A or B" may be interpreted as permitting only "A," only "B," or both "A" and "B"). In addition, as used herein, a phrase referring to "at least one" of a list of items refers to any combination of those items, including single members. For example, "at least one of a, b, or c" may encompass a, b, c, ab, ac, bc, and abc, as well as any combination with multiples of the same element (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other ordering of a, b, and c). In addition, the items represented in the drawings and the terms discussed herein may indicate one or more items or terms, and thus the items and terms in the written description in singular or plural form may be referred to interchangeably.

[0068] Although implementations for fusing optical zoom images into one digital zoom image have been described in language specific to certain features and / or methods, the subject matter of the appended claims is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations for fusing optical zoom images into one digital zoom image.

Claims

1. A method comprising: receiving a first image and a second image from a first camera and a second camera, respectively, the first image being captured at a first optical zoom and the second image being captured at a second optical zoom different from the first optical zoom, the first camera and the second camera having different fields of view and different viewpoints, the first image and the second image capturing the same scene with the different fields of view and the different viewpoints; receiving a desired digital zoom, the desired digital zoom being between the first optical zoom and the second optical zoom; determining an overlapping region of the first image in which the second image overlaps the first image; determining a fused image of the scene having the desired digital zoom, the determining being based on the first image and the second image, the determining applying a higher resolution of the second image than the first image to the overlapping region of the first image; as well as The fused image of the scene having the desired digital zoom is provided, the fused image of the scene having a higher resolution than the first image within at least a portion of the overlap region. 2 . The method of claim 1 , wherein the first camera and the second camera are separate cameras in a shared camera array, the separate cameras each having a different lens. 3 . The method of claim 1 , wherein the first camera and the second camera are separate cameras housed within the same mobile computing device, the separate cameras each having a different lens. 4 . The method of claim 3 , wherein the different lenses are configured to provide the first optical zoom and the second optical zoom.

5. The method of claim 4, wherein the first optical zoom is 1x or less, the second optical zoom is 2x or greater, and the desired digital zoom is between the first optical zoom 1x or less and the second optical zoom 2x or greater, exclusive of endpoints.

6. A method as claimed in any preceding claim, wherein the first image and the second image are captured simultaneously.

7. The method of claim 1 , wherein receiving the desired digital zoom comprises: A selection of a digital zoom by a user of a mobile computing device associated with the first camera and the second camera is received.

8. The method according to any one of the preceding claims, further comprising: The desired digital zoom is received prior to receiving the first image and the second image, and wherein receiving the first image and the second image includes causing the first camera and the second camera to capture the first image and the second image, respectively, in response to receiving the desired digital zoom.

9. A method as described in any of the preceding claims, wherein determining the fused image of the scene with the desired digital zoom applies a machine learning model, and the machine learning model is configured to compensate for the different fields of view of the first image and the second image.

10. The method of any one of claims 1 to 8, wherein determining the fused image of the scene with the desired digital zoom applies a machine learning model, the machine learning model being configured to compensate for the different viewpoints of the first image and the second image.

11. The method of claim 10, wherein compensating for the different viewpoints generates an occlusion mask that emphasizes portions of the second image that are not shared by the first image.

12. The method of claim 11, wherein determining the fused image of the scene with the desired digital zoom copies details not emphasized by the occlusion mask from the second image to the first image.

13. The method of any one of claims 9 to 12, wherein the machine learning model is trained by a first set of images, a second set of images, and a third set of images, wherein the first set of images is captured by a first training camera at a first training optical zoom, the second set of images is captured by a second training camera at a second training optical zoom different from the first training optical zoom, and the third set of images is captured by a third training camera at a training optical zoom the same as the first training optical zoom. The method of claim 13 , wherein the first training camera and the second training camera are physically separate cameras.

15. The method of claim 13, wherein the first training camera and the second training camera capture the first set of images and the second set of images while facing in the same direction from the same plane. The method of claim 13 , wherein the first training optical zoom is greater than the second training optical zoom.

17. The method of claim 13, wherein the first set of images and the second set of images are input images and the third set of images comprises a target output image.

18. A computing device comprising: at least two cameras, the at least two cameras having different optical zooms, different fields of view, and different viewpoints; one or more processors; as well as A memory storing: Instructions which, when executed by the one or more processors, cause the one or more processors to implement an image processing manager to provide image processing using the at least two cameras and the one or more processors by performing the method as claimed in any one of the preceding claims.

19. A computer readable medium comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of any one of claims 1 to 17.

Images