Systems and methods for content enhancement using four color filter array sensors

Abstract

A system and method for content enhancement using a four-color filter array sensor is provided. The invention discloses a system and method for content enhancement using a QCFA sensor of an electronic device. Embodiments herein include previewing content in a field of view (FOV) of a first QCFA sensor, where the first QCFA sensor is operating in a first mode. Embodiments herein include detecting whether a zoom level of the previewed content satisfies a first zoom level criterion and whether a light condition of the previewed content satisfies a light criterion. Further, the first QCFA sensor may be configured to switch from the first mode to the second mode in response to detecting that a zoom level of the previewed content satisfies a first zoom level criterion and a light condition of the previewed content satisfies a light criterion, in order to enhance a resolution of the content being previewed in the FOV of the first QCFA sensor.

Description

[0001] This application is a divisional application of a patent application with an application date of August 13, 2020, application number 202080056971.1, and invention name “System and method for content enhancement using a four-color filter array sensor”. Technical Field

[0002] The present disclosure relates to content enhancement using an image sensor of an electronic device, and more particularly, to systems and methods for content enhancement using a quad color filter array (QCFA) sensor of an electronic device. Background Art

[0003] Typically, an electronic device (e.g., a smart phone) can utilize an image sensor to capture images and image quality. In conventional methods, previews of images zoomed at high zoom levels result in loss of image details, resulting in blurred and out-of-focus images. Accordingly, the image quality when capturing images at high zoom levels is dramatically reduced. In conventional methods, during real-time preview zooming using a camera application of a smart phone, the output image (actual sensor data) of the image sensor is cropped based on the zoom ratio, and then the cropped image is enlarged. This results in loss of details, affecting the preview image quality. Accordingly, preview images and captured images using conventional methods may be blurred and / or out of focus. One of the key performance indicators (KPIs) for determining image quality is the clarity of the captured image. It would be advantageous for the captured image to contain additional image details. Lossless image quality is required for real-time preview zooming, making the preview image and captured image clearer and more focused even when the zoom level is high.

[0004] Figure 1 A digital scaling method of a QCFA sensor using electronics according to the prior art is shown.

[0005] Reference Figure 1 , when previewing content in the field of view of the QCFA sensor, the QCFA sensor operates in binning mode. Figure 1 As shown, a 64MP QCFA sensor is used. When previewing content, the QCFA sensor acquires 64MP of image data, and in binning mode, the QCFA sensor combines 4 adjacent pixels to create one pixel. Since 4 pixels are combined into one pixel, the final sensor output is 1 / 4 of the sensor size. As shown in this example, the QCFA sensor output will be 16MP (i.e., the actual sensor data is 16MP).

[0006] At step 102, the zoom level of the preview content is 1X. During the 1X zoom level, since the display resolution of the electronic device is 16MP, the image processor provides the same 16MP sensor output. Accordingly, there is no loss of detail at the 1X zoom level.

[0007] At step 104, the zoom level of the preview content is 2X. During the 2X zoom level, the image processor crops the central portion of the QCFA sensor output 16MP. That is, half of the width and half of the height are cropped, thereby generating 4MP image data (16MP / 4=4MP). In this example, since the display resolution of the electronic device is 16MP, the electronic device cannot save the 4MP image data. Accordingly, the image processor enlarges the 4MP image data to 16MP image data, resulting in loss of details at the 2X zoom level.

[0008] At step 106, the zoom level of the preview content is 8X. During the 8X zoom level, the image processor crops the QCFA sensor output 16MP to 1 / 8 of the width and 1 / 8 of the height, thereby generating 0.25MP image data (i.e., 16MP / 64=0.25MP). In this example, since the display resolution of the electronic device is 16MP, the electronic device cannot save the 0.25MP image data. Accordingly, the image processor enlarges the 0.25MP image data to 16MP image data, resulting in a serious loss of details at the 8X zoom level. Similarly, as the zoom level increases, the image data from the image processor becomes increasingly blurred and / or distorted.

[0009] Figure 2 A block diagram showing an implementation flow for amplifying content being previewed according to the prior art.

[0010] Reference Figure 2 In a conventional system, when a user zooms in on content being previewed by using a camera application of an electronic device, the camera application sends a zoom value to a zoom control block based on a zoom level of the content being previewed. The zoom value displays information about four corners of a rectangle that the user wants to zoom in on. The zoom control block converts cropping parameters to an image processor based on the zoom value received from the camera application, and the image processor crops the content that the user may want to zoom in on. In a conventional system, the zoom control block does not have the intelligence to output a lossless image or a clear and focused image even when the zoom level of the previewed content increases.

[0011] In view of the above, it is desirable to address the above-mentioned deficiencies or other drawbacks. Summary of the invention

[0012] Technical issues

[0013] It is an object of certain embodiments according to the present disclosure to provide a system and method for content enhancement using a QCFA sensor.

[0014] Another object according to certain embodiments of the present disclosure is to preview content in a field of view of a first QCFA sensor of an electronic device, wherein the first QCFA sensor is operating in a first mode.

[0015] Another object according to a specific embodiment of the present disclosure is to detect whether a zoom level of the previewed content meets a first zoom level criterion and whether a light condition of the previewed content meets a light criterion.

[0016] Another object according to a specific embodiment of the present disclosure is to configure the first QCFA sensor to switch from a first mode to a second mode to enhance the resolution of the content previewed in the field of view of the first QCFA sensor in response to detecting that the zoom level of the content previewed in the field of view of the first QCFA sensor satisfies a first zoom level standard and the light conditions of the content previewed in the field of view of the first QCFA sensor meet a light standard.

[0017] Another object according to a specific embodiment of the present disclosure is to maintain the first QCFA sensor in the first mode in response to detecting that the zoom level of the content previewed in the field of view of the first QCFA sensor does not meet the first zoom level standard and the light conditions of the content previewed in the field of view of the first QCFA sensor do not meet at least one of the light standards.

[0018] It is another object according to certain embodiments of the present disclosure to display content previewed in the field of view of the first QCFA sensor with enhanced resolution.

[0019] The above information is presented as background information only to assist in understanding the present disclosure. No determination has been made, and no assertion is made, as to whether any of the above might be applicable as prior art with respect to the present disclosure.

[0020] Technical Solution

[0021] Accordingly, embodiments herein disclose a system and method for content enhancement using a QCFA sensor of an electronic device. Embodiments include previewing content in a field of view of a first QCFA sensor of an electronic device, wherein the first QCFA sensor is operating in a first mode. In addition, specific embodiments include detecting whether a zoom level of the previewed content meets a first zoom level standard, and whether a light condition of the previewed content meets a light standard. In addition, specific embodiments include configuring the first QCFA sensor to switch from a first mode to a second mode to enhance the resolution of the content previewed in the field of view of the first QCFA sensor in response to detecting that a zoom level of the content previewed in the field of view of the first QCFA sensor meets the first zoom level standard and a light condition of the content previewed in the field of view of the first QCFA sensor meets the light standard. In addition, specific embodiments include maintaining the first QCFA sensor in the first mode in response to detecting that a zoom level of the content previewed in the field of view of the first QCFA sensor does not meet the first zoom level standard and a light condition of the content previewed in the field of view of the first QCFA sensor does not meet at least one of the light standard. In addition, embodiments include displaying the content previewed in the field of view of the first QCFA sensor with enhanced resolution.

[0022] In an embodiment, after configuring the first QCFA sensor to switch to the second mode, the method further includes previewing content in the field of view of the first QCFA sensor, wherein the first QCFA sensor is operating in the second mode. The method also includes determining that the zoom level of the previewed content meets the second zoom level criteria.

[0023] In an embodiment, the method further includes previewing the content in the field of view of the second QCFA sensor upon determining that the zoom level of the previewed content meets the second zoom level standard, wherein the second QCFA sensor is operating in the first mode. The method further includes detecting whether the zoom level of the content previewed in the field of view of the second QCFA sensor meets the third zoom level standard and whether the light condition of the content previewed in the field of view of the second QCFA sensor meets the light standard. The method further includes configuring the second QCFA sensor to switch from the first mode to the second mode to enhance the resolution of the content previewed in the field of view of the second QCFA sensor in response to detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor meets the third zoom level standard and the light condition of the content previewed in the field of view of the second QCFA sensor meets the light standard. The method further includes maintaining the second QCFA sensor in the first mode in response to detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor does not meet the third zoom level standard and the light condition of the content previewed in the field of view of the second QCFA sensor does not meet the light standard. The method further includes displaying the content previewed in the field of view of the second QCFA sensor with enhanced resolution.

[0024] In an embodiment, configuring the first QCFA sensor to switch to the second mode includes: switching the first QCFA sensor from the first mode to the second mode. The method also includes previewing content in a preview field of the first QCFA sensor, wherein the first QCFA sensor is operating in the second mode. The method also includes center-cropping the content previewed in the second mode of the first QCFA sensor in half in two dimensions.

[0025] In an embodiment, the step of displaying enhanced content in the field of view of the first QCFA sensor includes: enhancing the center cropped content in the preview field of the first QCFA sensor, wherein the first QCFA sensor is operating in the second mode. The method also includes displaying the content previewed in the field of view of the first QCFA sensor with enhanced resolution.

[0026] In an embodiment, the step of configuring the second QCFA sensor to switch to the second mode includes: switching the second QCFA sensor from the first mode to the second mode. The method also includes: previewing content in a preview field of the second QCFA sensor, wherein the second QCFA sensor is operating in the second mode; and center-cropping the content previewed in the second mode of the second QCFA sensor into half in two dimensions.

[0027] In an embodiment, wherein the step of displaying the enhanced content in the field of view of the second QCFA sensor comprises: enhancing the center cropped content in the preview field of the second QCFA sensor, wherein the second QCFA sensor is operating in the second mode. The method also includes displaying the enhanced content in the field of view of the second QCFA sensor.

[0028] When considered in conjunction with the following description and the accompanying drawings, these and other aspects of the embodiments herein will be better appreciated and understood. However, it should be understood that the following description, although indicating preferred embodiments and many of their specific details, is provided by way of illustration and not limitation. Without departing from the spirit of the present invention, many changes and modifications may be made within the scope of the embodiments herein, and the embodiments herein include all of these modifications.

[0029] Beneficial Effects of the Invention

[0030] The proposed method for content enhancement using a QCFA sensor of an electronic device can output lossless image quality at a zoom level of 2X and above, thereby producing a high-detail image. In the proposed method, preview images and captured images at zoom level 2X and above are clearer and more focused. When the subject of the image or the content of the image is clear, the image looks clear and can convey additional details, thereby achieving content enhancement (high-detail image) in the field of view of the QCFA sensor. By using multiple QCFA sensors, further lossless images at 8X zoom level are possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] For a more complete understanding of the present disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, wherein like reference numerals represent like parts:

[0032] Figure 1 A digital scaling method for a QCFA sensor using an electronic device according to the prior art is shown;

[0033] Figure 2 A block diagram showing an implementation flow for enlarging previewed content according to the prior art;

[0034] Figure 3A and 3B Schematic diagrams showing different modes of a QCFA sensor according to certain embodiments of the present disclosure;

[0035] Figure 4 shows a schematic diagram illustrating a Bayer image remosaic cropping mode of a QCFA sensor according to certain embodiments of the present disclosure;

[0036] Figure 5A method for content enhancement using a QCFA sensor and a method for digital zooming using a QCFA sensor of an electronic device according to certain embodiments of the present disclosure are shown;

[0037] Figure 6 A block diagram illustrating an electronic device for content enhancement using a QCFA sensor according to certain embodiments of the present disclosure;

[0038] Fig. 7A A flow chart illustrating a method for content enhancement using a QCFA sensor according to certain embodiments of the present disclosure;

[0039] Figure 7B A flow chart illustrating a method for performing content enhancement after configuring a first QCFA sensor to switch to a second mode according to certain embodiments of the present disclosure;

[0040] Figure 8 A block diagram showing an implementation flow for amplifying content being previewed according to a specific embodiment of the present disclosure;

[0041] Fig. 9 shows a schematic diagram illustrating switching of QCFA sensor modes based on a zoom level of content previewed in a field of view of the QCFA sensor and light conditions of content previewed in the field of view of the QCFA sensor in accordance with certain embodiments of the present disclosure;

[0042] Fig.10 A flow chart illustrating switching modes of a QCFA sensor based on a zoom level of content previewed in the field of view of the QCFA sensor and light conditions of content previewed in the field of view of the QCFA sensor according to certain embodiments of the present disclosure;

[0043] Fig.11A and Fig. 11B shows an example scenario illustrating image quality when previewing an image at a zoom level greater than 2.0X using prior art and proposed methods according to certain embodiments of the present disclosure;

[0044] Fig. 12A and Fig. 12B shows an example scene illustrating image quality when capturing images at a zoom level greater than 2.0X using prior art and proposed methods according to certain embodiments of the present disclosure;

[0045] Fig.13A and Fig. 13B A schematic diagram showing multiple QCFA sensors obtaining a lossless image at 8X zoom level according to certain embodiments of the present disclosure is shown;

[0046] Fig.14A and Fig. 14BAn example scenario showing digital zoom solution enhancement at 100X using prior art methods and the proposed method is shown according to certain embodiments of the present disclosure;

[0047] Fig.15A and Fig. 15B An example showing enhancement of an image captured at zoom level 30X using digital zoom solutions of prior art methods and the proposed method according to certain embodiments of the present disclosure; and

[0048] Fig.16A and Fig. 16B An example of enhancing an image captured at a zoom level of 100X using a digital zoom solution of the prior art and the proposed method according to certain embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0049] Before proceeding to the following detailed description, it may be helpful to set forth definitions of certain words and phrases used throughout this patent document: the terms "include" and "comprising" and their derivatives mean including but not limited to; the term "or" is inclusive, meaning and / or; the phrases "associated with" and "associated with" and their derivatives may mean including, included within, interconnected with, containing, contained within, connected to or connected with, coupled to or coupled with, communicable with, cooperating with, interleaved, juxtaposed, proximate to, bound to or bound with, having, having the property of, and the like; and the term "controller" means any device, system, or portion thereof that controls at least one operation, such device may be implemented in hardware, firmware, or software, or some combination of at least two thereof. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely.

[0050] In addition, the various functions described below can be implemented or supported by one or more computer programs, each of which is formed by a computer-readable program code and implemented in a computer-readable medium. The terms "application" and "program" refer to one or more computer programs, software components, instruction sets, processes, functions, objects, classes, examples, related data or a part thereof that are suitable for being implemented in a suitable computer-readable program code. The phrase "computer-readable program code" includes any type of computer code (including source code, object code and executable code). The phrase "computer-readable medium" includes any type of medium (such as a read-only memory (ROM), random access memory (RAM), hard drive, compact disk (CD), digital video disc (DVD) or any other type of memory) that can be accessed by a computer. "Non-transitory" computer-readable media does not include wired, wireless, optical or other communication links that transmit temporary electrical signals or other signals. Non-transitory computer-readable media include media that can permanently store data and media that can store data and rewrite it later (such as rewritable optical disks or erasable memory devices).

[0051] Definitions for certain words and phrases are provided throughout this patent document, those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

[0052] Discussed below Figures 1 to 16B The various embodiments used to describe the principles of the present disclosure in this patent document are illustrative only and should not be interpreted in any way to limit the scope of the present disclosure. Those skilled in the art will understand that the principles of the present disclosure can be implemented in any suitably arranged system or device.

[0053] The embodiments herein and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. The description of known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments herein. In addition, because some embodiments can be combined with one or more other embodiments to form new embodiments, the various embodiments described herein are not necessarily mutually exclusive. Unless otherwise indicated, the term "or" as used herein refers to a non-exclusive "or". The examples used herein are intended only to facilitate understanding of the manner in which the embodiments herein can be practiced, and further enable those skilled in the art to practice the embodiments herein. Accordingly, the examples should not be interpreted as limiting the scope of the embodiments herein.

[0054] As traditional in the art, embodiments can be described and illustrated according to the block of performing one or more functions described.These blocks (which may be referred to as units or modules, etc., herein) are physically implemented by analog or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.), and may be optionally driven by firmware and software.For example, the circuit may be implemented in one or more semiconductor chips, or may be implemented on substrate supports (such as printed circuit boards, etc.).The circuit constituting the block may be implemented by dedicated hardware, or by a processor (for example, one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the block and a processor that performs other functions of the block.Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interacting and discrete blocks.Similarly, without departing from the scope of the present disclosure, the block of the embodiment may be physically combined into more complex blocks.

[0055] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented here are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as extending to any changes, equivalents and substitutions other than those specifically set forth in the accompanying drawings. Although the terms first, second, etc. can be used here to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0056] According to certain embodiments of the present disclosure, systems and methods for content enhancement using a QCFA sensor of an electronic device are disclosed. Embodiments include previewing content in a field of view of a first QCFA sensor of an electronic device, wherein the first QCFA sensor is operating in a first mode. In addition, embodiments include detecting whether a zoom level of the previewed content meets a first zoom level standard and whether a light condition of the previewed content meets a light standard. In addition, embodiments include: in response to detecting that a zoom level of the content previewed in the field of view of the first QCFA sensor meets the first zoom level standard and a light condition of the content previewed in the field of view of the first QCFA sensor meets the light standard, configuring the first QCFA sensor to switch from a first mode to a second mode to enhance the resolution of the content previewed in the field of view of the first QCFA sensor. In addition, embodiments include: in response to detecting that a zoom level of the content previewed in the field of view of the first QCFA sensor does not meet the first zoom level standard and a light condition of the content previewed in the field of view of the first QCFA sensor does not meet at least one of the light standard, maintaining the first QCFA sensor in the first mode. In addition, embodiments include: displaying enhanced content in the field of view of the first QCFA sensor.

[0057] Reference is now made to the drawings, and more particularly to the illustrative examples of Figures 3 through 16. Where like reference numerals indicate corresponding features consistently throughout the drawings, such features are shown in specific embodiments.

[0058] Figure 3A and Figure 3B Schematic diagram showing different modes of a QCFA sensor according to embodiments disclosed herein.

[0059] Figure 3A An example of a first mode of a QCFA sensor is shown. This can be a binning mode that can produce a brighter image in a dim light scene. In a dim light scene, the QCFA sensor operates in binning mode. In a dim light scene operating in binning mode of the QCFA sensor, 4 adjacent pixels of the image can be combined into one pixel, thereby displaying a lower resolution, less noisy, brighter image. Since 4 pixels are combined into one, the final sensor output is 1 / 4 the sensor size. Figure 3A As shown in the figure, when previewing content, the QCFA sensor acquires 64MP of image data. In binning mode, the QCFA sensor can combine 4 adjacent pixels to produce one pixel. Figure 3A In the example shown, the QCFA sensor output would be 16MP (ie, the actual sensor data is 16MP).

[0060] Figure 3B An example of a second mode of a QCFA sensor is shown. The example may be operating in a Bayer image regeneration mode, wherein the Bayer image regeneration mode may generate a full resolution output in bright light scenes. In bright light scenes, the full resolution Bayer image regeneration mode of the QCFA sensor may output the full megapixel capability of the QCFA sensor. In this example, the QCFA sensor may provide an improved megapixel image in bright light scenes. Figure 3B As shown in the example, when capturing image content, the QCFA sensor obtains 64MP of image data; and in the full-resolution Bayer image regeneration mode, the QCFA sensor output will be 64MP (ie, the actual sensor data is 64MP).

[0061] In existing systems, when content is previewed in the field of view of the QCFA sensor, the QCFA sensor operates in a binning mode. Unlike existing systems, in at least one method according to the present disclosure, the QCFA sensor may be configured to switch from the binning mode to the full-resolution Bayer image regeneration mode when the zoom level of the previewed content satisfies the zoom level standard and the light condition of the previewed content satisfies the light standard. For example, the zoom level standard may be a 2X zoom, and the light standard may be an ambient light standard. When the zoom level of the content previewed in the field of view of the QCFA sensor satisfies the 2X zoom and the light condition of the content previewed in the field of view of the QCFA sensor satisfies the light standard (i.e., the ambient light standard (normal light or bright light)), the QCFA sensor switches from the binning mode to the full-resolution Bayer image regeneration mode. Proposed content enhancement method using a QCFA sensor of an electronic device 600 (see, e.g. Figure 6 ) can generate lossless image quality at zoom levels of 2X and above, resulting in high-detail images. In the proposed method, preview images and captured images at zoom levels of 2X and above can be clearer and better focused. When the subject of the image or the content of the image is clear, the image appears clearer and the image can convey additional details, providing enhanced content (high-detail images) in the field of view of the QCFA sensor.

[0062] Figure 4 A schematic diagram illustrating a cropping mode of a Bayer image regeneration of a QCFA sensor in accordance with certain embodiments of the present disclosure.

[0063] Reference Figure 4 As an illustrative example of the proposed method, a new mode for content enhancement using a QCFA sensor is disclosed (e.g., Bayer image regeneration cropping mode).

[0064] exist Figure 4 In an illustrative example of a Bayer image regeneration cropping mode of the QCFA sensor, when the zoom level of the previewed content meets the zoom level criteria and the light conditions of the previewed content meet the light criteria, the electronic device 600 may switch the QCFA sensor from the binning mode to the Bayer image regeneration mode. The QCFA sensor hardware may center-crop the image content in half in two dimensions (i.e., half the width and half the height are cropped) in the Bayer image regeneration mode of the QCFA sensor.

[0065] exist Figure 4In the illustrative example of the Bayer image regeneration crop mode, first, the QCFA sensor is configured for the Bayer image regeneration mode, and the QCFA sensor hardware outputs the center crop output of the Bayer image regeneration mode by configuring the crop register settings of the QCFA sensor. Since 1 / 2 of the width and 1 / 2 of the height are cropped, the final QCFA sensor output is 1 / 4 of the sensor size, but the field of view is magnified by 2 times. Figure 4 As shown, when the zoom level of the previewed content meets the zoom level criteria and the light conditions of the previewed content meet the light criteria, the QCFA sensor starts operating in the full-resolution Bayer image regeneration mode. The QCFA sensor acquires 64MP image data and, in the full-resolution Bayer image regeneration mode, implements Bayer encoding to produce a 64MP full-resolution image. The QCFA sensor hardware crops the center portion of the 64MP image. When the QCFA sensor hardware crops the center portion of the image by 1 / 2 of the width and 1 / 2 of the height, the QCFA sensor automatically generates a 2X zoomed image as output. As shown Figure 4 As shown in the example of , the QCFA sensor output is 16MP. Therefore, in the new mode of the example (i.e., Bayer image regeneration crop mode), the QCFA sensor output is a cropped 2X zoomed image. In addition, at zoom levels higher than 2X, the image processor 660 of the electronic device 600 (see, e.g., Figure 6 ) further crops the 2X zoomed image output of the QCFA sensor based on the zoom level.

[0066] Figure 5 A block diagram shows a method of performing content enhancement using a QCFA sensor according to a certain embodiment of the present disclosure, and also shows a method of performing digital zooming using the QCFA sensor of an electronic device 600 .

[0067] Reference Figure 5 In an illustrative example of FIG. 1 , when the zoom level of the content previewed in the field of view of the QCFA sensor is a 1X zoom level, the QCFA sensor operates in a first mode (binning mode). Figure 5 As shown, a 64MP QCFA sensor is used. When previewing content, the QCFA sensor acquires 64MP of image data, and in binning mode, the QCFA sensor combines 4 adjacent pixels to produce one pixel. Since 4 pixels are combined into one pixel, the final sensor output is 1 / 4 of the sensor size. Here, as shown by example, the QCFA sensor output will be 16MP (ie, the actual sensor data is 16MP).

[0068] During the IX zoom level (at step 502), since the display resolution of the electronic device 600 is 16MP, the image processor 660 outputs the same 16MP sensor output. Accordingly, there is no loss of detail at the IX zoom level.

[0069] When previewing content in the field of view of the QCFA sensor of the electronic device 600, the QCFA sensor is operating in binning mode. Now, when the zoom level of the previewed content meets the zoom level standard and the light condition of the previewed content meets the light standard, the electronic device 600 is configured to switch the QCFA sensor from binning mode to Bayer image regeneration mode. For example, the zoom level standard can be a 2X zoom level, and the light standard can be an ambient light standard (normal light or bright light). The image content is previewed in Bayer image regeneration mode. The QCFA sensor hardware crops the image content in half in two dimensions (i.e., length and width) in the Bayer image regeneration mode of the QCFA sensor. Therefore, in the Bayer image regeneration cropping mode, the QCFA sensor hardware outputs the center cropped output of the Bayer image regeneration mode by configuring the cropping register settings of the QCFA sensor. Since 1 / 2 of the width and 1 / 2 of the height are cropped, the final QCFA sensor output is 1 / 4 of the sensor size, but the field of view is magnified by 2 times.

[0070] like Figure 5 As shown in the example, when the zoom level of the content previewed in the field of view of the QCFA sensor is at a 2X zoom level and the light conditions of the previewed content meet the light criteria, the electronic device 600 can switch the QCFA sensor from the binning mode to the Bayer image regeneration mode. In this case, the QCFA sensor acquires 64MP image data, and in the full-resolution Bayer image regeneration mode, implements Bayer encoding to produce a 64MP full-resolution image. The QCFA sensor hardware can crop the center portion of the 64MP image. When the QCFA sensor hardware crops the center portion of the image with 1 / 2 of the width and 1 / 2 of the height, the QCFA sensor automatically generates a 2X zoomed image as output. As shown Figure 5 As shown in , the QCFA sensor output is 16MP, and the QCFA sensor output is a 2X zoomed image.

[0071] During the 2X zoom level (step 504), since the display resolution of the electronic device 600 is 16MP, the image processor 660 outputs the same 16MP sensor output. In other words, there is no loss of detail at the 2X zoom level. The 16MP sensor output can be preserved. The electronic device 600 outputs a lossless image at the 2X zoom level.

[0072] In the proposed method, at zoom levels higher than 2X, the image processor 660 of the electronic device 600 further crops the 2X zoomed image output of the QCFA sensor based on the zoom level.

[0073] like Figure 5 As shown in FIG. 5 , at step 506, the zoom level of the content previewed in the field of view of the QCFA sensor is 8X, and the light condition of the previewed content meets the configured light standard. In this example, since the zoom level of the content previewed in the field of view of the QCFA sensor exceeds the 2X zoom level, and the light condition of the previewed content meets the light standard, the QCFA sensor automatically provides a 2X zoom image as output. Figure 5 As shown in , the QCFA sensor output is 16MP, and the QCFA sensor output is a 2X zoom image. During the 8X zoom level, the image processor 660 of the electronic device 600 further crops the 2X zoom sensor output. The image processor 660 further crops the QCFA sensor output 16MP by 1 / 4 of the height and 1 / 4 of the width, thereby generating 1MP image data (16MP / 16=1MP). Since the display resolution of the electronic device 600 is 16MP, the image processor 660 enlarges the 1MP image data to 16MP image data.

[0074] As shown in the proposed method, the QCFA sensor hardware outputs a 2X scaled image as the final QCFA sensor output. Therefore, at the 2X zoom level, there is no need to scale and amplify the final QCFA sensor output by the image processor 660 of the electronic device 600. Figure 5 As shown in , 16MP image data received from the QCFA sensor hardware is displayed by the electronic device 600. Unlike the existing system, no cropping and enlargement is required at the 2X zoom level. Accordingly, the image quality for real-time preview zooming in the proposed method is better, and the preview image and the captured image are clear and focused.

[0075] Unlike existing systems, in the proposed method, instead of upscaling 0.25MP image data to 16MP at an 8X zoom level, 1MP image data is upscaled to 16MP at an 8X zoom level. Accordingly, the image quality of the real-time preview zoom is improved in the proposed method, and both the preview image and the captured image are clearer and more focused. Similarly, as the zoom level increases, unlike existing systems, the image quality of the real-time preview zoom in the proposed method is improved, and the preview image and the captured image are clearer and more focused.

[0076] Figure 6 A block diagram of an electronic device 600 for content enhancement using a QCFA sensor is shown in accordance with certain embodiments of the present disclosure.

[0077] Examples of electronic device 600 include, but are not limited to, mobile phones, smart phones, tablet computers, handheld devices, laptops, computers, wearable computing devices, Internet of Things (IoT) devices, cameras, web cameras, digital single-lens reflex (DSLR) cameras, video cameras, digital cameras, mirrorless cameras, still cameras, etc. Electronic device 600 may also include a user application interface and a camera application for previewing and capturing media or images. The camera application may be used to develop a graphical user interface (GUI). Electronic device 600 may include at least one of two image sensors, three image sensors, four image sensors, etc. An example of an image sensor may be a QCFA sensor.

[0078] Reference Figure 6 , the electronic device 600 includes a communicator 610, a memory 620, a processor 630, a content enhancement unit 640, a sensor 650, and an image processor 660. The sensor 650 may be an image sensor (e.g., a QCFA sensor). The image processor 660 (also referred to as an image signal processor (ISP) 660) is a dedicated digital signal processor for image processing in the electronic device 600. The electronic device 600 may include more than one sensor 650 (such as a first QCFA sensor and a second QCFA sensor).

[0079] The communicator 610 may be configured to communicate internally between the internal unit and the external device via one or more networks. The memory 620 may include one or more computer-readable storage media. The memory 620 may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory 620 may be considered to be a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not implemented in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as the memory 620 being non-removable. In some examples, the memory 620 may be configured to store a larger amount of information than the memory 620. In some examples, a non-transitory storage medium (e.g., in a random access memory (RAM) or a cache memory) may store data that may change over time.

[0080] The processor 630 may be coupled to the memory 620 and the communicator 610; and the processor 630 may be communicatively coupled to the content enhancement unit 640, the sensor 650, and the image processor 660. The processor 630 may be configured to preview content in a field of view of a first QCFA sensor of the electronic device 600, wherein the first QCFA sensor is operating in a first mode. For example, the first mode may be a binning mode of the QCFA sensor.

[0081] The processor 630 may be configured to detect whether the zoom level of the previewed content satisfies a first zoom level criterion and whether the light condition of the previewed content satisfies a light criterion. For example, the first zoom level criterion may be a 2X zoom level of the content previewed in the field of view of the first QCFA sensor. The light criterion may be an ambient light criterion (e.g., normal light or bright light).

[0082] Upon detecting that the zoom level of the content previewed in the field of view of the first QCFA sensor satisfies the first zoom level criterion and the light condition of the content previewed in the field of view of the first QCFA sensor satisfies the light criterion, the processor 630 is configured to switch the first QCFA sensor from the first mode to the second mode so as to enhance the resolution of the content previewed in the field of view of the first QCFA sensor. For example, the second mode may be a Bayer image regeneration mode of the QCFA sensor.

[0083] Upon detecting that the zoom level of the content previewed in the field of view of the first QCFA sensor does not satisfy the first zoom level standard and the light condition of the content previewed in the field of view of the first QCFA sensor does not satisfy the light standard, the processor 630 is configured to maintain the first QCFA sensor in the first mode. The processor 630 is also configured to display enhanced content in the field of view of the first QCFA sensor.

[0084] After configuring the first QCFA sensor to switch to the second mode, the processor 630 is also configured to preview content in the field of view of the first QCFA sensor, wherein the first QCFA sensor is operating in the second mode. The processor 630 can be configured to determine that the zoom level of the previewed content meets the second zoom level standard. For example, the second zoom level standard can be a 4X zoom level. The processor 630 can also be configured to activate the second QCFA sensor, wherein the second QCFA sensor is operating in the first mode. For example, the first QCFA sensor can be a wide-angle sensor and the second QCFA sensor can be a telephoto sensor.

[0085] The processor 630 may be configured to preview content in a field of view of a second QCFA sensor of the electronic device 600, wherein the second QCFA sensor is operating in the first mode. The processor 630 may also be configured to detect whether a zoom level of the previewed content satisfies a third zoom level criterion and whether a light condition of the previewed content satisfies a light criterion. For example, the third zoom level criterion may be an 8X zoom level. The light criterion may be an ambient light criterion (e.g., normal light or bright light).

[0086] The processor 630 can be configured to switch the second QCFA sensor from the first mode to the second mode to enhance the resolution of the content previewed in the field of view of the second QCFA sensor when it is detected that the zoom level of the content previewed in the field of view of the second QCFA sensor meets the third zoom level standard and the light conditions of the content previewed in the field of view of the second QCFA sensor meet the light standard.

[0087] Upon detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor does not satisfy at least one of the third zoom level criteria and the light conditions of the content previewed in the field of view of the second QCFA sensor do not satisfy the light criteria, the processor 630 may be configured to maintain the second QCFA sensor in the first mode. The processor 630 is also configured to display enhanced content in the field of view of the second QCFA sensor.

[0088] although Figure 6 The illustrative examples of the electronic device 600 show the hardware elements of the electronic device 600, but it should be understood that other embodiments are not limited thereto. For example, in other embodiments, the electronic device 600 may include fewer or greater numbers of elements. In addition, the markings or names of the elements are for illustrative purposes only and should not be interpreted in any way as limiting the scope of the present disclosure. One or more components may be combined together to perform the same or substantially similar functions of content enhancement using the QCFA sensor of the electronic device 600.

[0089] Fig. 7A A flow chart illustrating a process 700 of a method for content enhancement using a QCFA sensor according to certain embodiments of the present disclosure.

[0090] Reference Fig. 7AIn an illustrative example of, at step 710, the method may include previewing content in a field of view of a first QCFA sensor of the electronic device 600, wherein the first QCFA sensor is operating in a first mode. At step 720, the method may include detecting whether a zoom level of the previewed content satisfies a first zoom level standard and whether a light condition of the previewed content satisfies a light standard. At step 730, upon detecting that the zoom level of the content previewed in the field of view of the first QCFA sensor satisfies the first zoom level standard and upon detecting that the light condition of the content previewed in the field of view of the first QCFA sensor satisfies the light standard, the method may further include: configuring the first QCFA sensor to switch from the first mode to a second mode to enhance the resolution of the content previewed in the field of view of the first QCFA sensor. At step 740, upon detecting that the zoom level of the content previewed in the field of view of the first QCFA sensor does not satisfy the first zoom level standard and the light condition of the content previewed in the field of view of the first QCFA sensor does not satisfy at least one of the light standard, the method may include maintaining the first QCFA sensor in the first mode. The method may further include displaying enhanced content in the field of view of the first QCFA sensor.

[0091] The various actions, actions, blocks, steps, etc. in the method can be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some actions, actions, blocks, steps, etc. can be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

[0092] Figure 7B A flow chart illustrating a process 730 of a method for performing content enhancement after configuring a first QCFA sensor to switch to a second mode according to certain embodiments of the present disclosure.

[0093] Reference Figure 7BIn an illustrative example of, at step 731, the method may include previewing content in a field of view of a first QCFA sensor, wherein the first QCFA sensor is operating in a second mode. At step 732, the method may include determining whether a zoom level of the content being previewed meets a second zoom level standard. At step 733, the method may include activating a second QCFA sensor, wherein the second QCFA sensor is operating in the first mode. At step 734, the method may also include previewing content in a field of view of a second QCFA sensor of the electronic device 600, wherein the second QCFA sensor is operating in the first mode. At step 735, the method may include detecting whether a zoom level of the content being previewed meets a third zoom level standard and whether a light condition of the previewed content meets a light standard. At step 736, upon detecting that a zoom level of the content being previewed in the field of view of the second QCFA sensor meets the third zoom level standard and detecting that a light condition of the content being previewed in the field of view of the first QCFA sensor meets the light standard, the method may further include configuring the second QCFA sensor to switch from the first mode to the second mode to enhance the resolution of the content previewed in the field of view of the second QCFA sensor. At step 737, upon detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor does not satisfy at least one of the third zoom level criteria and the light conditions of the content previewed in the field of view of the second QCFA sensor do not satisfy the light criteria, the method may include maintaining the second QCFA sensor in the first mode. The method also includes displaying the enhanced content in the field of view of the second QCFA sensor.

[0094] The various actions, actions, blocks, steps, etc. in the method can be performed in the order presented, in a different order, or simultaneously. In addition, in some embodiments, some actions, actions, blocks, steps, etc. can be omitted, added, modified, skipped, etc. without departing from the scope of the present invention.

[0095] Figure 8 A block diagram showing an implementation flow for enlarging previewed content according to a specific embodiment of the present disclosure. Figure 8 In an illustrative example of FIG. 6 , a user may preview content in the field of view of a QCFA sensor of an electronic device 600. Whenever a user zooms in on content being previewed using a camera application of the electronic device 600, the camera application may send a zoom value to the smart zoom control block based on the zoom level of the content being previewed. The zoom value outputs information about the four corners of a rectangle that the user may want to zoom (i.e., the coordinates of the four corners of the rectangle). The 3A block may calculate the lux index of the content previewed in the field of view of the QCFA sensor. In the proposed method, as Figure 8As shown, after calculating the lux index, the 3A block can send a lux value to the smart zoom control block. The lux value enables the smart zoom control block to determine the ambient light conditions. Based on the zoom value received from the camera application and the lux value received from the 3A block, the smart zoom control block can check whether the zoom value received from the camera application (i.e., the zoom level of the previewed content) is at a 2X zoom level or above a 2X zoom level. The smart zoom control block can also check whether the lux value received from the 3A block (i.e., the light condition of the content previewed in the field of view of the QCFA sensor) is low light, normal light, bright light, or other.

[0096] If the QCFA sensor is operating in the first mode, after checking the scaling value and the lux value, the smart scaling control block may determine whether to switch the QCFA sensor from the first mode (e.g., binning mode) to the second mode (e.g., Bayer image regeneration mode) or to keep the QCFA sensor in the first mode (binning mode). Similarly, in the proposed method, if the QCFA sensor is operating in the second mode, the smart scaling control block determines whether to switch the QCFA sensor from the second mode (Bayer image regeneration mode) to the first mode (binning mode) or to keep the QCFA sensor in the second mode (Bayer image regeneration mode).

[0097] Depending on the zoom value and the lux value, if the QCFA sensor needs to be switched from the first mode to the second mode or from the second mode to the first mode, the smart zoom control block sends a command to the QCFA sensor to switch modes, so that the QCFA sensor switches its mode. The smart zoom control block also sends a command to the image processor 660 regarding cropping parameters (coordinates of the four corners of a rectangle) of the content previewed in the field of view of the QCFA sensor according to the zoom value received from the camera application.

[0098] Unlike conventional systems, if the smart zoom control block detects that the zoom level of the content previewed in the field of view of the QCFA sensor satisfies the zoom level standard and the light condition of the content previewed in the field of view of the QCFA sensor satisfies the light standard, the QCFA sensor may be configured by the electronic device 600 to switch from the first mode (binning mode) to the second mode (Bayer image regeneration mode) to enhance the resolution of the content previewed in the field of view of the QCFA sensor. If at least one of the zoom level of the content previewed in the field of view of the QCFA sensor does not satisfy the zoom level standard and the light condition of the content previewed in the field of view of the QCFA sensor does not satisfy the light standard is detected, the QCFA sensor remains in the first mode (binning mode).

[0099] In at least one proposed approach, the smart zoom control block can help achieve improved image quality by controlling the QCFA sensor and the image processor 660. Depending on the zoom level of the previewed content and the light conditions of the previewed content, the smart zoom control block can intelligently select a low-resolution binning mode or a high-resolution Bayer image regeneration mode to maintain optimal image quality.

[0100] Fig. 9 is an illustrative example of a schematic diagram showing switching of QCFA sensor modes based on a zoom level of content previewed in the field of view of the QCFA sensor and light conditions of content previewed in the field of view of the QCFA sensor in accordance with certain embodiments of the present disclosure.

[0101] Reference Fig. 9 In an illustrative example of, in state 1, the zoom level of the content previewed in the field of view of the QCFA sensor is less than the 2X zoom level, and a low light condition exists when previewing the content in the field of view of the QCFA sensor. Therefore, the QCFA sensor can operate in binning mode. In step 902, the zoom level of the content previewed in the field of view of the QCFA sensor is increased to above the 2X zoom level, but as shown above in state 2, since low light conditions prevail when previewing the content in the field of view of the QCFA sensor, the QCFA sensor operates in binning mode. The QCFA sensor may be sensitive to lighting conditions, and operating the QCFA sensor in binning mode under low light conditions may output improved image quality. In step 904, when the zoom level of the previewed content is reduced to below the 2X zoom level, the QCFA sensor returns to state 1 and operates in binning mode. In step 906, the ambient illumination increases (i.e., the lighting conditions when previewing the content in the field of view of the QCFA sensor are improved to normal light or bright light). As Fig. 9 As shown in state 3, since the zoom level of the previewed content is greater than or equal to the 2X zoom level, and the lighting condition of the previewed content is normal light or bright light, the QCFA sensor can start operating in the cropped Bayer image regeneration mode. Operating the QCFA sensor in the cropped Bayer image regeneration mode includes switching the QCFA sensor from the binning mode to the full resolution Bayer image regeneration mode, and center-cropping the content of the Bayer image regeneration mode of the QCFA sensor in half in two dimensions, thereby providing a lossless image at a 2X zoom level in the cropped Bayer image regeneration mode. Here, the sensor output is a 2X zoomed image, and further at a zoom level higher than 2X, the image processor 660 of the electronic device 600 further crops the 2X zoomed image output of the QCFA sensor based on the zoom level.

[0102] At step 910, as seen in state 4, the zoom level is reduced to below the 2X zoom level, and the QCFA sensor then begins operating in binning mode. In state 4, even if the lighting conditions are normal or bright, the QCFA sensor may operate in binning mode because the zoom level of the previewed content has been reduced to below the 2X zoom level. At step 912, if the zoom level of the previewed content is increased to above the 2X zoom level, the QCFA sensor may begin operating in the Bayer image regeneration crop mode (e.g., Fig. 9 In step 914, the ambient illumination may be reduced (i.e., the lighting condition when previewing the content in the field of view of the QCFA sensor is low light). Fig. 9 As shown in state 1 of FIG. 1 , the QCFA sensor may begin operating in binning mode. Fig. 9 As shown in state 4 of , even if the lighting conditions when previewing content in the field of view of the QCFA sensor are normal light or bright light, the QCFA sensor can also operate in the binning mode because the zoom level is lower than the 2X zoom level.

[0103] Table 1 shows some considerations when selecting binning mode or Bayer image regeneration cropping mode based on zoom level and light conditions.

[0104] [Table 1]

[0105]

[0106]

[0107] To maintain smooth transitions, frequent switching between modes is not recommended. This can be achieved by setting a hysteresis after the switch (i.e. updating the light condition and scaling variables only after a certain number of frames have been processed). This ensures that there is a delay of a fixed number of frames between two consecutive switches and the sensor is already in a stable state.

[0108] Unlike some methods and systems, the QCFA sensor can intelligently switch modes based on the zoom level of the content previewed in the field of view of the QCFA sensor and the light conditions of the content previewed in the field of view of the QCFA sensor, so the conversion of the QCFA sensor mode intelligently maintains optimal image quality.

[0109] Fig.10 A flow chart illustrating a process 1000 for switching a mode of a QCFA sensor based on a zoom level of content previewed in the field of view of the QCFA sensor and light conditions of content previewed in the field of view of the QCFA sensor, according to a particular embodiment of the present disclosure.

[0110] Reference Fig.10In an illustrative example, in one embodiment, at step 1002, when a user previews content in the field of view of the QCFA sensor, a frame request from the camera application indicates an area to be magnified. The frame request from the camera application outputs a cropped area [top, left, width, and height] representing the area to be magnified. At step 1004, using the frame request from the camera application, the smart zoom control block can determine that the zoom level of the previewed content is changed from 1X to 1.9X. At step 1010, the smart zoom control block determines whether the mode of the QCFA sensor needs to be switched, and whether the mode of the QCFA sensor needs to be switched is determined based on the zoom level of the previewed content and the light conditions of the previewed content. Since the zoom level of the previewed content is 1.9X, the QCFA sensor operates in a binning mode, and the QCFA sensor is also operating in a binning mode at a 1X zoom level. Accordingly, there is no need to switch the mode of the QCFA sensor. At step 1012, in response to determining that the mode of the QCFA sensor does not need to be switched, the smart zoom control block can also determine that a zoom transformation is not needed. The values ​​of the crop region may be based on the resolution (fixed size) of the binning mode of the QCFA sensor; and therefore, in the binning mode, these values ​​may be forwarded to the image processor 660 without any changes (i.e., no scaling transformation of the values ​​of the crop region is required). At step 1014, in conjunction with steps 1024 and 1026, the crop region parameters based on the binning mode resolution may be input to the image processor 660 of the electronic device 600.

[0111] In one embodiment, at step 1002, when the user is previewing content in the field of view of the QCFA sensor, a frame request from the camera application indicates an area to be magnified. The frame request from the camera application outputs a cropped area [top, left, width, height] representing the area to be magnified. At step 1006, using the frame request from the camera application, the smart zoom control block can determine that the zoom level of the previewed content is changed from 1.9X to 2X. The smart zoom control block can also determine whether the light condition of the previewed content is normal light or bright light. At step 1010, the smart zoom control block can determine whether the mode of the QCFA sensor needs to be switched, and the smart zoom control block can determine based on the zoom level of the previewed content and the light condition of the previewed content. The zoom level of the previewed content is changed from 1.9X to 2X, and the light condition of the previewed content is normal or bright light. Because the QCFA sensor is operating in binning mode at the 1.9X zoom level. Therefore, the mode of the QCFA sensor needs to be switched. In step 1016 in conjunction with step 1018, in response to determining that the mode of the QCFA sensor needs to be switched from the binning mode to the Bayer image regeneration cropping mode, the smart scaling control block sends a command to the QCFA sensor to switch modes (write I2C settings to switch QCFA sensor). In step 1020, the smart scaling control block may determine whether a scaling transformation is required. The value of the crop region may be based on the resolution (fixed size) of the binning mode of the QCFA sensor. When the QCFA sensor switches from the binning mode to the Bayer image regeneration cropping mode, a scaling transformation of the value of the crop region may be required. When the sensor mode is changed to the Bayer image regeneration cropping mode, the crop region based on the resolution of the binning mode does not correspond to the actual field of view (FOV) of the Bayer image regeneration cropping mode and may need to be converted. In step 1022 , in combination with step 1024 and step 1026 , the value of the cropping region parameter based on the binning mode resolution of the QCFA sensor is transformed by the electronic device 600 into a value according to the zoom level of the content previewed in the field of view of the QCFA sensor and fed to the image processor 660 of the electronic device 600 .

[0112] In one embodiment, at step 1002, while the user is previewing content in the field of view of the QCFA sensor, a frame request from the camera application displays or otherwise indicates an area to be magnified. The frame request from the camera application may output a cropped area [top, left, width, and height] indicating the area to be magnified. At step 1008, using the frame request from the camera application, the smart zoom control block determines that the zoom level of the preview content is greater than or equal to 2X. The smart zoom control block determines that the light condition of the preview content changes from normal light to low light. At step 1010, the smart zoom control block determines whether the mode of the QCFA sensor needs to be switched, and whether the mode of the QCFA sensor needs to be switched is determined based on the zoom level of the preview content and the light condition of the preview content. The zoom level of the previewed content is greater than or equal to the 2X zoom level, but the light condition of the previewed content is low light. The QCFA sensor is operating in the Bayer image regeneration crop mode at a zoom level greater than or equal to the 2X zoom level, but because the light condition is low light, the mode of the QCFA sensor needs to be switched from the Bayer image regeneration crop mode to the binning mode. In response to determining that the mode of the QCFA sensor needs to be switched from the Bayer image regeneration crop mode to the binning mode at step 1016 in conjunction with step 1018, the smart scaling control block sends a command to the QCFA sensor to switch modes (write I2C settings to switch QCFA sensor). At step 1020, the smart scaling control block determines whether a scaling transformation is required. The values ​​of the crop region may be based on the resolution (fixed size) of the binning mode of the QCFA sensor, and thus, in the binning mode, these values ​​may be forwarded to the image processor 660 without any changes (i.e., no conversion of the values ​​of the crop region is required). At step 1014, in conjunction with steps 1024 and 1026, the crop region parameters based on the binning mode resolution may be input to the image processor 660 of the electronic device 600.

[0113] Fig.11A and Fig. 11B An example scenario illustrating image quality when previewing an image at a zoom level greater than 2.0X using prior art and proposed methods is shown according to certain embodiments of the present disclosure.

[0114] like Fig. 11B As shown in the example of , when the preview is scaled to 8X, the proposed method can display the text in the scene more clearly, and the overall image is more clearly focused compared with the preview images of other methods.

[0115] Fig. 12A and Fig. 12B An example scene illustrating image quality when capturing images at a zoom level greater than 2.0X using prior art and proposed methods is shown in accordance with certain embodiments of the present disclosure.

[0116] like Fig. 12B As shown in the example of , when the image is captured at 2X and enlarged again above 2X zoom, the proposed method can output clearer text and can also add additional details to the scene.

[0117] Fig.13A and 13B A schematic diagram showing multiple QCFA sensors obtaining a lossless image at an 8x zoom level according to certain embodiments of the present disclosure.

[0118] Reference Fig.13A An illustrative example of using 2 QCFA sensors in order to obtain a lossless zoom at an 8X zoom level. The first QCFA sensor may operate as a wide angle camera and the second QCFA sensor may operate as a telephoto camera such that the second QCFA sensor starts operating at the 4X zoom level of the first QCFA sensor. The base field of view (FOV) of the second QCFA sensor is the same as the FOV of the first QCFA sensor center cropped to a 4.0X zoom.

[0119] The second QCFA camera can be activated at a zoom level of 4.0X. Between zoom levels 4.0X-8.0X, the second QCFA sensor can operate in binning mode and can switch to a Bayer image regeneration crop mode at 8.0X. As a result, zoom at 8.0X is lossless.

[0120] Reference Fig. 13B 6 shows an illustrative example of a method for content enhancement using multiple QCFA sensors. The proposed method may include previewing content in a field of view of a first QCFA sensor by an electronic device 600. The first QCFA sensor may be operating in a first mode (e.g., a binning mode). When a zoom level of the previewed content meets a first zoom level criterion (e.g., a zoom level criterion of 1 to 4), the content is previewed. Fig. 13B As shown in the example in , 2X zoom level) and the light conditions of the previewed content meet the light criteria (normal light or bright light), the electronic device 600 can switch the first QCFA sensor to the Bayer image regeneration cropping mode to enhance the resolution of the content previewed in the field of view of the first QCFA sensor. In order to operate the first QCFA sensor in the Bayer image regeneration cropping mode, the first QCFA sensor can be switched to the second mode (Bayer image regeneration mode) and can be center cropped by the electronic device 600 (i.e., the content previewed in the second mode of the first QCFA sensor is cropped in half in two dimensions (i.e., half the width and half the height)). As Fig. 13BAs shown in , after the 2X zoom level, the first QCFA sensor may be operating in the Bayer image regeneration crop mode. When the zoom level of the content previewed in the field of view of the first QCFA sensor meets the second zoom level criteria (e.g., Fig. 13B As shown in FIG. 6 , at a 4X zoom level), the electronic device 600 may activate the second QCFA sensor. When the second QCFA sensor is activated, the first QCFA sensor is deactivated by the electronic device 600. The second QCFA sensor is activated at the 4X zoom level, outputting a 4X lossless zoom. The proposed method includes previewing content in the field of view of the second QCFA sensor by the electronic device 600. The second QCFA sensor is operating in the first mode (binning mode).

[0121] When the zoom level of the previewed content meets the third zoom level standard (for example, Fig. 13B , 8X zoom level) and the light condition of the previewed content meets the light standard (normal light or bright light), the electronic device 600 can switch the second QCFA sensor to the Bayer image regeneration cropping mode to enhance the resolution of the content previewed in the field of view of the second QCFA sensor. In order to operate the second QCFA sensor in the Bayer image regeneration cropping mode, the second QCFA sensor can be switched to the second mode (Bayer image regeneration mode) and can be center-cropped by the electronic device 600 (i.e., the content previewed in the second mode of the second QCFA sensor is cropped to half the width and half the height).

[0122] At 8X scaling, the second QCFA sensor (remote QCFA sensor) may be switched to a Bayer image regeneration crop mode to output a 2X lossless scaling of the second QCFA sensor, which translates to a 4*2=8X lossless scaling.

[0123] Fig.14A and Fig. 14B An example scenario showing digital zoom solution enhancement at 100X using prior art methods as well as the proposed method is shown in accordance with certain embodiments of the present disclosure.

[0124] Reference Fig. 14B As an illustrative example, a digital zoom method in an existing method can use multiple frames acquired from a sensor as well as a noise reduction method and a frame fusion method to output a final zoomed image output. At least one of the proposed methods for content enhancement (lossless zoom) can be applied with existing digital solutions to enhance image capture capabilities at zoom levels such as 30X, 50X, 100X, etc. Fig. 14BAs shown in FIG, since the proposed method outputs a lossless 2X image, it may only be necessary to apply a 50X scaling to the input image by existing digital solutions in order to obtain a 100X scaled image. Fig. 14B ) shows the improvement in image detail compared to a 100X zoomed image using prior art methods.

[0125] Fig.15A and Fig. 15B An example of enhancing a captured image using digital zoom solutions of both the prior art method and the proposed method at zoom level 30X according to certain embodiments of the present disclosure is shown.

[0126] Fig.16A and 16B An example of enhancing a captured image using digital zoom solutions of both the prior art method and the proposed method at zoom level 100X according to certain embodiments of the present disclosure is shown.

[0127] like Fig. 15B and Fig. 16B As shown in the examples in , images captured with the proposed solution can be more focused, clearer, and retain image details even at 100X zoom level.

[0128] Unlike some existing systems, the proposed method for content enhancement using a QCFA sensor of an electronic device 600 can output lossless image quality at zoom levels of 2X and above, thereby producing high-detail images. In the proposed method, preview images and captured images at zoom levels 2X and above are clearer and more focused. When the subject of the image or the content of the image is clear, the image looks clear and can convey additional details, thereby generating enhanced content (high-detail images) in the field of view of the QCFA sensor. By using multiple QCFA sensors, further lossless images at 8X zoom levels are possible.

[0129] The embodiments disclosed herein can be implemented by at least one software program running on at least one hardware device and performing network management functions to control the elements. The elements shown in Figures 3 to 16 include blocks, elements, actions, actions, steps, etc., which can be at least one of a hardware device or a combination of a hardware device and a software module.

[0130] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can easily modify and / or adjust the various applications of these specific embodiments by applying current knowledge without departing from the general concepts, and therefore, these adjustments and modifications should and are intended to be understood as being within the meaning and scope of the equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for the purpose of description rather than limitation. Therefore, although the embodiments herein have been described according to the preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the spirit and scope of the embodiments as described herein.

[0131] Although the present disclosure has been described with various embodiments, various changes and modifications may be suggested to one skilled in the art. The present disclosure is intended to encompass such changes and modifications as fall within the scope of the appended claims.

Claims

1. A method for an electronic device having a four-color filter array (QCFA) sensor, the method comprising: previewing, by the electronic device, content in a field of view of a first QCFA sensor of the electronic device, wherein, during the preview, the first QCFA sensor is configured to operate in a 1X zoom first mode that combines four adjacent pixels of image data into one pixel and outputs the combined image data to an image processor for display, wherein when configured in the 1X zoom first mode, the first QCFA sensor outputs the combined image data having a first number of megapixels to the image processor for display of an image having the first number of megapixels; In response to detecting that a 2X zoom level is specified for the previewed content, the electronic device detects, by the electronic device, whether a light condition of the previewed content satisfies a light criterion; In response to detecting that light conditions of the previewed content do not meet the light criteria, maintaining the first QCFA sensor in a 1X zoom first mode; and In response to detecting that the light condition of the previewed content meets the light criterion, configuring the first QCFA sensor to switch from a 1X zoom first mode to a 2X zoom second mode, wherein when configured in the 2X zoom second mode, the first QCFA sensor regenerates pixels of the image data from a quad-Bayer pattern Bayer image into a Bayer pattern, crops a center portion of the image data regenerated from the Bayer image into respective width and height dimensions that are half the width dimension of the image data regenerated from the Bayer image and half the height dimension of the image data regenerated from the Bayer image, and Wherein, when configured in the 2X zoom second mode, the first QCFA sensor outputs cropped image data having a first number of megapixels to the image processor for display of an image having the first number of megapixels.

2. The method of claim 1, further comprising: After configuring the first QCFA sensor to switch from the 1X scaling first mode to the 2X scaling second mode, The electronic device determines whether the zoom level of the previewed content satisfies a second zoom level criterion.

3. The method of claim 2, further comprising: In response to the zoom level of the previewed content satisfying a second zoom level criterion, previewing, by the electronic device, the content in a field of view of a second QCFA sensor, wherein the second QCFA sensor is operating in a 1X zoom first mode; detecting, by the electronic device, whether a zoom level of the content previewed in the field of view of the second QCFA sensor satisfies a third zoom level standard, and detecting, by the electronic device, whether a light condition of the content previewed in the field of view of the second QCFA sensor satisfies the light standard; In response to detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor satisfies a third zoom level criterion and that the light condition of the content previewed in the field of view of the second QCFA sensor satisfies the light criterion, configuring the second QCFA sensor to switch from a 1X zoom first mode to a 2X zoom second mode and previewing the content in the field of view of the second QCFA sensor by the electronic device, wherein the second QCFA sensor operates in the 2X zoom second mode; and In response to detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor does not satisfy the third zoom level standard and the light conditions of the content previewed in the field of view of the second QCFA sensor do not satisfy at least one of the light standards, the second QCFA sensor is maintained in the 1X zoom first mode.

4. An electronic device comprising: The first four-color filter array QCFA sensor; Communication circuits; Memory; as well as At least one processor coupled to the memory and the communication circuitry, wherein one or more of the at least one processor is configured to: previewing content in a field of view of a first QCFA sensor of the electronic device, wherein, during the preview, the first QCFA sensor is configured to operate in a 1X zoom first mode that combines four adjacent pixels of image data into one pixel and outputs the combined image data to an image processor for display, wherein when configured in the 1X zoom first mode, the first QCFA sensor outputs the combined image data having a first number of megapixels to the image processor for display of an image having the first number of megapixels; In response to detecting that a 2X zoom level is specified for the previewed content, detecting whether a light condition of the previewed content satisfies a light criterion; In response to detecting that light conditions of the previewed content do not meet the light criteria, maintaining the first QCFA sensor in a 1X zoom first mode; and In response to detecting that the light condition of the previewed content meets the light criterion, configuring the first QCFA sensor to switch from a 1X zoom first mode to a 2X zoom second mode, wherein when configured in the 2X zoom second mode, the first QCFA sensor regenerates pixels of the image data from a quad-Bayer pattern Bayer image into a Bayer pattern, crops a center portion of the image data regenerated from the Bayer image into respective width and height dimensions that are half the width dimension of the image data regenerated from the Bayer image and half the height dimension of the image data regenerated from the Bayer image, and Wherein, when configured in the 2X zoom second mode, the first QCFA sensor outputs cropped image data having a first number of megapixels to the image processor for display of an image having the first number of megapixels.

5. The electronic device as claimed in claim 4, wherein: After configuring the first QCFA sensor to switch from a 1X scaling first mode to a 2X scaling second mode, the processor is further configured to: A determination is made as to whether the zoom level of the previewed content satisfies a second zoom level criterion.

6. The electronic device of claim 5, further comprising a second QCFA sensor, and in, The processor is further configured to: In response to the zoom level of the previewed content satisfying a second zoom level criterion, previewing the content in a field of view of a second QCFA sensor, wherein the second QCFA sensor is operating in a 1X zoom first mode; detecting whether a zoom level of the content previewed in the field of view of the second QCFA sensor satisfies a third zoom level criterion, and detecting whether a light condition of the content previewed in the field of view of the second QCFA sensor satisfies the light criterion; In response to detecting that a zoom level of content previewed in the field of view of the second QCFA sensor satisfies a third zoom level criterion and that light conditions of the content previewed in the field of view of the second QCFA sensor satisfy the light criterion, configuring the second QCFA sensor to switch from a 1X zoom first mode to a 2X zoom second mode and preview the content in the field of view of the second QCFA sensor, wherein the second QCFA sensor operates in the 2X zoom second mode; and In response to detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor does not satisfy the third zoom level standard and the light conditions of the content previewed in the field of view of the second QCFA sensor do not satisfy at least one of the light standards, the second QCFA sensor is maintained in the 1X zoom first mode.

7. A non-transitory computer-readable medium comprising instructions, wherein: When the instructions are executed by at least one processor of the electronic device, the electronic device performs the following operations: previewing content in a field of view of a first QCFA sensor of the electronic device, wherein, during the preview, the first QCFA sensor is configured to operate in a 1X zoom first mode that combines four adjacent pixels of image data into one pixel and outputs the combined image data to an image processor for display, wherein when configured in the 1X zoom first mode, the first QCFA sensor outputs the combined image data having a first number of megapixels to the image processor for display of an image having the first number of megapixels; In response to detecting that a 2X zoom level is specified for the previewed content, detecting whether a light condition of the previewed content satisfies a light criterion; In response to detecting that light conditions of the previewed content do not meet the light criteria, maintaining the first QCFA sensor in a 1X zoom first mode; and In response to detecting that the light condition of the previewed content meets the light criterion, configuring the first QCFA sensor to switch from a 1X zoom first mode to a 2X zoom second mode, wherein when configured in the 2X zoom second mode, the first QCFA sensor regenerates pixels of the image data from a quad-Bayer pattern Bayer image into a Bayer pattern, crops a center portion of the image data regenerated from the Bayer image into respective width and height dimensions that are half the width dimension of the image data regenerated from the Bayer image and half the height dimension of the image data regenerated from the Bayer image, Wherein, when configured in the 2X zoom second mode, the first QCFA sensor outputs cropped image data having a first number of megapixels to the image processor for display of an image having the first number of megapixels.

8. The computer-readable medium of claim 7, further comprising instructions that, when executed by the processor, cause the electronic device to perform the following operations after configuring the first QCFA sensor to switch from the 1X scaling first mode to the 2X scaling second mode: A determination is made as to whether the zoom level of the previewed content satisfies a second zoom level criterion.

9. The computer-readable medium of claim 8, further comprising instructions that, when executed by the processor, cause the electronic device to: In response to the zoom level of the previewed content satisfying a second zoom level criterion, previewing the content in the field of view of the second QCFA sensor, wherein: The second QCFA sensor is operating in the 1X zoom first mode; detecting whether a zoom level of the content previewed in the field of view of the second QCFA sensor satisfies a third zoom level criterion, and detecting whether a light condition of the content previewed in the field of view of the second QCFA sensor satisfies the light criterion; in response to detecting that a zoom level of content previewed in the field of view of the second QCFA sensor satisfies a third zoom level criterion and that a light condition of the content previewed in the field of view of the second QCFA sensor satisfies the light criterion, configuring the second QCFA sensor to switch from a 1X zoom first mode to a 2X zoom second mode and preview the content in the field of view of the second QCFA sensor, wherein the second QCFA sensor operates in the 2X zoom second mode; as well as In response to detecting that the zoom level of the content previewed in the field of view of the second QCFA sensor does not satisfy the third zoom level standard and the light conditions of the content previewed in the field of view of the second QCFA sensor do not satisfy at least one of the light standards, the second QCFA sensor is maintained in the 1X zoom first mode.

Images