Image processing method and electronic device

Abstract

The present application provides an image processing method and an electronic device. In an electronic device equipped with an image sensor having a high pixel array, an image processing method performed by the electronic device comprises: firstly, on the basis of a target zoom ratio and illumination, preprocessing, in a RAW domain, a raw image captured by the image sensor to obtain a preprocessed image in a target pixel format; and then, on the basis of the target zoom ratio and a target size, enlarging and cropping the preprocessed image in any one or more of the RAW domain, a RGB domain or a YUV domain, to obtain a high-quality image of the target size at the target zoom ratio.

Description

Image processing method and electronic device Technical Field

[0001] The present application relates to the field of terminals, and in particular to an image processing method and electronic device. Background Art

[0002] Currently, electronic devices support an increasing number of shooting methods, such as supporting image capture at different zoom ratios. How to process the captured raw images to output high-quality images corresponding to the zoom ratio is an urgent problem to be solved.

[0003] Summary of the Invention

[0004] The present application provides an image processing method and electronic device. In an electronic device equipped with an image sensor having a high pixel array, the electronic device can perform zoom processing on a raw image captured by the image sensor at the original zoom ratio to obtain an image at a target zoom ratio. Specifically, the method includes: first, preprocessing the raw (RAW) image in the RAW domain based on the target zoom ratio and illumination to obtain a preprocessed image in the target pixel format. Then, based on the target zoom ratio and target size, the preprocessed image is magnified and cropped in any one or more of the RAW domain, RGB domain, or YUV domain to obtain a high-quality image of the target size at the target zoom ratio.

[0005] In a first aspect, the present application provides an image processing method, which is applied to an electronic device including an image sensor, the method comprising: acquiring an original image through the image sensor, the zoom ratio of the original image being greater than 1; performing K processing on the original image at the zoom ratio based on a preset strategy to obtain a first image, K≥1; wherein the preset strategy comprises: when the target zoom ratio is in a first range, K=K1; when the target zoom ratio is in a second range and the illuminance is a first illuminance, K=K2; when the target zoom ratio is in the second range and the illuminance is a second illuminance, K=K3; the minimum value in the second range is greater than the maximum value in the first range, the first illuminance is greater than the second illuminance, and K2<K3≤K1; and enlarging the first image to obtain a second image of the target zoom ratio.

[0006] After implementing the method provided in the first aspect, the original image can be pixel-merged in combination with the target zoom ratio and illumination selected by the user. This not only improves the noise resistance of the image but also preliminarily performs zoom ratio processing on the original image. Then, after further zooming based on the preliminary zoom ratio processing, a high-quality image at the target zoom ratio is obtained.

[0007] In the present application, the zoom ratio corresponding to the original image output by the image sensor is greater than 1 times, and the zoom ratio corresponding to the original image is a shooting parameter set before the electronic device leaves the factory. The target zoom ratio is the shooting parameter selected by the user in the shooting scene or the shooting parameter used by the electronic device by default. The target zoom ratio may be greater than, less than or equal to the zoom ratio of the original image, so the image processing method provided in the present application needs to process the zoom ratio of the original image into the target zoom ratio. When the zoom ratio of the original image is N times and the target zoom ratio is Z, the image processing specifically includes: based on a preset strategy, the original image is processed from a zoom ratio of N times by K×K pixels into 1 pixel merging to obtain a first image with a zoom ratio of N / K times, and then, based on the first image with a zoom ratio of N / K times, the first image is magnified by Z / (N / K) times to obtain an image with a target zoom ratio.

[0008] In combination with the method provided in the first aspect, the preset strategy also includes: when the target zoom ratio is in the third range, if the illuminance is the third illuminance, K=K4, if the illuminance is the fourth illuminance, K=K5, if the illuminance is the fifth illuminance, K=K6; the third range is greater than the second range, the third illuminance is greater than the fourth illuminance, the fourth illuminance is greater than the fifth illuminance, and K4<K5<K6≤K1.

[0009] In this way, by dividing the target zoom ratio into multiple ranges, a more precise preset strategy can be formulated based on the range of the target zoom ratio and the illumination, thereby further improving the zoom efficiency and ensuring the quality of the zoomed image.

[0010] In combination with the method provided in the first aspect, after obtaining the first image, the method also includes: converting the first image from the RAW domain to the RGB domain or the YUV domain through a neural network; and amplifying the first image, specifically including amplifying the first image at any one or more of the following stages: before, during, or after the domain conversion of the first image through the neural network.

[0011] In this way, the image can be magnified in any one or more of the multiple stages according to the magnification requirements to obtain an image at a target zoom ratio, thereby improving the feasibility of the image processing method.

[0012] In combination with the method provided in the first aspect, when the target zoom ratio is larger, the first image is graded and magnified through more stages.

[0013] In this way, when the target zoom ratio is larger, the image magnification needs to be higher. Therefore, through graded magnification, the stability of the magnification effect can be guaranteed, and the difficulty of magnification in only one stage can be reduced.

[0014] In combination with the method provided in the first aspect, amplifying the first image before performing domain conversion on the first image through the neural network includes: amplifying the first image through the image sensor, and / or amplifying the first image through an algorithm.

[0015] In this way, the hardware amplification capability of the image sensor and the software amplification capability of the electronic device can be fully utilized. The hardware amplification capability of the image sensor can be, for example, in-sensor zoom, and the software amplification capability can be, for example, upsampling algorithm or super-resolution.

[0016] In combination with the method provided in the first aspect, the pixel format of the original image is N×N Quad-Bayer, and the pixel format corresponding to the first image is M×M Quad-Bayer, where N≥M, N>1, and M≥1.

[0017] In this way, by converting the pixel format of the original image, an image with better noise resistance can be obtained, and the output image under low illumination is still clear.

[0018] In combination with the method provided in the first aspect, the zoom magnification of the original image is N times, and the zoom magnification of the first image is N / K times.

[0019] In this way, the original image can be subjected to preliminary lossless zoom processing according to a preset strategy.

[0020] In combination with the method provided in the first aspect, when the target zoom ratio is Z, the first image is magnified, specifically including: magnifying the first image by C times, where C=Z / (N / K).

[0021] In this way, on the basis of the preliminary lossless zoom processing, the image at the target zoom ratio can be obtained by magnification.

[0022] In combination with the method provided in the first aspect, before obtaining the second image, the method further includes: cropping the first image before or after amplifying the first image.

[0023] In this way, an image of the target size selected by the user can be obtained.

[0024] In combination with the method provided in the first aspect, before performing K×K pixel-to-pixel processing on the original image based on a preset strategy, the method further includes: detecting the illuminance through the light sensor of the electronic device, or obtaining the illuminance through the exposure parameters of the image sensor.

[0025] In this way, the illumination of the scene to be photographed can be obtained in a variety of ways, thereby improving the feasibility of this solution.

[0026] In combination with the method provided in the first aspect, before performing K×K pixel-to-pixel processing on the original image based on a preset strategy, the method also includes: receiving a first operation, and in response to the first operation, setting the zoom ratio of the electronic device to the target zoom ratio.

[0027] In this way, the original image can be processed based on the target zoom magnification selected by the user, and a high-quality image at the target zoom magnification selected by the user can be obtained.

[0028] In combination with the method provided in the first aspect, the method further includes: displaying a preview interface, and displaying the second image in the preview interface.

[0029] In this way, the image processing method can also be applied in the image preview scenario to ensure the preview effect.

[0030] In combination with the method provided in the first aspect, the original image includes multiple frames of images, and the first image refers to: an image frame obtained after fusion processing of multiple frames of images that are processed with K×K pixels into 1 based on a preset strategy; the method also includes: receiving a photo operation before the original image is processed with K×K pixels into 1 based on the preset strategy; saving the second image, displaying a gallery interface, and displaying the second image in the gallery interface.

[0031] In this way, in a shooting scene, multiple frames of images processed by a preset strategy can be fused, and the photo quality can be further improved during the zoom processing process.

[0032] In a second aspect, the present application provides an electronic device comprising an image sensor, one or more memories, and one or more processors; the memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code comprising computer instructions, the one or more processors calling the computer instructions to enable the electronic device to execute a method as described in any one of the first aspects.

[0033] In a third aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on an electronic device, causes the electronic device to execute the method as described in any one of the first aspects.

[0034] In a fourth aspect, the present application provides a chip, which is applied to an electronic device. The chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute any method described in the first aspect.

[0035] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on an electronic device, enables the electronic device to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram of an image processing flow in a preview channel provided by an embodiment of the present application;

[0037] FIG2 is a schematic diagram of an image processing flow in a shooting channel provided in an embodiment of the present application;

[0038] FIG3 is a flow chart of an image processing method provided in an embodiment of the present application;

[0039] FIG4 is a schematic diagram of various pixel formats and their conversion relationships provided in an embodiment of the present application;

[0040] FIG5 is a schematic diagram of the image size conversion relationship corresponding to pixel format conversion provided by an embodiment of the present application;

[0041] FIG6 is a schematic diagram of an image output strategy involved in the image processing method provided in this application;

[0042] FIG7 is a schematic diagram of a specific method for implementing a graph output strategy provided by this application;

[0043] FIG8 is a schematic diagram of the hardware architecture of the electronic device provided in this application;

[0044] FIG9 is a schematic diagram of the system architecture of the electronic device provided in this application. DETAILED DESCRIPTION

[0045] The following is a clear and detailed description of the technical solutions in the embodiments of the present application, with reference to the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " represents the meaning of "or." For example, A / B can represent A or B. "and / or" in the text is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone.

[0046] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0047] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0048] The term "user interface (UI)" in the following embodiments of this application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0049] First, several related concepts involved in the embodiments of this application are introduced.

[0050] Zoom, also known as magnification, is a shooting parameter for electronic devices. Specifically, it refers to the multiple of the standard focal length of the device. This standard focal length can be the focal length supported by a fixed-focus camera (main camera or wide-angle camera) or one of the focal lengths supported by a telephoto camera. A larger zoom ratio increases the magnification of the captured image.

[0051] Digital zoom refers to achieving an effect similar to changing the focal length by upsampling and super-resolution processing the images captured by the camera without changing the focal length of the camera.

[0052] There are two ways of optical zoom. The first way is that when an electronic device is equipped with multiple cameras with different focal lengths or focal lengths, zoom is achieved by relaying multiple cameras. That is, by switching between multiple cameras with different focal lengths, cameras with different focal lengths are used to capture the preview stream and display it. The second way is that when a telephoto camera is available, the telephoto camera is pushed closer or farther away so that the telephoto camera captures the preview stream at different positions (at different focal lengths) and displays it. Since the first optical zoom method involves switching cameras, this will cause the preview stream to be interrupted. Since the second optical zoom method is to change the focal length by controlling the fixed camera to advance or pull away, a smooth and continuous zoom process can be achieved, that is, there will be no interruption in the preview stream.

[0053] The electronic device provided by the present application can realize the shooting of images at different zoom ratios through optical zoom and / or digital zoom, and particularly relates to the shooting of images at different zoom ratios through digital zoom. Regardless of whether digital zoom or optical zoom is used, the similarities between the two are: when the zoom ratio becomes smaller, the size of the preview image displayed on the preview interface of the electronic device and the image captured in the gallery remains unchanged, but the field of view angle becomes larger, and the picture content becomes more and smaller. Correspondingly, when the zoom ratio becomes larger, the size of the preview image displayed on the preview interface of the electronic device and the image captured in the gallery remains unchanged, but the preview image shows an effect of a smaller field of view angle, and the picture content becomes less and larger. However, the difference between the two is that: only optical zoom can support not reducing pixels, that is, the picture content becomes larger but the picture quality remains clear, while digital zoom reduces pixels, that is, the picture content becomes larger but the picture clarity decreases.

[0054] Color spaces include but are not limited to three-primary color space, referred to as (RGB), and brightness and color difference color space, referred to as (YUV). Among them, RGB is a color space that describes colors through the three primary colors of red, green, and blue, R = Red, G = Green, B = Blue, and RGB focuses on the sensitivity of the human eye to color. YUV is a color space that describes colors through brightness and color difference. Brightness is called Y, and chrominance is composed of two independent signals, namely U and V. YUV focuses on the visual sensitivity to brightness. In the embodiment of the present application, RGB and YUV can be converted to each other as needed.

[0055] Pixel format refers to the arrangement format of the pixel array. When the pixel array arrangement format set in the image sensor is different, the pixel format of the output raw image is different. In addition, the pixel format of the raw image can also be converted in the RAW domain to obtain images of other pixel formats.

[0056] The pixel format is not limited to: Bayer format, Quad-Bayer format. The pixel format described in this application is for the image in the original domain (RAW domain), not for the RGB domain / YUV domain processed image format after conversion.

[0057] Pixel format conversion technologies include but are not limited to: pixel merging technology (also known as Binning), and Remosaic technology, etc.

[0058] Regarding the pixel format conversion relationship and the corresponding image size transformation relationship during pixel format conversion, please refer to the detailed description of Figures 4 and 5 below, which will not be repeated here.

[0059] Most current electronic devices support zoom, allowing users to capture images at varying zoom ratios. However, due to limitations in size and cost, these devices often lack optical zoom capabilities or are unable to achieve a wide range of zoom. Consequently, most electronic devices rely on digital zoom, but the quality of images currently captured using digital zoom is often poor.

[0060] To this end, the present application provides an image processing method and electronic device. In an electronic device equipped with an image sensor having a high-pixel array, the electronic device performs image processing, including: first, preprocessing a raw (RAW) image captured by the image sensor in the RAW domain based on a target zoom ratio and illumination, thereby obtaining a preprocessed image in a target pixel format. Then, based on the target zoom ratio and target size, the preprocessed image is magnified and cropped in any one or more of the RAW domain, the RGB domain, or the YUV domain, thereby obtaining a high-quality image of the target size at the target zoom ratio.

[0061] The image obtained after preprocessing may also be referred to as a first image, and the image obtained after magnification and cropping may also be referred to as a second image.

[0062] Among them, the target zoom ratio is the zoom ratio adopted by the electronic device by default or selected by the user. The target size is the image size adopted by the electronic device by default or selected by the user. Usually, the target size is a fixed value, which is used to provide the user with an image of a uniform size. Illumination refers to the illumination of the scene to be photographed, that is, the degree to which the scene to be photographed is illuminated, specifically refers to the energy of visible light received per unit area of ​​the object. If the illumination of the scene to be photographed is relatively low, the image obtained by the capture will be blurred and the effect will be poor. If the illumination of the scene to be photographed is high, the image obtained by the capture will be clearer and the shooting effect will be good.

[0063] Among them, the pixel array of the original image output by the image sensor with a high pixel array is in N×N Quad-Bayer format (hereinafter referred to as N=4 as an example). The target pixel format of the preprocessed image obtained after preprocessing is in M×M Quad-Bayer format. Among them, 1≤M≤N, and when M=1, the target pixel format is Bayer format. High pixel array is a relative concept, specifically referring to the higher resolution of the original image of the high pixel array relative to the pixels of the image output by the electronic device in conventional shooting mode (shooting mode with a zoom magnification of 1×). In addition, the zoom magnification corresponding to the original image is greater than 1×, and this application is introduced by taking the zoom magnification corresponding to the original image as N× as an example.

[0064] The technologies involved in the pre-processing stage include but are not limited to: pixel merging (also known as binning) and pixel remosaicing.

[0065] (1) Binning is a pixel format conversion technology that is achieved by merging adjacent pixels on an image sensor. This technology has the following effects: in low illumination, it can improve light sensitivity, reduce image noise, and improve image clarity, and the more pixels are merged, the better the noise resistance; when acquiring image streams, the amount of image data can be reduced to obtain a higher frame rate; in image processing, the image output resolution is reduced to save processing resources. Common Binning modes include the following: 2×2 in 1 Binning, that is, merging every 2×2 pixels into 1 pixel, and 3×3 in 1 Binning, that is, merging every 3×3 pixels into 1 pixel, and 4×4 in 1 Binning, that is, merging every 4×4 pixels into 1 pixel, etc., and the embodiments of this application are not limited to this.

[0066] (2) Remosaic is a pixel format conversion technology that converts the original image's pixel array arrangement format into a traditional Bayer format. This technology has the following effects: outputting ultra-high-pixel images; and enabling the ISP to process images directly according to the standard Bayer format.

[0067] For the specific implementation methods of Binning and Remosaic, please refer to the introduction of Figures 3 and 4 later, which will not be repeated here.

[0068] Among them, magnification and cropping mean that due to the limited image processing capabilities of the preprocessing stage, it can only process a few images of fixed size and fixed zoom ratio. That is, the preprocessed image output after preprocessing may not meet the requirements of the target zoom ratio and target size. Therefore, it is necessary to further magnify the image based on the target zoom ratio and crop the required content based on the target size.

[0069] For details about the image processing method provided in this application, please refer to the detailed description of Figures 1 and 2 below, which will not be repeated here.

[0070] In summary, the image processing method provided in this application includes a set of image output strategies, which can be divided into the following two stages:

[0071] In the preprocessing stage, the two factors of zoom ratio and illumination are mainly combined to determine the pixel format conversion technology to be used. In this way, it is possible to ensure that the preprocessed image with high noise resistance performance is output under any illumination, and at the same time, preliminary zoom ratio processing can be performed, simplifying the workload of cropping and scaling based on the target zoom ratio in the subsequent stage.

[0072] In the zoom and cropping stage, it is only necessary to further zoom in and crop the image of the target zoom ratio and target size based on the zoom ratio and image size corresponding to the pre-processed image according to the target zoom ratio and target size. Moreover, the zooming and cropping can be performed in any one or more domains of the RAW domain, YUV domain or RGR domain. When the present application provides a neural network that will convert an image from the RAW domain to the YUV domain or the RGR domain, the zooming and cropping involved in the present application can be set at any one or more places before, during or after the neural network. In particular, when the target zoom ratio is high, the image needs to be magnified by a higher factor, and by dispersing the magnification in multiple domains, it is helpful to output an image with more stable image quality. When the zoom ratio is low, the image needs to be magnified by a smaller factor, and the image of the target zoom ratio can be obtained by performing grading at any one place.

[0073] Regarding the output strategy (also known as the preset strategy) involved in the image processing process provided in this application, including the preprocessing strategy and the subsequent magnification and cropping strategy, please refer to the detailed description of Figures 6-7 later, which will not be repeated here.

[0074] Next, the image processing method provided by this application will be introduced with reference to FIG1 to FIG3 .

[0075] FIG1 exemplarily shows a schematic diagram of the image processing flow in the preview channel provided by this application.

[0076] As shown in Figure 1, in the preview path, the original image stream output by the image sensor is processed in sequence by the general pre-processing module, the magnification module, and the cropping module, and finally outputs the image stream at the target zoom ratio and target size for display.

[0077] 1. The image sensor is used to capture light signals and output a raw image. The pixel format and image size of the raw image are determined by the hardware design of the image sensor. For example, if the pixel array in the image sensor is arranged in a 4x4 Quad-Bayer format, the pixel array size contains 4*4096×4*3072 pixels. The pixel format of the raw image stream is 4x4 Quad-Bayer, and the image size is 4*4096×4*3072.

[0078] 2. The general preprocessing module is used to preprocess the original image in the preview and capture paths. This preprocessing includes determining a preprocessing strategy based on the target zoom factor and illumination, and then processing the original image using binning and / or remosaicing techniques corresponding to the preprocessing strategy to output a preprocessed image. Preprocessed images obtained through varying degrees of binning will have their pixel format changed, resolution reduced, and image size reduced. Preprocessed images obtained through remosaicing alone will have their pixel format changed but their image size maintained. For example, if the pixel format of the original image is 4x4 Quad-Bayer and the image size is 4*4096×4*3072, if it is processed by 16-in-1 binning, the pixel format of the preprocessed image is Bayer and the pixel size is 4096×3072; if it is processed by 4-in-1 binning, the pixel format of the preprocessed image is 2x2 Quad-Bayer Bayer and the pixel size is 2*4096×2*3072; if it is processed by Remosaic, the pixel format of the preprocessed image is Bayer and the pixel size is 4*4096×4*3072.

[0079] Regarding the implementation of Binning and Remosaic technology in preprocessing, please refer to the detailed description of Figures 3 and 4 later, which will not be repeated here.

[0080] Regarding the method for determining the preprocessing strategy, please refer to the detailed description of FIG5 later, which will not be described here.

[0081] 3. The magnification module is used to magnify the pre-processed image content to the target zoom ratio based on the target zoom ratio. In the embodiment of the present application, the magnification module can be set at any one or more locations before, during, or after the neural network, and the embodiment of the present application is not limited to this. The neural network is the algorithm module for converting the image from the RAW domain to the YUV domain or the RGR domain described below.

[0082] 4. The cropping module is used to crop the desired content from the pre-processed image or the enlarged pre-processed image based on the target size. In the embodiment of the present application, the cropping module can be set independently from the enlargement module, or it can be integrated with the enlargement module into a module with both enlargement and cropping functions.

[0083] The reason for setting up the cropping and enlargement modules is that, given an original image with a 4x4 Quad-Bayer pixel format and an image size of 4*4096×4*3072, the preprocessed images obtained through binning to varying degrees will have their resolution and size reduced. However, the preprocessed images obtained through remosaicing alone do not have their resolution reduced and their size remains unchanged. While binning can compress images to some extent, its compression capability is limited and can only be performed by integer multiples. Users can select a wide range of zoom factors. If the user selects a small zoom factor, compression may only be performed through binning during the preprocessing phase. However, if the user selects a large zoom factor, enlargement will also be required, and the zoom factor may not be an integer multiple. Therefore, preprocessing alone may not be sufficient to directly obtain an image with the target zoom factor or size. Therefore, after preprocessing, the preprocessed image needs to be cropped and enlarged according to the target zoom factor and image size.

[0084] Optionally, the present application does not limit the processing order of the above-mentioned cropping module and magnification module in the preview channel. In addition to first cropping the image through the cropping module and then enlarging the image through the magnification module, in another possible embodiment, the image can be first enlarged to the size corresponding to the zoom ratio through the enlargement module, and then the image of the target size can be cropped through the cropping module.

[0085] Optionally, there may be one or more cropping modules and one or more magnification modules, respectively, and the cropping modules and magnification modules may be set in any one or more of the following domains: in the RAW domain, in the RGB domain, and in the YUV domain. In an embodiment of the present application, a neural network may be used to convert an image in the RAW domain into an image in the RGB domain or an image in the YUV domain. Therefore, when the cropping module and the magnification module are set in the RAW domain, they may be specifically set outside the neural network. When the cropping module and the magnification module are set in the RGB domain or the YUV domain, they may be specifically set inside or outside the neural network.

[0086] Preferably, when the target zoom ratio is small, the image can be cropped and magnified directly through the neural network. This fully utilizes the neural network's capabilities and eliminates the need for additional cropping and magnification functions at other stages. When the target zoom ratio is large, the image can be cropped and magnified separately through the neural network's front, middle, and back stages. This avoids instability when magnifying large magnifications solely through the neural network.

[0087] FIG2 exemplarily shows a schematic diagram of the image processing flow in the photo-taking channel provided by this application.

[0088] As shown in Figure 2, in the photo-taking path, the original image output by the image sensor is processed in sequence by the general pre-processing module, the photo-taking pre-processing module, the cropping module, and the magnification module, and finally outputs a photo of the target size at the target zoom ratio.

[0089] The photography pipeline shown in Figure 2 differs from the preview pipeline shown in Figure 1 in that a pre-processing module is added. This is because users have higher requirements for photo quality than for preview images. Therefore, the pre-processing module must perform additional image operations in the photography pipeline, including but not limited to multi-frame image registration to produce a high-quality fused image.

[0090] Regarding the functions of other modules in the photo-taking channel, please refer to the description of the corresponding modules in Figure 1 above, which will not be repeated here.

[0091] FIG3 exemplarily shows the flow of the image processing method provided by this application.

[0092] As shown in FIG3 , the method includes the following steps:

[0093] S31: The electronic device receives an operation for starting a shooting function.

[0094] Specifically, the operation received by the electronic device for starting the shooting function includes but is not limited to: turning on the electronic device and turning on a camera application in the electronic device.

[0095] S32: The electronic device obtains the illumination and target zoom ratio of the current shooting scene.

[0096] Specifically, after receiving an operation for starting a shooting function, the electronic device may detect the illumination of the current shooting scene and the target zoom ratio set in the electronic device.

[0097] The electronic device may detect the illumination of the scene to be captured using a light sensor, or may detect the illumination of the scene to be captured based on exposure parameters of an image sensor. Exposure parameters include, but are not limited to, any one or more of the following: exposure time, sensitivity, aperture, and other parameters.

[0098] The target zoom ratio is a default setting of the electronic device, or is set based on a focus adjustment operation input by a user after the shooting function is activated. The focus adjustment operation may be a sliding operation on a corresponding control on the display screen of the electronic device, a clicking operation, a two-finger pinching operation, etc., and the present embodiment does not impose any limitation on this.

[0099] S33, the electronic device performs general preprocessing on the original image stream collected by the image sensor according to the illumination and the target zoom ratio to obtain a general preprocessed image stream.

[0100] The specific electronic device stores a preset strategy that includes processing mechanisms corresponding to different illumination levels and target zoom ratios. For an introduction to general preprocessing, please refer to the previous description of the general preprocessing module in Figures 1 and 2, or the preprocessing strategy used in the preprocessing stage described in Figures 6 and 7.

[0101] S34 , the electronic device zooms in on the general pre-processed image stream to a target zoom magnification and outputs a preview image stream.

[0102] Specifically, after acquiring the general pre-processed image stream, the electronic device further magnifies it based on the target zoom magnification, and then outputs the preview image stream at the target zoom magnification in the preview interface.

[0103] Optionally, after acquiring the general pre-processed image stream, the electronic device may further crop it, and then output a preview image stream of a target size.

[0104] The specific implementation of the magnification operation includes but is not limited to upsampling or super-resolution, etc., which is not limited in the embodiments of the present application. For other magnification factors, magnification processing stages, etc., please refer to the magnification and cropping strategies adopted in the magnification and cropping stages described in Figures 6 and 7 below.

[0105] In addition, during the aforementioned image preview process, the electronic device can also take a photo based on a user operation, and the specific implementation includes the following steps:

[0106] S35, the electronic device receives a photo-taking operation.

[0107] Specifically, the photo-taking operation may occur at any time after the aforementioned step S31. The photo-taking operation includes, but is not limited to, a shutter operation in a camera application, a physical button operation on an electronic device, etc., which is not limited in the present embodiment.

[0108] S36, the electronic device performs photo preprocessing on the multiple frames of general preprocessed images, including image fusion.

[0109] Specifically, after general preprocessing multiple frames of original images, the electronic device can perform photo preprocessing on multiple frames of general preprocessed images corresponding to before and after the photo operation to obtain a frame of image. The photo preprocessing includes but is not limited to fusion, registration, etc.

[0110] S37, zooming in on the pre-processed image to a target zoom ratio and outputting the photo.

[0111] Optionally, before or after the pre-processed image is enlarged to the target zoom ratio, a cropping operation may be performed to make the final output photo the target size. Furthermore, the electronic device may also save the photo in a gallery for the user to open the gallery and view the photo.

[0112] Based on the image processing flow described in the above FIG1-FIG3, the implementation of the Binning and Remosaic technologies involved in the pre-processing module will be described in detail in conjunction with FIG4-FIG5.

[0113] Referring to FIG4 , FIG4 exemplarily shows a schematic diagram of multiple pixel formats and their conversion relationships.

[0114] 4 a shows a unit pixel array with a 4×4 Quad-Bayer pixel format, which includes four 4×4 pixels, specifically two 4×4 green pixels, one 4×4 blue pixel, and one 4×4 red pixel.

[0115] 4b shows a unit pixel array with a 3×3 Quad-Bayer pixel format, which includes four 3×3 pixels, specifically two 3×3 green pixels, one 3×3 blue pixel, and one 3×3 red pixel.

[0116] 4 c shows a unit pixel array with a 2×2 Quad-Bayer pixel format. The unit pixel array includes four 4×4 pixels, specifically two 2×2 green pixels, one 2×2 blue pixel, and one 2×2 red pixel.

[0117] 4 d-1 shows a unit pixel array with a pixel format of 1×1 Quad-Bayer (equivalent to standard Bayer). The unit pixel array includes 4 pixels, specifically 2 green pixels, 1 blue pixel, and 1 red pixel.

[0118] d-2 in FIG. 4 shows four unit pixel arrays with a pixel format of 1×1 Quad-Bayer (equivalent to standard Bayer). The arrangement of each unit pixel array is the same as d-1 in FIG. 4 .

[0119] Conversions between the above pixel formats include:

[0120] After 4-in-1 Binning of the 4×4 Quad-Bayer shown in Figure 4a, the 2×2 Quad-Bayer shown in Figure 4c can be obtained;

[0121] After 16-in-1 Binning of the 4×4 Quad-Bayer shown in Figure 4a, the 1×1 Quad-Bayer shown in Figure 4d-1 can be obtained;

[0122] After 4-in-1 Binning of the 2×2 Quad-Bayer shown in Figure 4c, the 1×1 Quad-Bayer shown in Figure 4d-1 can be obtained;

[0123] After remosaicing the 2×2 Quad-Bayer shown in Figure 4c, we can get the 1×1 Quad-Bayer shown in Figure 4d-2;

[0124] After 9-in-1 Binning of the 3×3 Quad-Bayer shown in FIG4b, a 1×1 Quad-Bayer as shown in FIG4d-1 can be obtained.

[0125] Based on the content shown in Figure 4, it can be seen that when using Binning to process an image, in addition to changing the pixel format, it also involves reducing the pixels (that is, reducing the image size), and different Binning modes will reduce the image size to different degrees. When using Remosaic, there is no change in the image size.

[0126] FIG5 is a schematic diagram of the pixel size conversion relationship corresponding to the pixel format conversion provided in an embodiment of the present application.

[0127] FIG. 5 a shows an image with a pixel size of 4096*4×3072*4 and a pixel format of 4×4 Quad-Bayer as shown in FIG. 4 a.

[0128] After the image shown in FIG5a is processed by Remosaic, an image with a pixel size of 4096*4×3072*4 and a pixel format of 1×1 Quad-Bayer can be obtained as shown in FIG5b.

[0129] By subjecting the image shown in FIG5a to 16-in-1 binning processing, an image shown in FIG5c with a pixel size of 4096×3072 and a pixel format of 1×1 Quad-Bayer can be obtained.

[0130] After the image shown in FIG5a is subjected to 4-in-1 binning processing, an image with a pixel size of 4096*2×3072*2 and a pixel format of 2×2 Quad-Bayer can be obtained as shown in FIG5d.

[0131] After the image shown in FIG5d is subjected to Remosaic processing, an image shown in FIG5e can be obtained, in which the pixel size is still 4096*2×3072*2, but the pixel format is changed to 1×1 Quad-Bayer.

[0132] In summary, remosaicing involves changing the pixel format during preprocessing. Binning involves changing the pixel format and reducing the pixel size (equivalent to reducing the image size and resolution). Different levels of binning result in varying degrees of pixel size reduction.

[0133] In the embodiment of the present application, the 4×4 Quad-Bayer shown in FIG4a is temporarily used as the color filter arrangement format on the image sensor, and is also used as the pixel format of the original image output by the image sensor, and the 4096*4×3072*4 shown in FIG5a is used as the pixel size of the original image output by the image sensor. The above FIG4-FIG5 only shows a few pixel formats, and also only shows a few ways to achieve pixel format conversion through Binning and / or Remosaic, and should not constitute a limitation of the present application. When the color filter arrangement format of the image sensor (the arrangement of the pixel array) is different, for example, when it is RGBW, pixel format conversion can also be achieved through other more technologies.

[0134] In one possible implementation provided herein, the binning and / or remosaic technology implemented by the preprocessing module can be integrated into an image sensor, equivalent to the image sensor being able to output images in multiple pixel formats. Specifically, the image sensor can select the corresponding binning and / or remosaic output mode based on the preprocessing strategy in the output strategy, thereby outputting images in multiple pixel formats.

[0135] In another possible implementation provided by this application, the aforementioned binning and / or remosaic technology implemented by the pre-processing module can be integrated into the image processing path after the image sensor. That is, the same image sensor in the electronic device only supports outputting images in a fixed pixel format, and the pre-processing module then performs pixel format conversion on this fixed pixel format image to obtain images in other pixel formats. For example, taking the image sensor of a telephoto camera as an example, the image sensor only supports outputting the 4×4 Quad-Bayer pixel format as shown in a in Figure 4. The subsequent pre-processing module performs 4-in-1 Binning processing on the 4×4 Quad-Bayer pixel format to obtain the 2×2 Quad-Bayer pixel format shown in c in Figure 4. If the 2×2 Quad-Bayer pixel format is further subjected to Remosaic processing, the Bayer pixel format shown in d-2 in Figure 4 is obtained. If the 2×2 Quad-Bayer pixel format is further subjected to 2-in-1 Binning processing, the Bayer pixel format shown in d-1 in Figure 4 is obtained; if the 4×4 Quad-Bayer pixel format is directly subjected to 16-in-1 Binning processing, the Bayer pixel format shown in d-1 in Figure 4 is obtained, and so on.

[0136] Based on the Binning and Remosaic technology implementation involved in the preprocessing module introduced in Figures 4 and 5 above, the image output strategy used in the image processing method will be specifically introduced in conjunction with Figures 6 and 7. It mainly includes: in the preprocessing stage, the original image is first pixel-merged according to the target zoom ratio range and the illumination range. This can achieve the purpose of preliminary zoom ratio processing while ensuring the noise resistance of the image contrast. In the zooming and cropping stage, it is only necessary to further zoom in, crop, and other operations on the preprocessed image obtained after the preliminary zoom ratio processing in the preprocessing stage according to the target ratio and target size to obtain the final image corresponding to the target zoom ratio and target size with excellent noise resistance.

[0137] FIG6 exemplarily illustrates the image output strategy involved in the image processing method provided in this application.

[0138] As shown in Figure 6, the image processing method provided in this application includes a preprocessing strategy and a cropping and magnification strategy. The two strategies are not formed independently, but together constitute a complete set of image output strategies for fully serving image processing under a certain zoom ratio and a certain illumination.

[0139] In the embodiments of this application, the pixel format of the original image is N×N Quad-Bayer. As previously described, since the image sensor used in this application has a high pixel array, the original image output by this image sensor has a higher resolution and pixel size than the pixels of the image output by the electronic device in normal shooting mode (shooting mode with a zoom magnification of 1×). In other words, the zoom magnification corresponding to the original image is greater than 1×, and this application uses the zoom magnification of N× corresponding to the original image as an example for description.

[0140] The preprocessing mechanism used to obtain the preprocessed image in the target pixel format based on the focal ratio and illumination includes:

[0141] 1. When the target zoom ratio is in a lower first range, for example, Zoom∈[Z1×, Z2×), no brightness binning is performed. Instead, the original image with the N×N Quad-Bayer pixel format is directly subjected to a K1×K1-to-1 binning operation to obtain a pre-processed image with the target pixel format of M1×M1 Quad-Bayer, where 1<K1≤N and M1=N / K1. This pre-processed image corresponds to a pixel ratio of N / K1. The subsequent zoom and cropping stage requires further processing based on the N / K1 ratio according to the target zoom ratio to obtain an image with the target zoom ratio and target size.

[0142] This is because the zoom ratio in the first range is relatively low; that is, the zoom ratio corresponding to the original image is much greater than the minimum value in this first range. Therefore, given the low zoom ratio, the original image generally requires a greater degree of compression. Furthermore, at low illumination, a more aggressive binning process produces a less noisy image; whereas at higher illumination, the noise in the image obtained without binning is already minimal. Therefore, at higher illumination, the impact of noise does not need to be considered. Only the compression ratio requirements in the second range need to be considered, and a more aggressive binning process is sufficient.

[0143] Therefore, in order to meet both noise requirements and zoom ratio requirements, when the target zoom ratio is in a lower range, such as the first range, there is no need to divide the brightness, and a large degree of binning processing can be directly performed according to the entire scene.

[0144] Taking a specific example, when N=4, Z1=1, and Z2=2, K1 can be 4, then M1=1. That is to say, when Zoom∈[1×, 2×), the original image with a pixel format of 4×4 Quad-Bayer is binned 4×4 to obtain a preprocessed image with a target pixel format of 1×1 Quad-Bayer (equivalent to Bayer). This preprocessing is equivalent to compressing the original image by 4 times, and just obtains a preprocessed image with a zoom ratio of 1×. This 1× preprocessed image has good noise resistance in low-light scenes. In addition, in the subsequent zooming and cropping stage, further processing is required based on the target zoom ratio on the basis of 1× to obtain an image with the target zoom ratio and target size.

[0145] 2. When the target zoom ratio is in the second medium range, for example, Zoom∈[Z2×, Z3×), the illumination can be preliminarily divided into two categories, for example, high illumination and medium-low illumination.

[0146] (1) When the illumination is high, the preprocessing stage directly performs K2×K2-to-1 binning on the original image with the pixel format of N×N Quad-Bayer to obtain a preprocessed image with the target pixel format of M2×M2 Quad-Bayer, where 1<K2<K1, M2=N / K2. The corresponding pixel magnification of this preprocessed image is N / K2. The high illumination refers to, for example, the first illumination.

[0147] (2) When the illumination is low, the preprocessing stage directly performs K3×K3 binning on the original image with the pixel format of N×N Quad-Bayer to obtain a preprocessed image with the target pixel format of M3×M3 Quad-Bayer, where K2<K3≤K1, and M3=N / K3. The corresponding pixel magnification of this preprocessed image is N / K3. Here, the low illumination refers to, for example, the second illumination, where the first illumination is greater than the second illumination.

[0148] This is because the zoom ratio in the second range is a medium zoom ratio, that is, the zoom ratio corresponding to the original image is slightly greater than the minimum value in the second range and less than or equal to the maximum value in the second range. Therefore, under the premise of a medium zoom ratio, the original image needs to be compressed to a medium degree overall. In addition, because at low illumination, medium binning still cannot reduce image noise, a larger degree of binning is required. At high illumination, the noise of the image obtained without binning is already very small. Therefore, at high illumination, there is no need to consider the impact of noise. Only the compression ratio requirement in the second range needs to be considered, and medium binning is sufficient.

[0149] Therefore, while satisfying both noise requirements and zoom ratio requirements, when the zoom ratio belongs to a medium range, such as the second range, the illumination is divided into high illumination and medium and low illumination, and the above-mentioned binning processing of different degrees is performed respectively.

[0150] Taking a specific example, when N = 4, Z2 = 2, and Z3 = 4, if K2 is 2, then M2 = 2, and if K3 is 4, then M3 = 1. That is, when Zoom∈[2×, 4×), in a high-illuminance scene, the original image with a pixel format of 4×4 Quad-Bayer is subjected to 2×2-in-1 binning to obtain a pre-processed image with a target pixel format of 2×2 Quad-Bayer. This pre-processing is equivalent to compressing the original image by 2 times, resulting in a pre-processed image with a zoom ratio of 2×. In addition, in the subsequent zooming and cropping stage, further processing based on 2× is required according to the target zoom ratio to obtain an image with the target zoom ratio and target size. When Zoom∈[2×, 4×), in low and medium illumination scenes, the original image with a pixel format of 4×4 Quad-Bayer is subjected to 4×4-in-1 binning to obtain a preprocessed image with a target pixel format of 1×1 Quad-Bayer (equivalent to Bayer). This preprocessing is equivalent to compressing the original image by 4 times to obtain a preprocessed image with a zoom ratio of 1×. In addition, in the subsequent magnification and cropping stage, further processing is required based on the target zoom ratio to obtain an image with the target zoom ratio and target size.

[0151] 3. When the target zoom ratio is in a larger third range, for example, Zoom∈[Z3×, Z4×], the illumination can be further divided into three categories, for example, high illumination, medium illumination and low illumination.

[0152] (1) At high illumination, the preprocessing stage does not perform binning on the original image with an N×N Quad-Bayer pixel format. Instead, it performs K4×K4-in-1 binning to obtain a preprocessed image with an M4×M4 Quad-Bayer target pixel format, where K4=1 and M4=N / K4. The corresponding pixel magnification of this preprocessed image is N / K4. High illumination refers here to, for example, the third illumination level.

[0153] (2) At medium illumination, the original image is subjected to medium binning, i.e., the original image with an N×N Quad-Bayer pixel format is subjected to K5×K5-in-1 binning to obtain a pre-processed image with a target pixel format of M5×M5 Quad-Bayer, where 1<K5<K1 and M5=N / K5. The pixel magnification corresponding to the pre-processed image is N / K5. The medium illumination referred to herein includes, for example, the fourth illumination, where the third illumination is greater than the fourth illumination.

[0154] (3) At low illumination, the original image is subjected to a large degree of binning processing, that is, the original image with a pixel format of N×N Quad-Bayer is subjected to K6×K6-in-1 binning to obtain a pre-processed image with a target pixel format of M6×M6 Quad-Bayer, where K5<K6≤K1 and M6=N / K6. The pixel magnification corresponding to the pre-processed image is N / K6. The low illumination referred to here includes, for example, the fifth illumination, where the fourth illumination is greater than the fifth illumination.

[0155] This is because the zoom ratio in the third range is a relatively large zoom ratio, that is, the zoom ratio corresponding to the original image is slightly less than or equal to the minimum value in the third range. Therefore, under the premise of a relatively large zoom ratio, there is generally no need to compress the original image. Also, when the illumination is low, the noise of the image cannot be reduced without binning processing, so a relatively large degree of binning processing is required; when the illumination is medium, only a moderate degree of binning processing is required to reduce the noise and reduce the pressure of subsequent image amplification; and when the illumination is high, the noise of the image obtained without binning processing is already very small. Therefore, when the illumination is high, there is no need to consider the impact of noise, and only the compression ratio requirement in the third range needs to be considered, without binning processing.

[0156] Therefore, while meeting both noise requirements and zoom ratio requirements, when the zoom ratio belongs to a larger range, such as the third range, the illumination is divided into high illumination, medium illumination and low illumination, and the above-mentioned binning processing of different degrees is performed respectively.

[0157] Taking a specific example, when N = 4, Z3 = 4, and Z4 = 16, if K4 is 1, then M4 = 4; if K5 is 2, then M5 = 2; and if K6 is 4, then M6 = 1. That is, when Zoom∈[4×, 16×], in a high-illuminance scene, the original image with a 4×4 Quad-Bayer pixel format is subjected to 1×1-in-1 binning to obtain a pre-processed image with a target pixel format of 4×4 Quad-Bayer. This pre-processing is equivalent to the uncompressed original image, and the pre-processed image with a zoom ratio of 4× is obtained. In addition, in the subsequent zooming and cropping stage, further processing based on 4× is required according to the target zoom ratio to obtain an image with the target zoom ratio and target size. When Zoom∈[4×,16×], in medium illumination scenarios, the original image with a pixel format of 4×4 Quad-Bayer is subjected to 2×2-in-1 binning to obtain a preprocessed image with a target pixel format of 2×2 Quad-Bayer. This preprocessing is equivalent to compressing the original image by 2 times, resulting in a preprocessed image with a zoom factor of 2×. In addition, in the subsequent zoom and cropping stage, further processing based on the 2× is required to obtain an image with the target zoom factor and target size. In medium illumination scenarios, the original image with a pixel format of 4×4 Quad-Bayer is subjected to 4×4-in-1 binning to obtain a preprocessed image with a target pixel format of 1×1 Quad-Bayer (equivalent to Bayer). This preprocessing is equivalent to compressing the original image by 4 times, resulting in a preprocessed image with a zoom factor of 1×. In addition, in the subsequent zoom and cropping stage, further processing based on the 1× is required to obtain an image with the target zoom factor and target size.

[0158] The pre-processing stage shown in FIG5 is only an example of Binning processing. In addition, after any degree of Binning processing, Remosaic processing can also be performed. Remosaic processing does not involve changes in zoom ratio and pixel size before and after, but can unify the pixel format into Bayer format, thereby facilitating the subsequent image signal processor to directly process data in this format.

[0159] The previous article only introduced the preprocessing strategy used in the preprocessing stage of the output strategy. The following will introduce the enlargement and cropping strategy that needs to be adopted on the basis of the preprocessing strategy.

[0160] 1. When the target zoom ratio is within the first, relatively low range, the preprocessing stage performs K1×K1-in-1 binning on the original image to output a preprocessed image with an M1×M1 Quad-Bayer pixel format, which is equivalent to outputting a preprocessed image with a zoom ratio of N / K1. Therefore, during the zooming and cropping stage, the preprocessed image must be zoomed by C1×-C2×. When the target zoom ratio is the minimum value within the first range, i.e., Z1, then C1 = Z1 / (N / K1) = Z1 / M1. When the target zoom ratio is the maximum value within the first range, i.e., Z2, then C2 = Z2 / (N / K1) = Z2 / M1.

[0161] Taking a specific example, when N = 4, Z1 = 1, Z2 = 2, K1 = 4, and M1 = 1, then C1 = 1 and C2 = 2. That is, when Zoom ∈ [1×, 2×), the aforementioned preprocessing stage is equivalent to compressing the original image by 4 times, resulting in a preprocessed image with a zoom magnification of 1×. Therefore, during the magnification and cropping stage, the 1× image must be further processed based on the target zoom magnification. If the target zoom magnification is 1×, no further magnification is performed. If the target zoom magnification is 1.5×, the 1× image is further magnified to 1.5×. If the target zoom magnification is 1.9×, the 1× image is further magnified to 1.9× to obtain an image with the target zoom magnification and target size. Furthermore, the image must be cropped to the target size based on the target size.

[0162] 2-1. When the target zoom ratio is in the medium second range and the illumination is high, for example, when Zoom∈[Z2×, Z3×), the preprocessing stage performs K2×K2-in-1 binning on the original image to output a preprocessed image with an M2×M2 Quad-Bayer pixel format, which is equivalent to outputting a preprocessed image with a zoom ratio of N / K2. Therefore, in the zoom and cropping stage, the preprocessed image needs to be magnified by C3×-C4×. When the target zoom ratio is the minimum value in the second range, i.e., Z2, then C3=Z2 / (N / K2)=Z2 / M2. When the target zoom ratio is the maximum value in the second range, i.e., Z3, then C4=Z3 / (N / K2)=Z3 / M2.

[0163] Taking a specific example, when N = 4, Z2 = 2, Z3 = 4, K2 = 2, and M2 = 2, then C3 = 1 and C4 = 2. That is, when Zoom ∈ [2×, 4×), the aforementioned preprocessing stage is equivalent to compressing the original image by 2x, resulting in a preprocessed image with a zoom factor of 2×. Therefore, during the magnification and cropping stage, the 2× image must be further processed based on the target zoom factor, based on the 2× image. If the target zoom factor is 2×, no further magnification is performed. If the target zoom factor is 3×, the 2× image is further magnified to 1.5×. If the target zoom factor is 3.9×, the 2× image is further magnified to 3.9× to obtain an image with the target zoom factor and target size. Furthermore, the image must be cropped to the target size based on the target size.

[0164] 2-2. When the target zoom ratio is in the medium second range and the illumination is medium-low, for example, when Zoom∈[Z2×, Z3×), the preprocessing stage performs K3×K3-in-1 binning on the original image to output a preprocessed image with an M3×M3 Quad-Bayer pixel format, which is equivalent to outputting a preprocessed image with a zoom ratio of N / K3. Therefore, in the zoom and cropping stage, the preprocessed image needs to be magnified by C5×-C6×. When the target zoom ratio is the minimum value in the second range, i.e., Z2, then C5=Z2 / (N / K3)=Z2 / M3. When the target zoom ratio is the maximum value in the second range, i.e., Z3, then C6=Z3 / (N / K3)=Z3 / M3.

[0165] Taking a specific example, when N = 4, Z2 = 2, Z3 = 4, K3 = 4, and M3 = 1, then C5 = 2 and C6 = 4. That is, when Zoom ∈ [2×, 4×), the aforementioned preprocessing stage is equivalent to compressing the original image by 4 times, resulting in a preprocessed image with a zoom magnification of 1×. Therefore, during the magnification and cropping stage, the 1× image must be further processed based on the target zoom magnification. If the target zoom magnification is 2×, the 1× image is further magnified to 2×; if the target zoom magnification is 3×, the 1× image is further magnified to 3×; if the target zoom magnification is 3.9×, the 1× image is further magnified to 3.9× to obtain an image with the target zoom magnification and target size. Furthermore, the image must be cropped to the target size based on the target size.

[0166] 3-1. When the target zoom ratio is within the larger third range and the illumination is high, for example, when Zoom∈[Z3×, Z4×], the preprocessing stage performs K4×K4-in-1 binning on the original image to output a preprocessed image with an M4×M4 Quad-Bayer pixel format, which is equivalent to outputting a preprocessed image with a zoom ratio of N / K4. Therefore, during the zoom and cropping stage, the preprocessed image must be magnified by C7×-C8×. When the target zoom ratio is the minimum value within the third range, i.e., Z3, then C7 = Z3 / (N / K4) = Z3 / M4. When the target zoom ratio is the maximum value within the third range, i.e., Z4, then C8 = Z4 / (N / K4) = Z4 / M4.

[0167] Taking a specific example, when N = 4, Z3 = 4, Z4 = 16, K4 = 1, and M4 = 4, then C7 = 1 and C8 = 4. That is, when Zoom ∈ [4×, 16×], the aforementioned preprocessing stage is equivalent to uncompressing the original image, resulting in a preprocessed image with a zoom magnification of 4×. Therefore, during the magnification and cropping stage, the 4× image must be further processed based on the target zoom magnification. If the target zoom magnification is 4×, no further magnification is performed. If the target zoom magnification is 12×, the 4× image is further magnified to 12×. If the target zoom magnification is 16×, the 4× image is further magnified to 4× to obtain an image with the target zoom magnification and target size. Furthermore, the image must be cropped to the target size based on the target size.

[0168] 3-2. When the target zoom ratio is within the third, larger range and the illumination is medium, for example, when Zoom∈[Z3×, Z4×], the preprocessing stage performs K5×K5-in-1 binning on the original image to output a preprocessed image with an M5×M5 Quad-Bayer pixel format, which is equivalent to outputting a preprocessed image with a zoom ratio of N / K5. Therefore, during the zoom and cropping stage, the preprocessed image needs to be magnified by C9×-C10×. When the target zoom ratio is the minimum value within the third range, i.e., Z3, then C9 = Z3 / (N / K5) = Z3 / M5. When the target zoom ratio is the maximum value within the third range, i.e., Z4, then C10 = Z4 / (N / K5) = Z4 / M5.

[0169] Taking a specific example, when N = 4, Z3 = 4, Z4 = 16, K5 = 2, and M5 = 2, then C9 = 2 and C10 = 8. That is, when Zoom ∈ [4×, 16×], the aforementioned preprocessing stage is equivalent to compressing the original image by 2x, resulting in a preprocessed image with a zoom factor of 2×. Therefore, during the magnification and cropping stage, the 2× image must be further processed based on the target zoom factor. If the target zoom factor is 4×, the 2× image is further magnified to 4×; if the target zoom factor is 12×, the 2× image is further magnified to 12×; and if the target zoom factor is 16×, the 2× image is further magnified to 16× to obtain an image with the target zoom factor and target size. Furthermore, the image must be cropped to the target size based on the target size.

[0170] 3-3. When the target zoom ratio is within a larger third range and the illumination is low, for example, when Zoom∈[Z3×, Z4×], the preprocessing stage performs K6×K6-in-1 binning on the original image to output a preprocessed image with an M6×M6 Quad-Bayer pixel format, which is equivalent to outputting a preprocessed image with a zoom ratio of N / K6. Therefore, during the zoom and cropping stage, the preprocessed image must be magnified by C11×-C12×. When the target zoom ratio is the minimum value within the third range, i.e., Z3, then C11=Z3 / (N / K6)=Z3 / M6. When the target zoom ratio is the maximum value within the third range, i.e., Z4, then C12=Z4 / (N / K6)=Z4 / M6.

[0171] For example, when N = 4, Z3 = 4, Z4 = 16, K6 = 4, and M6 = 1, then C11 = 4 and C12 = 16. That is, when Zoom ∈ [4×, 16×], the preprocessing stage is equivalent to compressing the original image by 4x, resulting in a preprocessed image with a zoom factor of 1x. Therefore, during the magnification and cropping stage, the 1x image must be further processed based on the target zoom factor. For example, if the target zoom factor is 4×, the 1x image is further magnified to 4×; if the target zoom factor is 12×, the 1x image is further magnified to 12×; and if the target zoom factor is 16×, the 1x image is further magnified to 16× to obtain an image with the target zoom factor and target size. Furthermore, the image must be cropped to the target size based on the target size.

[0172] The zoom ratio range division involved in the processing strategy described in Figure 6 above is only an example. Optionally, more or fewer ranges can be divided. The specific decision needs to be made comprehensively based on the zoom ratio range provided by the electronic device and the maximum pixel array of the image sensor to ensure that the high pixel array is fully utilized, the image noise resistance performance is improved, and the zoom processing is facilitated. The embodiments of the present application do not impose any restrictions on this.

[0173] From the above, it can be seen that the output strategy adopted by the image processing method provided by the present application mainly includes: in the preprocessing stage, the original image is first subjected to corresponding pixel merging according to the range of the target zoom ratio and the range of illumination, so that the purpose of preliminary zoom ratio processing can be achieved while ensuring the noise resistance of the image contrast. In the zooming and cropping stage, it is only necessary to further zoom in, crop, and other operations on the preprocessed image obtained after the preliminary zoom ratio processing in the preprocessing stage according to the target ratio and target size to obtain the final image corresponding to the target zoom ratio and target size with excellent noise resistance.

[0174] Furthermore, the magnification operation involved in the image processing method provided herein can be performed in any one or more of a variety of stages, including before, during, or after the neural network. The neural network's role includes, but is not limited to, converting the original image, such as converting the image from the RAW domain to the RGB domain or the YUV domain. For the specific implementation of the magnification operation in these three stages, please refer to the description of FIG. 7 below.

[0175] FIG7 is a schematic diagram of a specific method for implementing a drawing output strategy provided in this application.

[0176] As shown in FIG7 , the specific method for implementing the output strategy is illustrated by taking N=4, Z3=4, Z4=16, K5=2, the target zoom ratio is 4×, the target size is 4096×3072 and the output strategy in a medium illumination scene as an example.

[0177] 1. In the preprocessing stage, the original image with a pixel format of 4×4 Quad-Bayer and an image size of 4096*4×3072*4 is subjected to 2×2-in-1 binning to obtain a preprocessed image with a pixel format of 2×2 Quad-Bayer and an image size of 4096*2×3072*2.

[0178] 2-1. If zooming is performed before the neural network, the zooming stage involves zooming the preprocessed image by 2x in the RAW domain before the neural network, resulting in an image with a zoom factor of 4x. This image with a zoom factor of 4x is then scaled to 4096*4×3072*4. After inputting this zoomed image into the neural network, the output image maintains the same zoom factor and image size as the input image. Finally, the image is cropped to the target size of 4096×3072.

[0179] Optionally, when zooming in front of the neural network, the zoom function can be specifically performed by the in-sensor zoom function of the electronic device. In-sensor zoom refers to the operation performed in the image sensor, that is, the image sensor provided by the present application can have a certain zoom processing capability.

[0180] 2-2. If zooming is performed within the neural network, the preprocessed RAW image is directly fed into the neural network. In addition to converting the image to the RGB / YUV domain, the neural network also zooms it by 2x, resulting in an image with a zoom factor of 4x. The image with a zoom factor of 4x is then zoomed to 4096*4×3072*4. Therefore, the output image of the neural network has a zoom factor twice that of the input image, equivalent to zooming from 2x to 4x. The image size is also zoomed from 4096*2×3072*2 to 4096*4×3072*4. Finally, the target image size of 4096×3072 is simply cropped.

[0181] 2-3. If upscaling is performed after the neural network, the pre-processed image in the RAW domain is directly fed into the neural network. The neural network converts the image to the RGB / YUV domain without upscaling. Therefore, the zoom factor and image size of the output image remain the same as the pre-processed image. Therefore, after the neural network, the image needs to be upscaled by 2x to obtain an image with a zoom factor of 4x. The size of this 4x zoomed image is also upscaled to 4096*4×3072*4. Finally, the image only needs to be cropped to the target size of 4096×3072.

[0182] FIG7 only takes a scenario as an example to exemplify the magnification process of any one of the three stages included in the image output strategy. In addition, any multiple of the three stages can be superimposed to produce an image corresponding to the target zoom ratio. The embodiments of the present application will no longer give examples one by one. Optionally, in the case of a high zoom ratio, the image needs to be magnified by a higher multiple, so by dispersing the magnification in multiple stages, it helps to output an image with more stable image quality. In the case of a low zoom ratio, the image needs to be magnified by a smaller multiple, so the image of the target zoom ratio can be obtained by grading at any one place.

[0183] Next, the software and hardware architecture and product form of the electronic equipment involved in this application are introduced.

[0184] Electronic equipment can be equipped or portable terminal devices with other operating systems, such as mobile phones, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, as well as cellular phones, personal digital assistants (PDAs), augmented reality (AR) devices, virtual reality (VR) devices, artificial intelligence (AI) devices, wearable devices, in-vehicle devices, smart home devices and / or smart city devices, etc.

[0185] FIG8 shows a schematic diagram of the hardware architecture of an electronic device.

[0186] As shown in FIG8 , the electronic device may include: a processor 110, an external memory interface 120, an internal memory 126, a camera 130, a display screen 140, an audio module 150, a speaker 150A, a receiver 150B, a microphone 150C, an earphone interface 150D, and a sensor module 160. The sensor module 160 may include a pressure sensor 160A, a distance sensor 160F, a proximity light sensor 160G, a touch sensor 160K, a light sensor 160L, and the like.

[0187] It is understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the electronic device. In other embodiments of the present application, the electronic device may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The illustrated components can be implemented in hardware, software, or a combination of software and hardware. For example, the electronic device may further include buttons, motors, indicators, and subscriber identification module (SIM) card interfaces, etc. For another example, the sensor module may further include: a gyroscope sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a fingerprint sensor, a temperature sensor, a bone conduction sensor, and the like.

[0188] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0189] The controller can be the nerve center and command center of the electronic device. The controller can generate operation control signals based on instruction operation codes and timing signals to complete the control of instruction fetching and execution.

[0190] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0191] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0192] In an embodiment of the present application, the processor 110 can be used to run a camera application, and the processor 110 is used to control the various modules in the camera application, as well as the camera 130, etc. to execute the image processing process described in Figures 1-2 described above, and in this image processing process, the image output strategy described in Figures 6-7 described above is specifically adopted.

[0193] It is understood that the interface connection relationship between the modules illustrated in the embodiments of the present application is only for illustrative purposes and does not constitute a structural limitation on the electronic device. In other embodiments of the present application, the electronic device may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0194] The internal memory 126 may include one or more random access memories (RAM) and one or more non-volatile memories (NVM).

[0195] Random access memory may include static random-access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM, for example, the fifth generation of DDR SDRAM is generally referred to as DDR5 SDRAM), etc.

[0196] Non-volatile memory may include disk storage devices and flash memory.

[0197] Flash memory can be divided into NOR FLASH, NAND FLASH, 3D NAND FLASH, etc. according to the operating principle; single-level cell (SLC), multi-level cell (MLC), triple-level cell (TLC), quad-level cell (QLC), etc. according to the storage cell potential level; universal flash storage (UFS) and embedded multi media card (eMMC) can be divided into UFS and eMMC according to the storage specification.

[0198] The random access memory can be directly read and written by the processor 110, and can be used to store executable programs (such as machine instructions) of the operating system or other running programs, and can also be used to store user and application data.

[0199] The non-volatile memory may also store executable programs and user and application data, etc., and may be loaded into the random access memory in advance for direct reading and writing by the processor 110 .

[0200] The external memory interface 120 can be used to connect to an external non-volatile memory device to expand the storage capacity of the electronic device. The external non-volatile memory device communicates with the processor 110 via the external memory interface 120 to implement data storage. For example, files such as music and videos can be stored in the external non-volatile memory device.

[0201] In an embodiment of the present application, the above-mentioned memory can be used to store the operating data of the camera application and user data, including but not limited to: zoom ratio, image output strategy corresponding to illumination, photos taken, etc.

[0202] The electronic device implements its display functionality through a GPU, display screen 140, and an application processor. The GPU is a microprocessor for image processing that connects display screen 140 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0203] The display screen 140 is used to display images, videos, etc. The display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD). The display screen panel can also be made of an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniLED, a microLED, a micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device can include one or N display screens 140, where N is a positive integer greater than one.

[0204] In the embodiment of the present application, the display screen 140 may be used to display interfaces provided by the camera application and interfaces provided by the gallery, including but not limited to a preview interface, a shooting interface, a settings interface, etc. The camera application may display controls / icons / options that allow the user to adjust the zoom ratio and image size, etc.

[0205] The electronic device can implement a shooting function through an ISP, a camera 130 , a video codec, a GPU, a display screen 140 , and an application processor.

[0206] The ISP processes data fed back by the camera 130. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and other factors. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within the camera 130.

[0207] The camera 130 is used to capture still images or videos. The camera 130 includes the image sensor described above. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device may include 1 or N cameras 130, where N is a positive integer greater than 1.

[0208] In the embodiment of the present application, the image sensor is a component of the camera 130 .

[0209] In the embodiment of the present application, the camera 130 may include one or more cameras, such as a main camera, a wide-angle camera, a telephoto camera, and the like.

[0210] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when an electronic device selects a frequency, the DSP performs a Fourier transform on the frequency energy.

[0211] Video codecs are used to compress or decompress digital video. Electronic devices may support one or more video codecs. This allows them to play or record videos in a variety of encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0212] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in electronic devices, such as image recognition, face recognition, speech recognition, and text comprehension.

[0213] In an embodiment of the present application, a neural network may be used to perform domain conversion on an image, or to magnify or crop an image, etc., during the image processing stage.

[0214] The electronic device can implement audio functions such as music playback and recording through the audio module 150, the speaker 150A, the receiver 150B, the microphone 150C, the headphone jack 150D, and the application processor.

[0215] The audio module 150 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 150 can also be used to encode and decode audio signals. In some embodiments, the audio module 150 can be provided in the processor 110, or some functional modules of the audio module 150 can be provided in the processor 110.

[0216] The speaker 150A, also called a "speaker," is used to convert audio electrical signals into sound signals. The electronic device can listen to music or make hands-free calls through the speaker 150A.

[0217] The receiver 150B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the user can place the receiver 150B close to the ear to hear the voice.

[0218] Microphone 150C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 150C to input the sound signal into the microphone 150C. The electronic device can be provided with at least one microphone 150C. In other embodiments, the electronic device can be provided with two microphones 150C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 150C to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0219] The headphone jack 150D is used to connect a wired headphone and can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0220] Pressure sensor 160A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 160A can be located on display screen 140. There are many types of pressure sensors 160A, such as resistive, inductive, and capacitive. A capacitive pressure sensor can include at least two parallel plates made of conductive material. When force acts on pressure sensor 160A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on this change in capacitance. When a touch operation is applied to display screen 140, the electronic device detects the intensity of the touch operation based on pressure sensor 160A. The electronic device can also calculate the location of the touch based on the detection signal from pressure sensor 160A. In some embodiments, touch operations applied to the same touch location but with different touch operation intensities can correspond to different operation instructions. For example, when a touch operation with an intensity less than a first pressure threshold is applied to a short message application icon, an instruction to view short messages is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to a short message application icon, an instruction to create a new short message is executed.

[0221] Distance sensor 160F is used to measure distance. Electronic devices can measure distance using infrared or laser technology. In some embodiments, when capturing a scene, the electronic device can use distance sensor 160F to measure distance for rapid focusing. In embodiments of the present application, the electronic device can use distance sensor 160F to capture object distance in scene data, i.e., the distance between the captured object and the electronic device.

[0222] The proximity light sensor 160G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device emits infrared light outward through the light emitting diode. The electronic device uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device. When insufficient reflected light is detected, the electronic device can determine that there is no object near the electronic device. The electronic device can use the proximity light sensor 160G to detect when the user holds the electronic device close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 160G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0223] The touch sensor 160K is also called a "touch panel." The touch sensor 160K can be provided on the display screen 140. The touch sensor 160K and the display screen 140 form a touch screen, also called a "touch screen." The touch sensor 160K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 140. In other embodiments, the touch sensor 160K can also be provided on the surface of the electronic device, at a location different from that of the display screen 140.

[0224] Light sensor 160L senses ambient light. The electronic device can adaptively adjust the brightness of display screen 140 based on the perceived ambient light. Light sensor 160L can also be used to automatically adjust white balance when taking photos. Light sensor 160L can also work with proximity sensor 160G to detect whether the electronic device is in a pocket to prevent accidental touches.

[0225] In the embodiment of the present application, the ambient light brightness sensed by the light sensor 160L can be used as the illuminance information mentioned above, and the electronic device can process the image according to the corresponding image output strategy by combining the illuminance information and the target zoom ratio. The illuminance mentioned in the present application can also be referred to as brightness, etc.

[0226] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a microservice architecture, or a cloud architecture. The embodiment of the present application takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device.

[0227] Refer to FIG9 , which is a schematic diagram of the system architecture of an electronic device according to an embodiment of the present application.

[0228] As shown in Figure 9, the layered architecture divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: from top to bottom, the application layer, the application framework layer, the hardware abstraction layer, the kernel layer, and the hardware layer.

[0229] The application layer can include a series of application packages. For example, an application package can include applications such as camera and gallery.

[0230] The camera application can initiate preview or photo capture requests to the underlying layer based on user operations. Specifically, it can call the framework layer's continuous image acquisition API, "setRepeatingRequest()," to implement the preview function, or call the framework layer's image acquisition API, "capture()," to implement the photo capture function.

[0231] Optionally, the camera application may also include but is not limited to the following modules not shown in Figure 9: a photo taking module, a video recording module, a Zoom control module, and a UI control module, etc. Among them, the photo taking module is used to execute the photo taking function of the camera application. The video recording module is used to execute the video recording function of the camera application. The Zoom control module is used to execute the zoom function of the camera application, that is, to control the Camera HAL to obtain the image according to the corresponding output strategy based on the target zoom ratio selected by the user. The UI control module is used to control the display module, that is, the display screen 140 to output the corresponding content according to the corresponding UI strategy, for example, after the user adjusts the zoom ratio, the adjusted zoom ratio is displayed.

[0232] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example, the application framework layer may include a camera interface and display services. The camera interface includes, but is not limited to, setRepeatingRequest() and capture().

[0233] The camera interface may also be referred to as a camera access interface, camera service, etc., and is specifically used to provide an application programming interface and programming framework for camera applications.

[0234] Display services include visual controls, such as controls for displaying text and images. Display services can be used to build the display interface of an application, which can be composed of one or more views.

[0235] The hardware abstraction layer (HAL) is an interface layer located between the application framework layer and the kernel layer, providing a virtual hardware platform for the operating system.

[0236] In the embodiment of the present application, the hardware abstraction layer may include a camera hardware abstraction layer (CameraHAL) and a camera algorithm library.

[0237] The Camera HAL provides virtual hardware for the camera module. It controls the corresponding sensors to acquire scene data, such as image sensors. It also transmits acquired image and illumination data to the camera algorithm library.

[0238] The camera algorithm library may include algorithm modules and the neural network described above.

[0239] The algorithm module includes several algorithms for processing images, which can be used to capture images with a target zoom ratio.

[0240] The kernel layer is the layer between hardware and software. The kernel layer includes drivers for various hardware.

[0241] In some embodiments, the kernel layer may include a camera driver, a digital signal processor driver, an image processor driver, and a display driver, etc.

[0242] Among them, the camera driver is used to drive the corresponding sensor of the camera module to collect images, the digital signal processor driver is used to drive the digital signal processor to process images, and the image processor driver is used to drive the image signal processor to process images.

[0243] The hardware layer mainly includes components in the camera module, such as image sensor 1, image sensor 2, TOF, multispectral sensor, image signal processor, digital signal processor, image processor, etc. Among them, sensor 1 and sensor 2 refer to the photosensitive elements in the camera.

[0244] The following describes the workflow of the electronic device software and hardware in conjunction with capturing a photo scene.

[0245] When touch sensor 160K receives a touch operation, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single-touch operation and the control corresponding to the single-touch operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then calls the kernel layer to start the camera driver, capturing a still image or video through the camera 130.

[0246] It should be understood that each step in the above method embodiments provided herein can be implemented by hardware integrated logic circuits in a processor or by software instructions. The method steps disclosed in the embodiments of this application can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0247] The present application also provides an electronic device, which may include: a memory and a processor, wherein the memory may be used to store a computer program; and the processor may be used to call the computer program in the memory so that the electronic device executes the method in any one of the above embodiments.

[0248] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0249] In one possible design, the chip system further includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0250] The chip system can be composed of chips, or can include chips and other discrete devices.

[0251] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0252] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0253] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD) or other integrated chips.

[0254] The present application also provides a computer program product, which includes: a computer program (also referred to as code, or instruction), which, when executed, enables a computer to execute the method executed by the electronic device in any of the above embodiments.

[0255] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0256] The various implementation modes of this application can be combined arbitrarily to achieve different technical effects.

[0257] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described herein are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive).

[0258] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0259] In short, the above description is only an embodiment of the technical solution of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made based on the disclosure of the present invention should be included in the scope of protection of the present invention.

Claims

1. An image processing method, characterized in that: The method is applied to an electronic device including an image sensor, and the method includes: Acquire an original image through the image sensor, wherein the zoom ratio of the original image is greater than 1; Performing K×K pixel-in-1 processing on the original image at the zoom factor based on a preset strategy to obtain a first image, where K≥1; Wherein, the preset strategy includes: when the target zoom ratio is in the first range, K=K1; when the target zoom ratio is in the second range and the illumination is the first illumination, K=K2; when the target zoom ratio is in the second range and the illumination is the second illumination, K=K3; the minimum value in the second range is greater than the maximum value in the first range, the first illumination is greater than the second illumination, and K2<K3≤K1; The first image is enlarged to obtain a second image with the target zoom factor.

2. The method according to claim 1, characterized in that The preset strategy also includes: When the target zoom ratio is in the third range, if the illumination is the third illumination, K=K4, if the illumination is the fourth illumination, K=K5, if the illumination is the fifth illumination, K=K6; the third range is greater than the second range, the third illumination is greater than the fourth illumination, the fourth illumination is greater than the fifth illumination, and K4<K5<K6≤K1.

3. The method according to claim 1 or 2, characterized in that: After obtaining the first image, the method further includes: converting the first image from the RAW domain to the RGB domain or the YUV domain through a neural network; Enlarging the first image specifically includes enlarging the first image at any one or more of the following stages: before, during, or after domain conversion of the first image by the neural network.

4. The method according to claim 3, characterized in that When the target zoom ratio is larger, the first image is hierarchically magnified through more stages.

5. The method according to claim 3 or 4, characterized in that: Amplifying the first image before domain converting the first image by the neural network includes: amplifying the first image by the image sensor, and / or amplifying the first image by an algorithm.

6. The method according to any one of claims 1 to 5, characterized in that The pixel format of the original image is N×NQuad-Bayer, and the pixel format corresponding to the first image is M×MQuad-Bayer, where N≥M, N>1, and M≥1.

7. The method according to claim 6, characterized in that The zoom magnification of the original image is N times, and the zoom magnification of the first image is N / K times.

8. The method according to claim 7, characterized in that When the target zoom ratio is Z, the first image is enlarged, specifically including: enlarging the first image by C times, where C=Z / (N / K).

9. The method according to any one of claims 1 to 8, characterized in that Before obtaining the second image, the method further includes: before or after amplifying the first image, cropping the first image.

10. The method according to any one of claims 1 to 9, characterized in that Before performing K×K pixel-in-one processing on the original image based on a preset strategy, the method further includes: Detecting the illumination by a light sensor of the electronic device, Alternatively, the illumination is acquired through exposure parameters of the image sensor.

11. The method according to any one of claims 1 to 10, characterized in that Before performing K×K pixel-in-one processing on the original image based on a preset strategy, the method further includes: A first operation is received, and in response to the first operation, a zoom magnification of the electronic device is set to the target zoom magnification.

12. The method according to any one of claims 1 to 11, characterized in that The method further includes: displaying a preview interface, and displaying the second image in the preview interface.

13. The method according to any one of claims 1 to 11, characterized in that The original image includes multiple frames of images, and the first image refers to: a frame of image obtained by fusion processing of multiple frames of images subjected to K×K pixel fusion processing based on a preset strategy; The method further includes: receiving a photo-taking operation before performing K×K pixel-in-1 processing on the original image based on a preset strategy; saving the second image, displaying a gallery interface, and displaying the second image in the gallery interface.

14. An electronic device, characterized in that: The electronic device includes an image sensor, one or more memories, and one or more processors; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the electronic device to execute the method described in any one of claims 1-13.

15. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on an electronic device, the electronic device is caused to execute the method as claimed in any one of claims 1 to 13.

16. A chip, applied to electronic equipment, characterized in that: The chip includes one or more processors, and the processor is used to call computer instructions to enable the electronic device to execute the method as described in any one of claims 1-13.

17. A computer program product comprising instructions, characterized in that When the computer program product is executed on an electronic device, the electronic device is enabled to execute the method according to any one of claims 1 to 13.

Images