An image processing method and an electronic device

Abstract

Provided are an image processing method and electronic device, applied to the field of image processing technology. The method is applied to an electronic device, the electronic device including a camera, and comprises: starting the camera; obtaining a zoom ratio and a dynamic range of a current shooting scene; determining an image output mode of the camera based on the zoom ratio and the dynamic range; when the zoom ratio is greater than or equal to a first ratio and less than a second ratio, and the dynamic range satisfies a first dynamic range (DR) constraint, the camera outputs the image in a first mode; and performing image processing based on the image data output by the camera, thereby improving image quality in high-dynamic shooting scenes, avoiding ghosting issues, and helping to enhance the user's shooting experience.

Description

[0001] This application claims priority to Chinese Patent Application No. 202210191283.7, filed with the State Intellectual Property Office of China on February 28, 2022, entitled “An Image Processing Method and Electronic Device”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing technology, and more specifically, to an image processing method and an electronic device. Background Technology

[0003] With the rapid development of smart devices, camera functionality has become an essential feature. Users' demands and experiences regarding camera (photos and / or videos) capabilities on smart devices are constantly increasing. Smart devices can capture images using digital zoom. Digital zoom refers to cropping and / or enlarging images through software algorithms.

[0004] Currently, in high dynamic range shooting scenarios, multi-exposure synthesis technology is usually used for shooting, which can easily lead to ghosting problems, resulting in poor image quality and affecting the user's shooting experience. Summary of the Invention

[0005] In view of this, this application provides an image processing method, an electronic device, a computer-readable storage medium, and a computer program product, which can improve image quality in high dynamic range shooting scenes, avoid ghosting problems, and help improve the user's shooting experience.

[0006] In a first aspect, an image processing method is provided, the method being applied to an electronic device, the electronic device including a camera, the method comprising:

[0007] Turn on the camera;

[0008] Obtain the zoom ratio and dynamic range of the current shooting scene;

[0009] The image output method of the camera is determined based on the zoom ratio and the dynamic range;

[0010] When the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and the dynamic range (specifically, the dynamic range value) satisfies the first dynamic range (DR) constraint condition, the camera outputs images in the first mode.

[0011] Image processing is performed based on the image data output by the camera.

[0012] The first mode is the dual-gain conversion DCG mode. In DCG mode, the image output by the camera is a fused image frame consisting of a long exposure frame and a short exposure frame. After fusion, there is no ghosting problem, and the dynamic range is improved.

[0013] The dynamic range that satisfies the first dynamic range (DR) constraint is defined as a high dynamic range. The dynamic range that does not satisfy the first dynamic range (DR) constraint is defined as a low dynamic range.

[0014] Optionally, the first dynamic range (DR) constraint can be determined based on the histogram of the RAW image of the shooting scene. The first DR constraint is determined according to a first ratio, a second ratio, and a third ratio. The first ratio refers to the proportion of pixel values ​​greater than the first pixel value in the image pixel distribution, and the second ratio refers to the proportion of pixel values ​​less than the second pixel value in the image pixel distribution. If both the first ratio and the second ratio are greater than the third ratio, the first DR constraint is considered to be satisfied, and the dynamic range is defined as high dynamic range. If neither the first ratio nor the second ratio is greater than the third ratio (or in other words, at least one of the first ratio and the second ratio is not greater than the third ratio), the first DR constraint is considered not satisfied, and the dynamic range is defined as low dynamic range.

[0015] In addition, when determining the image output method of the camera, the ambient light level can also be considered. Optionally, the method further includes: acquiring the ambient light level of the shooting scene; and determining the image output method of the camera based on the zoom ratio, dynamic range, and ambient light level.

[0016] Optionally, when the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and the dynamic range satisfies the first dynamic range (DR) constraint condition, the camera will output images in the first mode regardless of whether the ambient illumination is high or low.

[0017] As one possible implementation, when the zoom ratio is greater than or equal to the first zoom ratio but less than the second zoom ratio, and the dynamic range does not meet the first DR constraint condition, the camera uses a second image output mode. The second mode is a binning mode. Here, when the dynamic range is low, the binning mode is sufficient.

[0018] Optionally, when the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and the dynamic range does not meet the first DR constraint condition, the camera will output images in the second mode regardless of whether the ambient illumination is high or low.

[0019] In the above implementation, when the zoom level is equal to the first zoom level (e.g., 1x), the processing of the back-path is consistent regardless of the image output method used by the sensor. The processing flow of the preview stream, the image capture stream, and the thumbnail stream are described below.

[0020] As one possible implementation, when the zoom ratio is equal to the first magnification and the dynamic range satisfies the first DR constraint, the camera outputs images in a first mode; or, when the zoom ratio is equal to the first magnification and the dynamic range does not satisfy the first DR constraint, the camera outputs images in a second mode, and the first image data adopts a first image format.

[0021] The first image data is stored in a first cache, and the method further includes:

[0022] Receive the user's first operation, which is used to trigger taking a picture;

[0023] The image processing based on the image data output by the camera includes:

[0024] In response to the first operation, the first image data is retrieved from the first cache;

[0025] The first image data is processed by the post-processing algorithm module or the second module of the image signal processor (ISP) to obtain the second image data.

[0026] The second image data is processed by the third module of the ISP using RGB or YUV processing to obtain YUV format data, and the captured image is output.

[0027] Therefore, for cases where the zoom ratio is equal to the first zoom ratio, the quality of the photographed image can be improved based on the above steps.

[0028] Optionally, the method further includes:

[0029] In the thumbnail stream, the first image data is de-mosaiced by the second ISP module to obtain RGB format data;

[0030] The RGB format data is processed by the third module of the ISP to obtain YUV format data and output thumbnails.

[0031] Therefore, for cases where the zoom level is equal to the first zoom level, the quality of the thumbnail can be improved based on the above steps.

[0032] Optionally, the method further includes:

[0033] In the preview path, the first image data is de-mosaiced by the second ISP module to obtain RGB format data;

[0034] The RGB format data is processed by the third module of the ISP to obtain YUV format data, and a preview image is output.

[0035] Therefore, for cases where the zoom level is equal to the first zoom level, the quality of the preview image can be improved based on the above steps.

[0036] This application embodiment can also use the ambient illumination of the shooting environment as a factor in deciding the image output method of the sensor.

[0037] As one possible implementation, the method further includes:

[0038] Obtain the ambient illuminance of the current shooting scene;

[0039] The image output method of the camera is determined based on the ambient illumination, the zoom ratio, and the dynamic range.

[0040] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio, the ambient illumination is low, and the dynamic range satisfies the first DR constraint condition, the camera outputs images in the first mode.

[0041] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio, the ambient illumination is low, and the dynamic range does not meet the first DR constraint condition, the camera uses the second mode to output the image.

[0042] Therefore, in scenarios where the zoom ratio is greater than or equal to the second zoom ratio and the ambient light is low, using DCG mode to output images when the dynamic range is high helps to improve the dynamic range of the captured images; when the dynamic range is low, using the merge mode is sufficient.

[0043] In the above implementation, the post-path processing can be consistent in some cases. Specifically, for the following situations: when the zoom ratio is greater than the first magnification and less than the second magnification, and the dynamic range satisfies the first DR constraint, the camera uses the first mode for image output; or, when the zoom ratio is greater than the first magnification and less than the second magnification, and the dynamic range does not satisfy the first DR constraint, the camera uses the second mode for image output; or, when the zoom ratio is greater than or equal to the second magnification, the ambient illumination is a low-light scene, and the dynamic range satisfies the first DR constraint, the camera uses the first mode for image output; or, when the zoom ratio is greater than or equal to the second magnification, the ambient illumination is a low-light scene, and the dynamic range does not satisfy the first DR constraint, the camera uses the second mode for image output; the post-path processing can be consistent.

[0044] In all the above situations, the camera outputs third image data, and the third image data adopts the first image format;

[0045] The third image data is stored in the first cache, and the method further includes:

[0046] Receive a second operation from the user, which triggers the taking of a photo;

[0047] In response to the second operation, third image data is retrieved from the first cache;

[0048] The third image data is processed by the post-processing algorithm module or the second ISP module to obtain the fourth image data;

[0049] The fourth image data is cropped and upsampled by the post-processing algorithm module or the third ISP module to output the captured image.

[0050] Therefore, based on the steps described above, the quality of the photographed image can be improved in the aforementioned situations.

[0051] Optionally, in all the above cases, the method further includes:

[0052] In the thumbnail stream, the third image data is de-mosaiced by the second module of the ISP to obtain RGB format data;

[0053] The RGB format data is processed by the third module of the ISP to obtain YUV format data, which is then cropped and upsampled to output a thumbnail.

[0054] Therefore, based on the above steps, the quality of the thumbnail image can be improved.

[0055] Optionally, in all the above cases, the method further includes:

[0056] In the preview path, the third image data is de-mosaiced by the second ISP module to obtain RGB format data;

[0057] The RGB format data is processed by the third module of the ISP to obtain YUV format data, and then cropped and upsampled to output a preview image.

[0058] Therefore, based on the above steps, the quality of the preview image can be improved.

[0059] As one possible implementation, the method further includes:

[0060] When the zoom ratio is greater than or equal to the second zoom ratio, the ambient illumination is high, and the dynamic range meets the first DR constraint condition, the camera outputs images in the third mode. The third mode is a non-merging + cropping mode.

[0061] Therefore, in high-light scenes with a zoom ratio greater than or equal to the second zoom ratio, when the dynamic range is high, using the non-merge + crop mode can ensure better image sharpness. Furthermore, the dynamic range of the shooting scene can be improved by generating separate long and short exposure frames and using multi-frame post-processing algorithms. In other words, the third mode is used here to take into account the image sharpness requirements of the shooting scene.

[0062] As one possible implementation, the method further includes:

[0063] When the zoom ratio is greater than or equal to the second zoom ratio, the ambient illuminance is high illuminance, and the dynamic range does not meet the first DR constraint condition, the camera uses the third mode for image output.

[0064] Therefore, in high-light scenes where the zoom ratio is greater than or equal to the second zoom ratio, and the dynamic range is low, sharpness is the primary consideration. Thus, the sensor uses a non-merge + crop mode to output images, which can improve image sharpness.

[0065] When the camera outputs images using the third mode, the processing in the subsequent path can remain consistent. Optionally, the data output by the camera is fifth image data, and the fifth image data uses the second image format.

[0066] The image processing based on the image data output by the camera includes:

[0067] In the preview path, the fifth image data is processed by Bayer image regeneration through the first module of the ISP to obtain the sixth image data, which adopts the first image format.

[0068] The sixth image data is de-mosaiced using the second module of the ISP to obtain RGB format data.

[0069] The RGB format data is processed by the third module of the ISP to obtain YUV format data, and a preview image is output.

[0070] Therefore, based on the above steps, the quality of preview images under high magnification and high illumination can be improved.

[0071] Optionally, the fifth image data is stored in the first cache, and the method further includes:

[0072] Receive a third operation from the user, the third operation being used to trigger taking a picture;

[0073] In response to the third operation, the fifth image data is retrieved from the first cache;

[0074] The image processing based on the image data output by the camera includes:

[0075] The fifth image data is processed by the post-processing algorithm module to obtain the seventh image data, which adopts the first image format.

[0076] The seventh image data is de-mosaiced using the second module of the ISP to obtain RGB format data.

[0077] The RGB format data is processed by the third module of the ISP to obtain YUV format data, and then the captured image is output.

[0078] Optionally, the post-processing algorithm module can also perform multi-frame fusion processing to improve the dynamic range of the captured images.

[0079] Therefore, based on the above steps, the quality of images taken under high magnification and high illumination can be improved.

[0080] Optionally, the method further includes:

[0081] In the thumbnail stream, the fifth image data is processed by the first module of the ISP to obtain data in Bayer format.

[0082] The Bayer format data is de-mosaiced using the second module of the ISP to obtain RGB format data.

[0083] The RGB format data is processed by the third module of the ISP to obtain YUV format data and output thumbnails.

[0084] Therefore, based on the above steps, the quality of thumbnails under high magnification and high illumination can be improved.

[0085] In a second aspect, an electronic device is provided, including a unit for performing any of the methods in the first aspect. The electronic device may be a terminal or a chip within a terminal. The electronic device includes an input unit, a display unit, and a processing unit.

[0086] When the electronic device is a terminal, the processing unit may be a processor, the input unit may be a communication interface, and the display unit may be a graphics processing module and a screen; the terminal may also include a memory for storing computer program code, which, when the processor executes the computer program code stored in the memory, causes the terminal to perform any of the methods in the first aspect.

[0087] When the electronic device is a chip within a terminal, the processing unit can be a logic processing unit inside the chip, the input unit can be an output interface, pin, or circuit, and the display unit can be a graphics processing unit inside the chip. The chip may also include a memory, which can be memory within the chip (e.g., registers, caches, etc.) or memory located outside the chip (e.g., read-only memory, random access memory, etc.). The memory is used to store computer program code, and when the processor executes the computer program code stored in the memory, the chip executes any of the methods in the first aspect.

[0088] In one implementation, the processing unit is used to activate the camera;

[0089] Obtain the zoom ratio and dynamic range of the current shooting scene;

[0090] The image output method of the camera is determined based on the zoom ratio and the dynamic range;

[0091] When the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and the dynamic range satisfies the first dynamic range (DR) constraint condition, the camera is invoked to output an image in the first mode.

[0092] Image processing is performed based on the image data output by the camera.

[0093] As one possible implementation, the processing unit is further configured to: when the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and the dynamic range does not meet the first DR constraint condition, invoke the camera to output images in the second mode.

[0094] As one possible implementation, when the zoom ratio is equal to the first magnification and the dynamic range satisfies the first DR constraint, the camera outputs images in a first mode; or, when the zoom ratio is equal to the first magnification and the dynamic range does not satisfy the first DR constraint, the camera outputs images in a second mode, wherein the first image data adopts a first image format; and the first image data is stored in a first cache.

[0095] The input unit is used to receive a first operation from the user, which triggers the taking of a picture. The processing unit is used to perform image processing based on the image data output by the camera, specifically including: in response to the first operation, obtaining the first image data from the first cache; calling the post-processing algorithm module or the second module of the image signal processor (ISP) to perform image processing on the first image data to obtain second image data; calling the third module of the ISP to perform RGB processing or YUV processing on the second image data to obtain YUV format data, and outputting the captured image.

[0096] As one possible implementation, in the thumbnail stream, the processing unit is also used to call the second ISP module to perform de-mosaic processing on the first image data to obtain RGB format data;

[0097] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data, and a thumbnail is output.

[0098] As one possible implementation, in the preview path, the processing unit is also used to call the second ISP module to perform de-mosaic processing on the first image data to obtain RGB format data;

[0099] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data, and a preview image is output.

[0100] As one possible implementation, the processing unit is also used to acquire the ambient illuminance of the current shooting scene; and to determine the image output method of the camera based on the ambient illuminance, the zoom ratio, and the dynamic range.

[0101] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio, the ambient illumination is low, and the dynamic range satisfies the first DR constraint condition, the processing unit is further configured to call the camera to output images in the first mode.

[0102] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio, the ambient illumination is low, and the dynamic range does not meet the first DR constraint condition, the processing unit is further configured to call the camera to output images in the second mode.

[0103] As one possible implementation, the following scenarios are handled consistently: When the zoom ratio is greater than a first magnification and less than a second magnification, and the dynamic range satisfies the first DR constraint, the camera outputs images in a first mode; or, when the zoom ratio is greater than a first magnification and less than a second magnification, and the dynamic range does not satisfy the first DR constraint, the camera outputs images in a second mode; or, when the zoom ratio is greater than or equal to the second magnification, the ambient illumination is a low-light scene, and the dynamic range satisfies the first DR constraint, the camera outputs images in a first mode; or, when the zoom ratio is greater than or equal to the second magnification, the ambient illumination is a low-light scene, and the dynamic range does not satisfy the first DR constraint, the camera outputs images in a second mode; the subsequent path processing remains the same.

[0104] Optionally, the camera outputs third image data, which adopts a first image format;

[0105] The third image data is stored in the first cache, and the processing unit is further configured to:

[0106] Receive a second operation from the user, which triggers the taking of a photo;

[0107] The processing unit is used to perform image processing based on the image data output by the camera, specifically including:

[0108] In response to the second operation, third image data is retrieved from the first cache;

[0109] The processing unit is also used to call the post-processing algorithm module or the second ISP module to perform image processing on the third image data to obtain the fourth image data;

[0110] The processing unit is also used to call the post-processing algorithm module or the third ISP module to perform cropping and upsampling processing on the fourth image data and output the captured image.

[0111] As one possible implementation, in the thumbnail stream, the processing unit is also used to call the second ISP module to perform de-mosaic processing on the third image data to obtain RGB format data;

[0112] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data. The YUV format data is then cropped and upsampled to output a thumbnail.

[0113] As one possible implementation, in the preview path, the processing unit is also used to call the second ISP module to perform de-mosaic processing on the third image data to obtain RGB format data;

[0114] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data. The YUV format data is then cropped and upsampled to output a preview image.

[0115] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio, the ambient illuminance is high illuminance, and the dynamic range satisfies the first DR constraint condition, the processing unit is further configured to call the camera to output images in a third mode.

[0116] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio, the ambient illuminance is high illuminance, and the dynamic range does not meet the first DR constraint condition, the processing unit is further configured to call the camera to output images in a third mode.

[0117] As one possible implementation, the data output by the camera is fifth image data, and the fifth image data adopts a second image format;

[0118] The processing unit is used to perform image processing based on the image data output by the camera, specifically including:

[0119] In the preview path, the first module of the ISP is called to perform Bayer image regeneration processing on the fifth image data to obtain the sixth image data, which adopts the first image format.

[0120] The second module of the ISP is invoked to perform de-mosaic processing on the sixth image data to obtain RGB format data;

[0121] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data, and a preview image is output.

[0122] As one possible implementation, the fifth image data is stored in the first buffer, and the input unit is further configured to: receive a third operation from the user, the third operation being used to trigger taking a picture;

[0123] The processing unit is further configured to, in response to the third operation, retrieve the fifth image data from the first cache;

[0124] The processing unit is used to perform image processing based on the image data output by the camera, specifically including:

[0125] The post-processing algorithm module is invoked to perform Bayer image regeneration processing on the fifth image data to obtain the seventh image data, which adopts the first image format.

[0126] The second module of the ISP is invoked to perform de-mosaic processing on the seventh image data to obtain RGB format data;

[0127] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data, and then the captured image is output.

[0128] The processing unit is also used to call the first module of the ISP to perform Bayer image regeneration processing on the fifth image data to obtain data in Bayer format;

[0129] The second module of the ISP is invoked to perform de-mosaic processing on the Bayer format data to obtain RGB format data;

[0130] The ISP's third module is invoked to perform RGB or YUV processing on the RGB format data to obtain YUV format data, and a thumbnail is output.

[0131] Thirdly, a computer-readable storage medium is provided that stores computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the methods in the first aspect.

[0132] Fourthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed by an electronic device, causes the electronic device to perform any of the methods in the first aspect. Attached Figure Description

[0133] Figure 1 This is an example diagram illustrating an application scenario of an embodiment of this application;

[0134] Figure 2This is a schematic diagram of the image output method of the sensor under different zoom levels, different dynamic ranges and different ambient illuminance according to the embodiments of this application;

[0135] Figure 3 This is a schematic block diagram of a photographing system according to an embodiment of this application;

[0136] Figure 4 This is a schematic flowchart of a photographing method according to an embodiment of this application;

[0137] Figure 5 This is a schematic block diagram of another photographing system according to an embodiment of this application;

[0138] Figure 6 This is a schematic flowchart of another photographing method according to an embodiment of this application;

[0139] Figure 7 This is a schematic block diagram of another photographing system according to an embodiment of this application;

[0140] Figure 8 This is a schematic flowchart illustrating another photographing method according to an embodiment of this application;

[0141] Figure 9 This is a schematic diagram of an architecture applied in an embodiment of this application;

[0142] Figure 10 This is a schematic diagram of the structure of an electronic device applicable to this application. Detailed Implementation

[0143] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0144] In the embodiments of this application, unless otherwise stated, "multiple" can mean two or more.

[0145] The embodiments of this application are applicable to electronic devices, which may be mobile phones, smart screens, tablets, wearable electronic devices, in-vehicle electronic devices, augmented reality (AR) devices, virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), projectors, etc.

[0146] The electronic device in this application embodiment is equipped with an image acquisition device (such as a camera).

[0147] This application does not limit the specific type of electronic device. The image processing method of this application is described below using a mobile phone as an example.

[0148] The following combination Figure 1 The following are examples of scenarios. Figure 1 This is an example diagram illustrating an application scenario of an embodiment of this application. For example... Figure 1 In the mobile phone interface shown in (1), the interface can display multiple applications: Application 1, Application 2, ..., Application 7, and the camera application. When the user clicks the camera application, the phone launches the camera. After the camera is running, the mobile phone interface displays as shown below. Figure 1 The interface shown in (2) is the camera's shooting interface. This interface can be called the camera's shooting interface. It may include a viewfinder 11, a zoom level 12 (default 1x), an album icon 13, a shooting control 14, and a camera rotation control. The user can take a photo by clicking the shooting control 14. In this interface, the album icon 13 displays thumbnails. The camera rotation control can be used to switch cameras.

[0149] The viewfinder 11 is used to capture a preview image and can display the preview image in real time.

[0150] The phone supports digital zoom. When using the shooting function, users can select different zoom levels by operating the touchscreen.

[0151] As an example, such as Figure 1 As shown in (2), the user clicks Figure 1 The zoom ratio of (2) is 12, and the interface displays as follows: Figure 1 The interface shown in (3) displays a zoom ratio selection option 15 (for example, the highest zoom ratio is 8x and the lowest zoom ratio is 1x). The user drags the zoom ratio 12 upwards in selection option 15, and releases the finger when the zoom ratio is 2x. The interface then displays as shown in (3). Figure 1 As shown in (4), the zoom ratio is selected as 2x. Of course, after selecting the zoom ratio, the zoom ratio selection option 15 can be hidden, that is, the selected zoom ratio is displayed as 2x on the interface.

[0152] It should be understood that Figure 1 The photographing scenario shown is merely illustrative of one application scenario of this application and does not constitute a limitation on the embodiments of this application. In fact, the embodiments of this application can also be applied to other scenarios using cameras, such as video recording scenarios, video call scenarios, and live video streaming scenarios.

[0153] It should also be understood that Figure 1 (2) shows a schematic diagram of an interface for a user to take a photo in portrait mode on a mobile phone, but this application is not limited to this. For example, a user can take a photo in landscape mode on a mobile phone.

[0154] It should also be understood that Figure 1 The upper and lower limits of the zoom ratio selection option 15 shown in (3) (i.e., the maximum zoom ratio and the minimum zoom ratio) can depend on the implementation of the mobile phone. Figure 1 The upper and lower limits shown in (3) are merely examples, and the embodiments of this application are not limited thereto.

[0155] It should also be understood that Figure 1 The position of zoom magnification option 15 shown in (3) in the interface is only an example, and the embodiments of this application are not limited thereto.

[0156] In some embodiments, the mobile phone may employ a Quadra Color Filter Array (Quadra CFA) sensor camera. Key components of the camera include optical lenses and an image sensor. Once the camera is activated, the sensor can generate an image based on the acquired image signals.

[0157] The dynamic range (DR) of a scene affects the quality of the captured image. Because image sensors have a limited dynamic range, multi-exposure synthesis is often used to enhance the dynamic range that the image sensor can capture when shooting high dynamic range scenes. Current multi-exposure imaging techniques typically have different start and total exposure times for different exposure frames. This can lead to ghosting issues in the fused image when shooting moving objects, affecting image quality and resulting in a poor user experience.

[0158] In view of this, embodiments of this application provide an image processing method and an electronic device, which determines the image output mode of the camera by the zoom ratio and dynamic range of the shooting scene. When the zoom ratio is greater than or equal to a first ratio and less than a second ratio, and the dynamic range is high, the sensor uses DCG mode to output the image, which can improve the image quality in high dynamic shooting scenes, avoid the problem of ghosting, and help improve the user's shooting experience.

[0159] Furthermore, the image processing method in this application embodiment considers three factors: zoom level, ambient illumination, and dynamic range value of the shooting scene. It controls the sensor's image output method under different shooting scenarios, designs different shooting processes for different scenarios, and fully leverages the advantages of each component to improve image quality and enhance user experience in various scenarios. In this application embodiment, the image output method adopted by the sensor depends on the dynamic range value, zoom level, and ambient illumination of the shooting scene.

[0160] To facilitate understanding, before introducing the image processing method of the embodiments of this application, some terms involved in the embodiments of this application will be explained first.

[0161] In this embodiment, the sensor supports outputting images using a first mode, a second mode, or a third mode. For clarity, the first mode is dual conversion gain (DCG) mode. The second mode is binning mode. The third mode is non-binning + clipping mode. The DCG mode is essentially a binning mode, which will be explained in detail below.

[0162] DCG mode refers to adding DCG to CMOS pixels, enabling the sensor to simultaneously possess high sensitivity and high dynamic range. DCG includes high conversion gain (HCG) and low conversion gain (LCG). High HCG conversion gain corresponds to long exposure frames, resulting in excellent signal-to-noise ratio in dark areas. Low LCG conversion gain corresponds to short exposure frames, ensuring good highlight detail. When the sensor operates in DCG mode, it captures both long and short exposure frames, then merges them, and the merged image frame becomes the sensor's output. In other words, the image output by the sensor in DCG mode is a merged image frame of long and short exposure frames, resulting in no ghosting issues and improved dynamic range.

[0163] In the aforementioned exposure frames (including long and short exposure frames), "exposure" refers to the exposure value (EV), not the exposure time. The exposure value can be determined by both the exposure time and the gain.

[0164] For example, exposure time follows the formula: EV = Gain * Exposure Time. Here, EV represents the exposure amount. Factors affecting Gain include, but are not limited to, CG and Iso. CG is conversion gain, and Iso is ISO.

[0165] It is understood that the DCG described above uses long exposure frames and short exposure frames as examples, but the embodiments of this application are not limited to this. For example, in addition to HCG and LCG, the DCG mode can include more CGs (for example, the number of CGs can be increased by adding capacitors in the hardware), and each CG can output a corresponding image frame.

[0166] The merge mode refers to adding the charges sensed by adjacent pixels together and reading them out as a single pixel. The image data output through the merge mode is in Bayer raw format.

[0167] In non-binning mode, an image with the same resolution as the sensor is provided. The sensor outputs images in non-binning mode in quadra raw format, not bayer raw.

[0168] To facilitate ISP processing, the Quadra raw obtained in non-merge mode needs to be converted to Bayer raw.

[0169] Bayer image regeneration (Remosaic) converts data from non-merged mode into standard Bayer raw by rearranging or swapping pixels. In this embodiment, Remosaic processing can be implemented through a post-processing algorithm module or the first ISP module.

[0170] Demosaic is used to convert Bayer format data to RGB format. Demosaic can be understood as color interpolation, which restores the real-world colors that match the color display device from the Bayer data obtained from the sensor.

[0171] Cropping involves cropping the acquired image to obtain a field of view corresponding to the zoom level. Cropping can be achieved using the `crop` function. For example, the `crop` function can be used to obtain the field of view at 2x zoom.

[0172] Upsampling refers to restoring the resolution of a feature map to the resolution of the original image. Upsampling can be achieved by scaling the image using a scaling function. For example, after obtaining a 2x image, the scaling function can be used to enlarge the 2x image so that its size is the same as the 1x image.

[0173] Ambient illuminance refers to the intensity of light in the shooting environment in which the user is taking a picture. The value of ambient illuminance can be represented by the following indicators: lighting value (LV), lux, or luxindex, etc.

[0174] LV is used to estimate ambient light, and its specific calculation formula is as follows:

[0175]

[0176] Where Exposure is the exposure time, Aperture is the aperture size, Iso is the ISO, and Luma is the average value of Y in the XYZ color space.

[0177] In this application embodiment, the format of the image data output through the merge mode or DCG mode is named as the first image format, for example, the first image format is Bayer Raw.

[0178] In this application embodiment, the format of the image data output through non-merge mode (or non-merge + cropping mode) is named the second image format. For example, the second image format is quadra raw.

[0179] For different dynamic range scenarios, different zoom levels, and different ambient illumination, the sensor in this embodiment employs an appropriate image output method to improve image quality. The following will combine... Figure 2 Describe it.

[0180] refer to Figure 2 , Figure 2 This is a schematic diagram illustrating the image output methods of the sensor under different zoom levels, different dynamic range scenarios, and different ambient illuminance conditions according to embodiments of this application.

[0181] Figure 2 (1) is a schematic diagram of the sensor's output mode when the zoom magnification is greater than or equal to the first magnification and less than the second magnification. Figure 2 (2) is a schematic diagram of the sensor's output mode when the zoom magnification is greater than or equal to the second magnification. The second magnification is greater than the first magnification, meaning it is a zoom magnification greater than or equal to 2. For example, the first magnification is 1x, and the second magnification is 2x.

[0182] exist Figure 2 In (1), the ambient illuminance increases from left to right, and the dynamic range increases from bottom to top. Similarly, Figure 2 The same trend is observed in the changes of ambient illuminance and dynamic range in (2).

[0183] Regarding ambient illuminance, Figure 2 In this embodiment of the application, the ambient illuminance is divided into low illuminance scenes (or dark light environments) and high illuminance scenes (or bright light environments) based on a first illuminance threshold.

[0184] For example, if the ambient illuminance is greater than or equal to the first brightness threshold, it is a high-illuminance scene; if the ambient illuminance is less than the first brightness threshold, it is a low-illuminance scene.

[0185] It should be understood that the scenario where the ambient illuminance value equals the first luminance threshold is classified as a high-illuminance scene, but the embodiments of this application are not limited to this. For example, the scenario where the ambient illuminance equals the first luminance threshold can also be classified as a low-illuminance scene. The following description uses high-illuminance scenes and low-illuminance scenes.

[0186] As one possible implementation, ambient illuminance is represented by LV, and correspondingly, the first luminance threshold is the first LV value.

[0187] As one possible implementation, ambient illuminance is represented by the luxindex, and correspondingly, the first luminance threshold is the luxindex value. A larger luxindex value indicates lower ambient illuminance, while a smaller luxindex value indicates higher ambient illuminance.

[0188] Regarding dynamic range, it can be divided into high dynamic range and low dynamic range based on the first dynamic range constraint. If the dynamic range of the shooting scene meets the first dynamic range constraint, it is considered high dynamic range; if the dynamic range of the shooting scene does not meet the first dynamic range constraint, it is considered low dynamic range.

[0189] Optionally, the first DR constraint (which can also be specifically a DR value) can be determined based on the histogram of the RAW image of the scene being shot. Specifically, the dynamic range of the scene is determined based on the percentage of overexposed pixels and the percentage of underexposed pixels in the image.

[0190] For example, based on the histogram of the RAW image of the shooting scene, the first dynamic range (DR) constraint is determined according to a first ratio, a second ratio, and a third ratio. The first ratio refers to the proportion of pixels in the image pixel distribution that are greater than a first pixel value, and the second ratio refers to the proportion of pixels in the image pixel distribution that are less than a second pixel value. If both the first ratio and the second ratio are greater than the third ratio, the first DR constraint is considered to be satisfied, and the dynamic range is defined as high dynamic range. If neither the first ratio nor the second ratio is greater than the third ratio (for example, either the first ratio or the second ratio is not greater than the third ratio, or neither the first ratio nor the second ratio is greater than the third ratio), the first DR constraint is considered not satisfied, and the dynamic range is defined as low dynamic range. It should be understood that the description of the first DR constraint here is merely an exemplary description, and the embodiments of this application are not limited thereto.

[0191] It should be understood that the above description of the first luminance threshold and the first DR constraint condition is... Figure 2 Zhong (1) and Figure 2 Both (2) are applicable.

[0192] Figure 2(1) essentially illustrates two scenarios when the zoom magnification is greater than or equal to the first magnification and less than the second magnification. The distinction between these two scenarios depends on whether the dynamic range is high or low. Specifically:

[0193] Scenario 1, such as Figure 2 As shown in (1), when the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and it is a high dynamic range, the sensor will use the first mode for image output, i.e., DCG mode, regardless of whether it is a high-light or low-light scene. In a high dynamic range scene, using DCG mode can improve the dynamic range of the scene and help improve image quality.

[0194] Scenario 2, such as Figure 2 As shown in (1), when the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and it is a low dynamic range scene, the sensor uses the second mode for image output, i.e., the merge mode, regardless of whether the ambient illumination is a high-illuminance scene or a low-illuminance scene. Using the merge mode for image output under low dynamic range can improve the signal-to-noise ratio.

[0195] Figure 2 Figure (2) shows four scenarios when the zoom magnification is greater than or equal to the second magnification. The division of these four scenarios depends on whether the dynamic range is high dynamic range or low dynamic range, and also on whether the ambient illuminance is a high illuminance scene or a low illuminance scene.

[0196] Scenario 3, such as Figure 2 As shown in (2), when the zoom ratio is greater than or equal to the second zoom ratio and it is a low-light scene, if it is a high dynamic range, the sensor uses DCG mode to output the image.

[0197] Scenario 4, such as Figure 2 As shown in (2), when the zoom ratio is greater than or equal to the second zoom ratio and it is a low-light scene, if it is a low dynamic range, the sensor uses a merging mode to output the image.

[0198] Scenario 5, such as Figure 2 As shown in (2), when the zoom ratio is greater than or equal to the second zoom ratio and it is a high illumination scene, if it is a high dynamic range, the sensor uses the third mode to output the image, that is, non-merging mode + cropping processing.

[0199] Scenario 6, such as Figure 2 As shown in (2), when the zoom ratio is greater than or equal to the second zoom ratio and it is a high-light scene, if it is a low dynamic range, the sensor also uses the third mode to output the image, that is, non-merging mode + cropping processing.

[0200] In summary, under different dynamic ranges, different ambient light levels, and different zoom ratios, the sensor may output images in three modes: DCG (first mode), merge mode (second mode), and crop mode (third mode).

[0201] It should be noted that regardless of the method used by the sensor to output the image, the bit size (or bit width) of the image output by the sensor is consistent (for example, the bit size is 12 bits), which ensures that the bit size of the image input to the ISP module is also consistent.

[0202] In the above scenario, as the shooting scene changes, the sensor may involve switching between merge mode and DCG mode (including switching from DCG mode to merge mode, and vice versa). To avoid frequent switching, a first DR threshold and a second DR threshold are set to implement the switching between merge mode and DCG mode. The sensor's output mode is determined by comparing the DR value of the shooting scene with the first DR threshold or the second DR threshold. The DR value characterizes the dynamic range of the shooting scene. For example, if the DR value of the shooting scene is greater than the first DR threshold, the sensor switches from merge mode to DCG mode; if the DR value of the shooting scene is less than the second DR threshold, the sensor switches from DCG mode to merge mode, where the first DR threshold is greater than the second DR threshold. The DR value of the shooting scene can be calculated based on the histogram of the RAW image.

[0203] For example, assuming the first DR threshold is 1000 and the second DR threshold is 800, when the DR value of an image is greater than 1000, the mode can be switched from merge mode to DCG. As the shooting scene changes, the DR value of the image also changes. If the DR value decreases, the mode will not switch immediately, but will only switch from DCG mode to merge mode when the DR value of the shooting scene is determined to be less than 800, thus avoiding frequent switching. It should be understood that the DR threshold values ​​described here are exemplary, and the embodiments of this application are not limited thereto.

[0204] Furthermore, in the scenarios described above, the sensor may also be involved in switching between non-merged + cropped mode and DCG mode as the shooting scene changes (including switching from DCG mode to non-merged + cropped mode, and vice versa). For example, in high-light, high dynamic range scenes, when the zoom ratio changes from 1x to 2x or higher, the sensor's output mode switches from DCG mode to non-merged + cropped mode. As another example, in high dynamic range scenes with a zoom ratio of 2x, when the ambient light changes from low to high light, the sensor's output mode switches from DCG mode to non-merged + cropped mode. In scenarios where switching between non-merged + cropped mode and DCG mode occurs, a smooth transition in brightness and dynamic range can be achieved by adjusting modules in the ISP, such as tone mapping modules or automatic exposure (AE) modules.

[0205] The tone mapping module calculates the average brightness of the scene based on the current scene, selects a suitable brightness range based on this average brightness, and then maps the entire scene to this brightness range to obtain the correct result. The tone mapping module includes global tone mapping and local tone mapping.

[0206] The AE module is used to automatically adjust the sensor exposure time to adjust the image brightness when the ambient lighting conditions change.

[0207] The following describes the sensor's performance under different dynamic ranges, zoom ratios, and ambient light levels. Figure 2 The technical effects of the corresponding output modes shown are illustrated.

[0208] In high-light scenes where the zoom ratio equals the first magnification (or the zoom ratio is greater than the first magnification but less than the second magnification), and the dynamic range is high, the sensor uses DCG mode to output the image, which helps improve the dynamic range of the captured image. When the dynamic range is low, DCG mode is unnecessary, and the sensor can use image merging mode. Choosing the appropriate output method helps save power consumption of electronic devices. Furthermore, if the zoom ratio equals the first magnification, there is no resolution loss at the first magnification, and the image resolution is sufficient.

[0209] In low-light scenarios where the zoom ratio is equal to the first magnification (or the zoom ratio is greater than the first magnification but less than the second magnification), when the dynamic range is high, the sensor uses DCG mode to output the image, which helps improve the image's dynamic range. When the dynamic range is low, DCG mode is unnecessary; the sensor can use a merged mode to output the image, which helps save power consumption of electronic devices. Furthermore, there is no resolution loss at the first magnification, thus meeting the image resolution requirements.

[0210] In high dynamic range scenarios where the zoom ratio is equal to the first zoom ratio (or the zoom ratio is greater than the first zoom ratio but less than the second zoom ratio), dynamic range is the primary consideration regardless of whether it is a high-light or low-light scenario. Therefore, the sensor uses DCG mode to output images, which helps to improve the dynamic range of the image.

[0211] In low dynamic range scenarios where the zoom ratio is equal to the first zoom ratio (or the zoom ratio is greater than the first zoom ratio but less than the second zoom ratio), dynamic range does not need to be considered in either high-light or low-light scenarios. Therefore, the sensor can be output in a merged mode, which also helps to improve the signal-to-noise ratio of the image in low-light scenarios.

[0212] In high-light scenes with a zoom ratio greater than or equal to the second zoom level, when the dynamic range is high, compared to DCG mode, the sensor uses a non-merging + cropping mode to output the image, which improves the dynamic range of the preview scene. This ensures better image sharpness. Furthermore, the dynamic range of the shooting scene can be improved by generating separate long and short exposure frames and processing them through multi-frame post-processing algorithms (e.g., through a post-processing algorithm module and / or an ISP module), thus achieving a good effect in the shooting scene (balancing image sharpness and dynamic range). When the dynamic range is low, sharpness is the primary consideration, so the sensor uses a non-merging + cropping mode to output the image, which can improve image sharpness.

[0213] In low-light scenes with a zoom ratio greater than or equal to the second magnification, signal-to-noise ratio (SNR) is the primary consideration, and image merging mode is preferred. However, since DCG mode is also a type of merging mode, when the dynamic range is high, the sensor typically uses DCG mode to output the image, which helps improve the image's dynamic range. When the dynamic range is low, DCG mode is unnecessary, and the sensor can use merging mode to output the image, which helps improve the SNR.

[0214] In high dynamic range (HDR) scenarios with a zoom ratio greater than or equal to the second zoom level, both signal-to-noise ratio (SNR) and dynamic range are factors to consider when the ambient light is low. Since DCG mode is also a type of image merging mode, using DCG mode for image output not only ensures improved dynamic range but also helps improve the image's SNR. In high-light scenarios, the sensor uses a non-merging + cropping mode for image output, ensuring better image sharpness. Furthermore, the dynamic range of the shooting scene can be improved by generating separate long and short exposure frames and processing them through multi-frame post-processing algorithms (e.g., through a post-processing algorithm module and / or an ISP module), thus achieving a better effect in the shooting scene (balancing image sharpness and dynamic range). Therefore, the sensor does not use DCG mode for image output (DCG mode can improve the dynamic range of the preview scene) but instead uses a non-merging + cropping mode to improve image sharpness in the shooting scene.

[0215] In low dynamic range scenes with a zoom ratio greater than or equal to the second zoom ratio, the signal-to-noise ratio (SNR) of the image is the primary consideration when the ambient light is low. To ensure good sensitivity in low light, the sensor uses a merging mode to output images, which helps improve the SNR. In high light scenes, image sharpness is the primary consideration, so the sensor uses a non-merging + cropping mode to output images, which can improve image sharpness.

[0216] In low dynamic range and low-light scenes, regardless of the zoom ratio (greater than or equal to the first zoom ratio and less than the second zoom ratio, or greater than or equal to the second zoom ratio), the image signal-to-noise ratio is the primary consideration. Therefore, the sensor uses a merge mode to output images, which can ensure good light sensitivity in low light and help improve the image signal-to-noise ratio.

[0217] In low dynamic range and high illumination scenes, when the zoom ratio is equal to the first magnification, the resolution is sufficient, and dynamic range is not a concern. The sensor can output images in a merge mode to meet the requirements. When the zoom ratio is greater than the first magnification but less than the second magnification, dynamic range is also not a concern. In addition, due to the limitations of the platform size, to avoid the sensor outputting images too large and affecting the preview frame rate, the sensor still uses a merge mode for image output. When the zoom ratio is greater than or equal to the second magnification, image resolution becomes the primary consideration. Therefore, the sensor uses a non-merge + cropping mode for image output, which helps to improve image clarity.

[0218] In high dynamic range and low-light scenes, regardless of the zoom ratio (greater than or equal to the first zoom level and less than the second zoom level, or greater than or equal to the second zoom level), signal-to-noise ratio (SNR) is the primary consideration, and image merging mode is preferred. Since DCG mode is also a merging mode, sensors typically use DCG mode for image output in high dynamic range scenarios, which helps improve the image's dynamic range.

[0219] In high dynamic range and high illumination scenes, when the zoom ratio is greater than or equal to the first magnification and less than the second magnification, dynamic range is a priority factor. Therefore, the sensor uses DCG mode to output images, which helps to improve the dynamic range of the image. When the zoom ratio is greater than or equal to the second magnification, compared to DCG mode, the sensor uses non-merging + cropping mode to output images, which ensures better image sharpness. Furthermore, the dynamic range of the shooting scene can be improved by generating long and short exposure frames and using multi-frame post-processing algorithms (for example, through a post-processing algorithm module and / or an ISP module), thereby achieving a better effect in the shooting scene (balancing image sharpness and dynamic range).

[0220] In summary, this application embodiment selects an appropriate image output mode for the sensor in different scenarios, balancing power consumption, sharpness, and image quality. Furthermore, this application embodiment also allows for smooth switching between the three modes. Additionally, by uniformly controlling the ISP parameters, consistent color and brightness effects can be achieved in the images.

[0221] Additionally, it's understandable that for cases where the zoom magnification is less than the first magnification, should the following be adopted? Figure 2 The processing method shown in (1) may vary depending on the specific implementation of the product, and this application does not impose specific limitations on the embodiments. For example, when the zoom ratio selected by the user is greater than 0x and less than 1x, if the electronic device has an ultra-wide-angle camera, then the ultra-wide-angle camera can be switched for processing. Alternatively, when the zoom ratio selected by the user is greater than 0x and less than 1x, the camera may not be switched, and a different processing method may be used. Figure 2 The processing method shown in (1) is as follows.

[0222] In scenarios one and two above, when the zoom level is equal to the first magnification, the sensor output mode may be either DCG mode or merge mode. At this time, after the sensor outputs an image using DCG mode or merge mode, the processing of the back path remains consistent. The meaning of "consistent" is that when the zoom level is equal to the first magnification, (1) the processing of the preview stream when the sensor outputs an image using DCG mode is consistent with the processing of the preview stream when the sensor outputs an image using merge mode; (2) the processing of the image capture stream when the sensor outputs an image using DCG mode is consistent with the processing of the image capture stream when the sensor outputs an image using merge mode; (3) the processing of the thumbnail stream when the sensor outputs an image using DCG mode is consistent with the processing of the thumbnail stream when the sensor outputs an image using merge mode.

[0223] The following combination Figure 3 and Figure 4 Describe the processing flow of the back path (including the preview path, the shooting path, and the thumbnail path) when the zoom level is equal to the first zoom level.

[0224] Figure 3 This is a schematic block diagram of a photographing system 300 according to an embodiment of this application. Figure 3 As shown, the photography system 300 includes a decision module, an ISP parameter configuration module, a camera, a first buffer, a first image processing module, an ISP first module, an ISP second module, and an ISP third module. The camera includes optical lenses and an image sensor.

[0225] It should be understood that the ISP first module, ISP second module, and ISP third module involved in the preview stream, the ISP first module, ISP second module, and ISP third module involved in the photo stream, and the ISP first module, ISP second module, and ISP third module involved in the thumbnail stream may be partially or completely reused, or they may be independent of each other. This application embodiment does not limit this.

[0226] For example, the preview stream corresponds to one set of ISP modules 1, 2, and 3; the photo stream corresponds to one set of ISP modules 1, 2, and 3; and the thumbnail stream corresponds to one set of ISP modules 1, 2, and 3. Alternatively, the preview stream, photo stream, and thumbnail stream may share the same set of ISP modules 1, 2, and 3. Another example is where the preview stream and photo stream reuse the ISP module 1, the preview stream corresponds to one set of ISP modules 2 and 3, the photo stream corresponds to one set of ISP modules 2 and 3, and the thumbnail stream corresponds to one set of ISP modules 2 and 3, and so on.

[0227] It is understood that the above descriptions of the ISP first module, ISP second module and ISP third module are merely illustrative descriptions, and the embodiments of this application are not limited thereto.

[0228] It is also understandable that the above descriptions of ISP Module 1, ISP Module 2, and ISP Module 3 are relevant to the following text. Figure 5 and Figure 7 This also applies, and will not be elaborated further below.

[0229] The decision module determines the parameter configurations of each module (including the camera, ISP first module, ISP second module, ISP third module, and post-processing algorithm module) based on the dynamic range, zoom ratio, and ambient illumination of the shooting scene. The relevant parameter configurations for each module can be distributed to each module via the ISP parameter configuration module. In other words, the decision module controls or allocates the functions or roles of each module through the ISP parameter configuration module, such as enabling or disabling which ISP modules, and how the ISP modules process image data.

[0230] As one possible implementation, in a full-illumination scene (including high-illumination and low-illumination scenes) where the zoom magnification is equal to the first magnification, and in a high dynamic range scene, the decision module determines at least the following parameter configuration: the sensor output mode is DCG mode (corresponding to the first mode).

[0231] As one possible implementation, in a full-illumination scene (including high-illumination and low-illumination scenes) where the zoom magnification is equal to the first magnification, and in a scene with low dynamic range, the decision module determines at least the following parameter configuration: the sensor output mode is a merging mode (corresponding to the second mode).

[0232] Taking 1x zoom as an example, there is no resolution loss in 1x scenes, which meets the image resolution requirements. In 1x high dynamic range scenes, the sensor uses DCG mode to output images, which can improve the dynamic range of the scene. Additionally, the merge mode can improve the image signal-to-noise ratio and enhance image quality in low-light environments. Therefore, in full-illumination scenes with 1x zoom, and in low dynamic range scenes, the merge mode can be used to output images with a good signal-to-noise ratio. In high dynamic range scenes, using DCG mode can improve the dynamic effects of the image, thereby improving image quality.

[0233] Optionally, the decision module can also configure other parameters of each ISP module, such as color, brightness effect, zoom ratio, etc., to control and ensure that the ISP parameter configuration of each ISP module is consistent.

[0234] The camera is used to acquire image signals and process the acquired image signals through a merging mode or a DCG mode.

[0235] like Figure 3 As shown, the sensor outputs the first image data through DCG mode or merge mode, and the format of the first image data is the first image format (Bayer format).

[0236] exist Figure 3 In this process, the first cache is used to store the first image data output by the camera, so that the data of the captured frame can be obtained from the first cache as soon as the photo capture command is triggered.

[0237] The first image processing module is used to process image data from the imaging path. The first image processing module includes a post-processing algorithm module, an ISP first module, an ISP second module, and an ISP third module.

[0238] The post-processing algorithm module is used to process images offline in photo mode to improve image quality. For example, the post-processing algorithm module is used to perform one or more of the following processes on the image: multi-frame fusion noise reduction processing, multi-frame HDR processing, etc.

[0239] exist Figure 3 In the preview stream, the data stream passes through the ISP first module, ISP second module, and ISP third module. The ISP first module does not process the data obtained from the sensor; in other words, the data is transmitted to the ISP second module after passing through the ISP first module. The ISP second module processes the Bayer domain and outputs RGB format data. The ISP third module processes either the RGB or YUV domain and outputs YUV format data.

[0240] The following describes the further processing functions that the ISP second module and ISP third module can include. For clarity, when the ISP second module and ISP third module are enabled, the processing described below for the ISP second module and ISP third module will not apply in other modes (hereinafter referred to as second and third modes) or to the camera system (e.g., Figure 5 or Figure 7 (such as the camera system) or processing procedures (e.g.) Figure 4 , Figure 6 or Figure 8 This also applies to the processing flow within the ISP (Image Stream, Thumbnail Stream) or other pathways. Further processing functions will not be elaborated upon in the sections on ISP Module 2 and ISP Module 3 mentioned below.

[0241] Optionally, the second ISP module may also include one or more of the following processing methods: bad pixel correction (BPC), black level correction (BLC), lens shading correction (LSC), automatic white balance (AWB), bayer domain noise reduction (NR), Demosaic, etc.

[0242] Optionally, the third module of the ISP may also include one or more of the following processing methods: color correction (CC), YUV domain noise reduction (NR), color enhancement (CE), sharpening, tone mapping, etc.

[0243] like Figure 3 As shown, for the image capture stream, the data stream passes through the first buffer and the first image processing module. The data flow within the first image processing module can also have various paths. This embodiment does not specifically limit the processing of the first data after it enters the first image processing module. Within the first image processing module, the decision module can select which modules the data stream passes through, or which modules it does not pass through.

[0244] Figure 3 The diagram illustrates two possible paths of data flow within the first image processing module. One possible implementation is as follows: Figure 3 As shown, the first data obtained from the first buffer, after being sent to the first image processing module, is first transmitted to the post-processing algorithm module, that is, without passing through the first ISP module, and then through the second ISP module and the third ISP module. The second ISP module and the post-processing algorithm module are connected by a bidirectional arrow (i.e., bidirectional interaction); the third ISP module and the post-processing algorithm module are also connected by a bidirectional arrow.

[0245] For example, after the data is processed by the post-processing algorithm module, it can be sent to the second ISP module for further processing. Then, after the second ISP module completes its processing, the data can be returned to the post-processing algorithm module for further processing, or it can be transmitted to the third ISP module for further processing. After the post-processing algorithm module completes its processing, the data can be sent to the third ISP module for further processing. Finally, after the third ISP module completes its processing, the data is returned to the post-processing algorithm module for further processing, which helps to improve image quality.

[0246] Another possible implementation, such as Figure 3As shown, after the first image data is sent to the first image processing module, it is first transmitted to the first ISP module. The first ISP module processes the first image data (e.g., binning, HDR fusion, etc.). Next, the first ISP module sends the processed image data to the post-processing algorithm module. After processing, the post-processing algorithm module can transmit the processed image data to the second ISP module. After processing by the second ISP module, the image data can be returned to the post-processing algorithm module for further processing, or it can be transmitted to the third ISP module for further processing. Then, after processing by the post-processing algorithm module, the data can be sent to the third ISP module for further processing. Finally, after processing by the third ISP module, the data is returned to the post-processing algorithm module for further processing, which helps to improve image quality.

[0247] like Figure 3 As shown, for the thumbnail stream, the data stream passes through the first buffer, the second ISP module, and the third ISP module. The reason the data stream doesn't need to pass through the first ISP module is that the data obtained from the first buffer is in the first image format, i.e., Bayer data, therefore, the first ISP module doesn't need to process it to obtain Bayer data. The thumbnail stream is processed offline, so the second and third ISP modules can be called to process the image. Additionally, it should be noted that because thumbnails prioritize image output speed, the image clarity requirement is lower than that of the capture stream; therefore, the thumbnail stream does not need to undergo post-processing algorithm module processing.

[0248] exist Figure 3 Based on this, the embodiments of this application design a corresponding post-processing flow, specifically involving preview stream, image capture stream, and thumbnail stream. The following, in conjunction with... Figure 4 Description uses Figure 3 The photo-taking process of the photo-taking system 300 is shown.

[0249] Figure 4 A schematic flowchart illustrating the shooting method when the zoom level is equal to the first magnification is shown. Figure 4 As shown, the method for taking a photo includes the following steps:

[0250] Step 401: The acquired image signal is processed using either a merging mode or a DCG mode to obtain first image data. The first image data uses a first image format. The first image format is Bayer Raw.

[0251] In other words, the sensor uses DCG mode or merging mode to output a Bayer raw image, such as the first image data. The first image data can be sent to the preview path for processing, or it can be stored in the first buffer for subsequent image capture path retrieval.

[0252] Step 402: Send the first image data into the preview channel, and at the same time, store the first image data in the first cache.

[0253] The first cache is used to store the data of the captured images. After the capture command is triggered, the captured image data can be retrieved from the first cache and returned to the user.

[0254] This application does not specifically limit the type of the first buffer. The first buffer can be a general buffer or a buffer specific to a certain shooting mode. For example, the first buffer is a zero-shutter lag (ZSL) buffer. The ZSL buffer is used to store the image data directly output from the sensor. In ZSL mode, after the shooting command is issued, the system selects the image data of the corresponding frame from the ZSL buffer and sends it to the shooting stream and thumbnail stream for processing.

[0255] Step 403-1: The original data of the first image is de-mosaiced by the second ISP module to obtain RGB format data;

[0256] Step 403-2: The RGB format data is processed by the third module of the ISP to obtain YUV format data, and a preview image is output.

[0257] Steps 403-1 to 403-2 above describe the workflow of the preview path. This process can improve the image quality of the preview image.

[0258] Step 404: Receive the user's first operation, which is used to trigger taking a picture.

[0259] The first operation is a photo-taking command. This application does not specifically limit the form of the first operation in its embodiments. For example, the first operation may be manually clicking the shooting control, or the first operation may be controlling the phone to take a photo via voice, or the first operation may be a timed activation of the photo-taking function (e.g., automatically taking a photo after 3 seconds). For example, the first operation is a user clicking... Figure 1 The operation of the shooting control 14 shown.

[0260] Step 405: In response to the first operation, retrieve the first image data from the first cache.

[0261] Upon receiving a photo capture command, the image frame with the corresponding photo capture timestamp needs to be retrieved from the first cache. For example, the data of the image frame is the first image data.

[0262] Step 406-1: Perform image processing on the first image data using the post-processing algorithm module or the second ISP module to obtain the second image data.

[0263] For example, the first image data can be processed in the Bayer domain by the post-processing algorithm module or the second ISP module to obtain RGB format data. It is understood that the post-processing algorithm module or the second ISP module can also perform other image processing operations; please refer to the previous description for details, which will not be repeated here.

[0264] Step 406-2: The second image data is processed by the third module of the ISP using RGB or YUV processing to obtain YUV format data and output the captured image.

[0265] Steps 404 to 406-2 above describe the workflow of the image capture path. This process can improve the image quality of captured images.

[0266] Step 407-1: The first image data is processed by the second ISP module to obtain RGB format data;

[0267] Step 407-2: The RGB format data is processed by the third module of the ISP to obtain YUV format data and output thumbnails.

[0268] Steps 407-1 to 407-2 above describe the workflow of the thumbnail stream. This process can improve the image quality of thumbnails.

[0269] In scenarios one, two, three, and four above, the sensor output may use either DCG mode or merging mode. The specific mode used by the sensor in each scenario has been described in the preceding scenarios and will not be repeated here.

[0270] In scenarios where the zoom ratio is greater than the first magnification but less than the second magnification, and in low-light scenarios where the zoom ratio is greater than or equal to the second magnification, the processing of the back-path remains consistent after the sensor outputs images in DCG mode or merge mode. The meaning of "maintaining consistency" has already been explained above and will not be repeated here.

[0271] The following combination Figure 5 and Figure 6 Another processing flow describes the shooting scene (including scenes with a zoom ratio greater than the first zoom ratio but less than the second zoom ratio, and low-light scenes with a zoom ratio greater than or equal to the second zoom ratio).

[0272] Figure 5This is a schematic block diagram of a photographing system 500 according to an embodiment of this application. The photographing system 500 includes the same components as the photographing system 300. The modules included in the photographing system 500 will not be described again here. Figure 5 and Figure 3 The differences lie at least in the following aspects: Figure 5 In the process, the ISP third module or post-processing algorithm module needs to perform corresponding cropping and upsampling processing based on the zoom ratio. Figure 5 The reason why the ISP third module or post-processing algorithm module needs to perform cropping and upsampling is that: Figure 5 The illustrated camera system is designed for scenes with a zoom ratio greater than the first magnification but less than the second magnification, and for scenes with a zoom ratio greater than or equal to the second magnification in low light. Figure 3 The photography system shown is designed for scenes where the zoom level is equal to the first magnification (e.g., 1x). Figure 3 The ISP third module or post-processing algorithm module in the process does not require cropping and upsampling.

[0273] As a possible implementation, in the following two cases: (1) the zoom ratio is greater than or equal to the second zoom ratio and it is a low-light high dynamic scene; (2) the zoom ratio is greater than the first zoom ratio and less than the second zoom ratio and it is a high dynamic scene with full ambient light (including high light and low light), the decision module determines at least the following parameter configurations: the output mode of the camera is DCG mode, the image processing of the ISP third module in the preview path includes cropping and upsampling, the image processing of the post-processing algorithm module or the ISP third module in the shooting path includes cropping and upsampling, and the image processing of the ISP third module in the thumbnail path includes cropping and upsampling.

[0274] As a possible implementation, in the following two cases: (1) the zoom ratio is greater than or equal to the second zoom ratio and it is a low-light, low-dynamic scene; (2) the zoom ratio is greater than the first zoom ratio and less than the second zoom ratio and it is a low-dynamic scene with full ambient light (including high light and low light), the decision module determines at least the following parameter configurations: the image output mode of the camera is a merging mode; the image processing of the ISP third module in the preview path includes cropping and upsampling; the image processing of the post-processing algorithm module or the ISP third module in the shooting path includes cropping and upsampling; and the image processing of the ISP third module in the thumbnail path includes cropping and upsampling.

[0275] exist Figure 5In the process, the sensor outputs a third image data using either a merge mode or a DCG mode, and this third image data adopts the first image format. After obtaining the third image data, it is sent to the preview path, and simultaneously, the third image data is stored in the first cache. The description of the first cache can be found in the previous description (e.g., the description of the first cache below step 402 above), and will not be repeated here.

[0276] For example, in low-light scenes with a 2x zoom ratio, using a merged mode in low dynamic range mode can improve the signal-to-noise ratio; while using DCG mode in high dynamic range mode can improve the dynamic range. Additionally, cropping is required based on the 2x zoom ratio to obtain a field of view corresponding to 2x. The preview or captured image should be the same size as the 1x image, therefore upsampling is necessary after cropping.

[0277] Similarly, this also involves the processing flow of preview streams, photo streams, and thumbnail streams. Figure 5 The data flow is illustrated with examples.

[0278] Figure 5 The data flow in the preview stream and Figure 3 The data flow in the preview stream is the same. Similarly, in Figure 5 In this process, the first ISP module does not need to process the data obtained from the sensor; in other words, the data is transmitted to the second ISP module after passing through the first ISP module. Furthermore, the third ISP module is used for image cropping and upsampling. Cropping yields the field of view for the corresponding zoom level; upsampling produces a preview image.

[0279] Figure 5 Data trends in the photo stream and Figure 3 The data flow in the photo stream follows the same pattern. Figure 5 and Figure 3 The first image processing module in [the system] contains the same modules. Similarly, Figure 5 The data flow within the first image processing module can also take several forms. The difference between the two lies at least in... Figure 5 In the post-processing algorithm module or the third module of the ISP, the processing includes cropping and upsampling to ensure that the image resolution remains unchanged.

[0280] like Figure 5As shown, for the thumbnail stream, the data stream passes through the first buffer, the second ISP module, and the third ISP module. The reason the data stream doesn't need to pass through the first ISP module here is that the data obtained from the first buffer is in Bayer data format, therefore, the first ISP module doesn't need to process it to obtain Bayer data. Furthermore, this thumbnail stream is processed offline, so the second and third ISP modules can be called to process the image. The third ISP module is used to crop and upsample the image to ensure that the resolution of the thumbnail image remains unchanged.

[0281] exist Figure 5 Based on this, the embodiments of this application design a corresponding post-processing flow, specifically involving preview stream, image capture stream, and thumbnail stream. The following, in conjunction with... Figure 6 Description uses Figure 5 The image capture process of the 500-bit photo capture system shown is illustrated.

[0282] Figure 6 A schematic flowchart of another photographing method is shown. (e.g.) Figure 6 As shown, the method for taking a photo includes the following steps:

[0283] Step 601: For the acquired image signal, process it through a merging mode or DCG mode to obtain third image data. The third image data adopts a first image format. The first image format is Bayer Raw.

[0284] In other words, the sensor uses a merge mode to output a Bayer raw image, such as the third image data. This third image data can be sent to the preview path for processing or stored in the first buffer for subsequent image capture.

[0285] Step 602: Send the third image data into the preview path, and at the same time, store the third image data in the first cache.

[0286] The description of the first cache can be found in the previous text, and will not be repeated here.

[0287] Step 603-1: The third image data is demosaiced by the second module of the ISP to obtain RGB format data;

[0288] Step 603-2: The RGB format data is processed by the third module of the ISP to obtain YUV format data, and the YUV format data is cropped and upsampled to output a preview image.

[0289] Steps 603-1 to 603-2 above describe the workflow of the preview path. These steps can improve the quality of the preview image.

[0290] Step 604: Receive the user's second operation, which is used to trigger taking a picture.

[0291] The second operation is the photo-taking command. The description of the second operation can be found in the description of the first operation above, and will not be repeated here.

[0292] Step 605: In response to the second operation, retrieve the third image data from the first cache.

[0293] Step 606-1: Perform image processing on the third image data using the post-processing algorithm module or the second ISP module to obtain the fourth image data.

[0294] For example, the third image data can be processed in the Bayer domain by the post-processing algorithm module or the second ISP module to obtain RGB format data. It is understood that the post-processing algorithm module or the second ISP module can also perform other image processing operations, as described above, and will not be repeated here.

[0295] Step 606-2: The fourth image data is cropped and upsampled by the post-processing algorithm module or the third ISP module to output the captured image.

[0296] Steps 604 to 606-2 above describe the workflow of the photo capture stream in the second mode. These steps can improve the quality of captured images.

[0297] Step 607-1: The third image data is processed by the second ISP module to obtain RGB format data;

[0298] Step 607-2: The RGB format data is processed by the third module of the ISP to obtain YUV format data, and the YUV format data is cropped and upsampled to output a thumbnail.

[0299] Steps 607-1 to 607-2 above describe the workflow of the thumbnail stream. These steps can improve the quality of thumbnails.

[0300] In scenarios five and six above, the sensor output method is the third mode (i.e., non-merge + crop mode). The following combines... Figure 7 and Figure 8 Describe the processing flow of the post-pathway.

[0301] Figure 7This is a schematic block diagram of a photographing system 700 according to an embodiment of this application. The photographing system 700 includes the same components as the photographing system 300. The modules included in the photographing system 700 will not be described again here. Figure 7 and Figure 3 The differences lie at least in the following aspects: Figure 7 In this process, the sensor output method is non-merging mode + cropping processing; the first module of ISP or the post-processing algorithm module needs to perform Remosaic processing.

[0302] As one possible implementation, when the zoom ratio is greater than or equal to the second zoom ratio and it is a high-light scene, regardless of whether it is a high dynamic range or a low dynamic range, the decision module determines at least the following parameter configurations: the image output method of the camera is non-merging mode + cropping processing, the image processing of the first ISP module in the preview path includes remosaic processing, the first ISP module or post-processing algorithm module in the shooting path needs to perform remosaic processing, and the image processing of the first ISP module in the thumbnail path includes remosaic processing.

[0303] exist Figure 7 In this process, the camera is used to acquire image signals, and the acquired image signals are processed using a non-merging mode plus cropping to output fifth image data. The fifth image data uses a second image format (i.e., Quadra Raw). In other words, the sensor uses a non-merging mode plus cropping method to output the image.

[0304] It's important to note that since the sensor output here uses a non-merging mode, it doesn't lose image resolution (or sharpness) compared to the merging mode. At high zoom levels (e.g., zoom levels greater than or equal to the second magnification), image sharpness must be prioritized; therefore, the non-merging mode is more suitable for high zoom levels. Furthermore, after processing the image signal using the non-merging mode, only cropping is needed to obtain the image with the corresponding field of view at high zoom levels; no further upsampling is required.

[0305] For example, assuming the sensor resolution is 50M (or 50 million pixels) and the zoom ratio is 2x, after the sensor outputs the image using non-merging mode, it still needs to be cropped by 2x field of view (FOV) to obtain a 12.5M image.

[0306] The first cache is used to store the fifth image data output by the camera, so that the data of the captured frame can be retrieved from the first cache as soon as the photo capture command is triggered.

[0307] The first image processing module is used to process image data from the imaging path. The first image processing module includes a post-processing algorithm module, an ISP first module, an ISP second module, and an ISP third module.

[0308] The post-processing algorithm module is used to perform remosaic processing on images offline in photo mode to improve image quality.

[0309] Optionally, the post-processing algorithm module can also be used to perform one or more of the following processes on the image: multi-frame fusion noise reduction processing, multi-frame HDR processing, etc.

[0310] like Figure 7 As shown, for the preview stream, the data stream passes through the ISP first module, ISP second module, and ISP third module. The ISP first module performs Remosaic processing on the fifth image data. The ISP second module performs Bayer domain processing, outputting RGB format data. The ISP third module performs either RGB or YUV domain processing, outputting YUV format data.

[0311] The second or third ISP module may also include further processing functions, which can be referred to in the previous text and will not be repeated here.

[0312] Optionally, the ISP first module also includes one or more of the following processes: binning, HDR fusion, etc. For clarity, when the ISP first module is enabled, the further processing functions included in the ISP first module described herein are not relevant to other places where the ISP first module appears in this application embodiment (e.g., Figure 3 or Figure 5 The ISP first module (etc.) can also be applied.

[0313] like Figure 7 As shown, for the image capture stream, the data stream passes through the first buffer and the first image processing module. The data flow within the first image processing module can also have various paths. This embodiment does not specifically limit the processing of the fifth image data after it is sent into the first image processing module. The decision module can select which modules the data passes through, or which modules it does not pass through, within the first image processing module.

[0314] Figure 7 The diagram illustrates two possible paths of data flow within the first image processing module. One possible implementation is as follows: Figure 7As shown, the fifth image data obtained from the first buffer, after being sent to the first image processing module, is first transported to the post-processing algorithm module, that is, without passing through the first ISP module. The post-processing algorithm module performs Remosaic processing on the fifth image data, and then it passes through the second ISP module and the third ISP module. The second ISP module and the post-processing algorithm module are connected by a bidirectional arrow (i.e., bidirectional interaction); the third ISP module and the post-processing algorithm module are also connected by a bidirectional arrow.

[0315] For example, after the fifth image data undergoes Remosaic processing by the post-processing algorithm module, it can be sent to the second ISP module for further processing. Then, after the second ISP module completes its processing, the image data can be returned to the post-processing algorithm module for further processing, or it can be transmitted to the third ISP module for further processing. After the post-processing algorithm module completes its processing, the data can be sent to the third ISP module for further processing. Finally, after the third ISP module completes its processing, the data is returned to the post-processing algorithm module for further processing, which helps to improve image quality.

[0316] Since the noise model is not destroyed when the fifth image data (in quadra raw format) is sent to the post-processing algorithm module, the post-processing algorithm module can perform noise reduction based on the fifth image data (quadra raw). In this way, the noise modeling method can achieve better noise reduction results. Compared to sending the fifth image data to the ISP or the camera's internal remosaic processing, this method achieves better noise reduction, improves the clarity of the captured image, and enhances the user's shooting experience.

[0317] Another possible implementation, such as Figure 7 As shown, after the fifth image data is sent to the first image processing module, it is first transmitted to the first ISP module. The first ISP module performs Remosaic processing on the fifth image data. Then, the image data processed by the first ISP module is sent to the post-processing algorithm module. After processing, the post-processing algorithm module can transmit the processed image data to the second ISP module. After processing by the second ISP module, the image data can be returned to the post-processing algorithm module for further processing, or it can be transmitted to the third ISP module for further processing. Then, after processing by the post-processing algorithm module, the data can be sent to the third ISP module for further processing. Finally, after processing by the third ISP module, the data is returned to the post-processing algorithm module for further processing, which helps to improve image quality.

[0318] like Figure 7As shown, for the thumbnail stream, the data stream passes through a first buffer, ISP module 1, ISP module 2, and ISP module 3. ISP module 1 performs Remosaic processing on the fifth image data, resulting in Bayer raw image data. The Bayer raw data obtained after processing by ISP module 1 can be further processed by ISP modules 2 and 3. ISP module 2 performs Bayer domain processing, outputting RGB format data. ISP module 3 performs RGB or YUV domain processing, outputting YUV format data.

[0319] It should be noted that, since thumbnails need to prioritize image output speed, the image clarity requirements for thumbnail streams are lower than those for image capture streams. Therefore, thumbnail streams do not require post-processing algorithms to process the images.

[0320] This application's embodiment designs a post-processing flow for image output when the sensor uses a third mode, specifically involving the preview stream, image capture stream, and thumbnail stream. The following, in conjunction with... Figure 8 Description uses Figure 7 The photo-taking process of the photo-taking system 700 is shown.

[0321] Figure 8 A schematic flowchart illustrating another shooting method for a shooting scenario (i.e., a high-light scene with a zoom ratio greater than or equal to the second magnification). Figure 8 As shown, the method for taking a photo includes the following steps:

[0322] Step 801: For the acquired image signal, the fifth image data is obtained through non-merging mode and cropping processing, and the fifth image data adopts the second image format.

[0323] Specifically, the sensor outputs the acquired image signal in a non-merging mode and crops the image based on the quadra raw format (for example, using the crop function) to obtain a quadra raw image.

[0324] Step 802: Send the fifth image data into the preview channel, and at the same time, store the fifth image data in the first buffer.

[0325] The description of the first cache can be found in the previous text, and will not be repeated here.

[0326] Step 803-1: The fifth image data is processed by the first ISP module to obtain the sixth image data, and then the sixth image data is sent to the second ISP module. The sixth image data adopts the first image format (Bayer Raw).

[0327] In other words, the image output by the sensor is in Quadra Raw format, and after processing by the first module of the ISP, it becomes a Bayer Raw format image.

[0328] Step 803-2: The sixth image data is de-mosaiced by the second ISP module to obtain RGB format data.

[0329] Step 803-3: The RGB format data is processed by the third module of the ISP to obtain YUV format data, and a preview image is output.

[0330] Steps 803-1 to 803-3 above describe the workflow of the preview stream.

[0331] Step 804: Receive the user's third operation, which is used to trigger taking a picture.

[0332] The third operation can be referred to the description of the first operation above, and will not be repeated here.

[0333] Step 805, in response to the third operation, retrieve the fifth image data from the first cache.

[0334] Upon receiving the photo capture command, it is necessary to retrieve the image frame with the corresponding photo capture timestamp from the first buffer. For example, the data of the image frame is the fifth image data.

[0335] Step 806-1: The fifth image data is processed using a post-processing algorithm module to obtain the seventh image data. The seventh image data is in the first image format (bayer raw).

[0336] Furthermore, as mentioned earlier, in high-light scenes where the zoom ratio is greater than or equal to the second zoom ratio, and the dynamic range is high dynamic range, the dynamic range of the scene can be improved by generating separate long and short exposure frames and using a multi-frame post-processing algorithm. Accordingly, in step 806-1, optionally, the post-processing algorithm module is also used to perform multi-frame fusion processing on multiple image frames output by the sensor in DCG mode to improve the dynamic range of the scene.

[0337] Alternatively, the first module of the ISP can perform multi-frame fusion processing on multiple image frames output by the sensor in DCG mode.

[0338] Step 806-2: The seventh image data is de-mosaiced by the second ISP module to obtain RGB format data.

[0339] Step 806-3: The RGB format data is processed by the third module of the ISP to obtain YUV format data and output the captured image.

[0340] Steps 804 to 806-3 above describe the workflow of the image capture stream. In the processing of the image capture path, compared to sending the fifth image data to the ISP or the camera itself for remosaic processing, using the post-processing algorithm module to remosaic process the fifth image data can achieve better noise reduction and improve the clarity of the captured image.

[0341] In addition, in the image capture path, Remosaic processing can also be performed by the first module of the ISP. Figure 8 The example shown only illustrates the case of Remosaic processing via the post-processing algorithm module.

[0342] Optionally, step 806-1 above can also be replaced by: performing Remosaic processing on the fifth image data through the first ISP module to obtain the seventh image data. The seventh image data adopts the first image format (bayer raw). Correspondingly, steps 806-2 and 806-3 can also be adjusted accordingly. For example, step 806-2 can be replaced by: performing image processing on the seventh image data through the post-processing algorithm module, and sending the image data processed by the post-processing algorithm module to the second ISP module for processing; step 806-3 can be replaced by: processing the image data output by the second ISP module through the third ISP module to output the captured image. The specific processing method of the image data by the post-processing algorithm module, the second ISP module, or the third ISP module is not limited, and can be referred to the above description.

[0343] Step 807-1: The fifth image data is processed by the first module of the ISP to obtain data in Bayer format.

[0344] Step 807-2: The Bayer format data is processed by Demosaic through the second ISP module to obtain RGB format data;

[0345] Step 807-3: The RGB format data is processed by the third module of the ISP to obtain YUV format data and output a thumbnail.

[0346] Steps 804, 805, and 807-1 to 807-3 above describe the workflow of the thumbnail stream. These steps can improve the quality of thumbnails.

[0347] It is understandable that the above Figure 3 , Figure 5 and Figure 7 The photographic system shown is merely an exemplary description, and the embodiments of this application are not limited thereto.

[0348] It is understandable that the above Figure 4 , Figure 6 and Figure 8 The process of the photographing method shown is merely an exemplary description and does not constitute a limitation on the protection scope of the embodiments of this application.

[0349] The following combination Figure 9 and Figure 10 This application describes the software system and hardware architecture used in its embodiments.

[0350] Figure 9 This is a schematic diagram of the architecture (including software system and some hardware) used in the embodiments of this application. Figure 9 As shown, the application architecture is divided into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the application architecture can be divided into five layers, from top to bottom: the application layer, the application framework layer, the hardware abstraction layer (HAL), the driver layer, and the hardware layer.

[0351] like Figure 9 As shown, the application layer includes the camera and the gallery.

[0352] Understandable. Figure 9 The examples shown are only a portion of the applications; in fact, the application layer can include other applications as well, and this application does not limit this. For example, the application layer may also include applications such as messaging, alarm clock, weather, stopwatch, compass, timer, flashlight, calendar, and Alipay.

[0353] like Figure 9 As shown, the application framework layer includes a camera access interface. The camera access interface includes camera management and camera devices. The hardware abstraction layer includes a camera hardware abstraction layer and a camera algorithm library. The camera hardware abstraction layer includes multiple camera devices. The camera algorithm library includes post-processing algorithm modules and decision-making modules.

[0354] It should be understood that the decision-making module can also be placed in other layers. As one possible implementation, the decision-making module can be placed in the application layer or the application framework layer.

[0355] The driver layer is used to drive hardware resources. The driver layer can include multiple driver modules. For example... Figure 9 As shown, the driver layer includes camera device drivers, digital signal processor drivers, and graphics processor drivers, etc.

[0356] The hardware layer includes sensors, an image signal processor, a digital signal processor, and a graphics processor. The sensors include multiple sensors, a Time-of-Flight (TOF) camera, and a multispectral sensor. The image signal processor includes an ISP first module, an ISP second module, and an ISP third module.

[0357] For example, a user can tap the camera application. When the user taps the camera to take a picture, the shooting command is sent to the camera hardware abstraction layer (HAL) via the camera access interface. The HAL calls the camera device driver and the camera algorithm library. The decision module in the camera algorithm library determines the shooting mode (e.g., mode 1, mode 2, or mode 3) based on the zoom ratio, ambient light, and dynamic range value, and sends the configured parameters (including sensor output method, parameter configurations for each ISP module, and parameter configurations for the post-processing algorithm module) to the HAL. The HAL sends the parameters configured by the decision module to the camera device driver. The camera device driver sends the configuration parameters sent by the HAL to the hardware layer; for example, it sends the sensor output method to the sensor and the parameter configurations for each ISP module to the image signal processor. The sensor outputs the image based on the sensor output method. The image signal processor performs corresponding processing based on the parameter configurations of each ISP module. The camera algorithm library is also used to send digital signals to the digital signal processor driver in the driver layer, so that the digital signal processor driver can call the digital signal processor in the hardware layer for digital signal processing. The digital signal processor (DSP) can return the processed digital signals to the camera algorithm library via its driver. The camera algorithm library also sends digital signals to the graphics signal processor (GSP) driver in the driver layer, allowing the GSP driver to invoke the graphics processor (GPU) in the hardware layer for digital signal processing. The GSP can then return the processed image data to the camera algorithm library via its driver.

[0358] Additionally, the image output from the image signal processor can be sent to the camera device driver. The camera device driver can then send the image output from the image signal processor to the camera hardware abstraction layer. The camera hardware abstraction layer can then send the image to the post-processing algorithm module for further processing, or it can send the image to the camera access interface. The camera access interface can then send the image returned by the camera hardware abstraction layer to the camera.

[0359] The software system used in the embodiments of this application has been described in detail above. The following section, in conjunction with... Figure 10 Describe the hardware system of electronic device 1000.

[0360] Figure 10 A schematic diagram of the structure of an electronic device 1000 applicable to this application is shown.

[0361] Electronic device 1000 may include processor 110, external memory interface 120, internal memory 121, universal serial bus (USB) interface 130, charging management module 140, power management module 141, battery 142, antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0362] It should be noted that, Figure 10 The structure shown does not constitute a specific limitation on the electronic device 1000. In other embodiments of this application, the electronic device 1000 may include more than Figure 10 The components shown may include more or fewer components, or the electronic device 1000 may include... Figure 10 The components shown may be a combination of certain components, or the electronic device 1000 may include... Figure 10 The components shown are sub-components of certain components. For example, Figure 10 The proximity sensor 180G shown is optional. Figure 10 The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0363] Processor 110 may include one or more processing units. For example, processor 110 may include at least one of the following processing units: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, video codec, digital signal processor (DSP), baseband processor, and neural network processing unit (NPU). These different processing units may be independent devices or integrated devices.

[0364] The controller can generate operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0365] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0366] In some embodiments, processor 110 may include one or more interfaces. For example, processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and a USB interface.

[0367] Figure 10 The connection relationships between the modules shown are merely illustrative and do not constitute a limitation on the connection relationships between the modules of the electronic device 1000. Optionally, the modules of the electronic device 1000 may also adopt a combination of various connection methods described in the above embodiments.

[0368] The charging management module 140 receives power from a charger, which can be either a wireless or wired charger. In some wired charging embodiments, the charging management module 140 receives current from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 receives electromagnetic waves (current path shown as dashed lines) via the wireless charging coil of the electronic device 1000. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 1000 via the power management module 141.

[0369] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, providing power to the processor 110, internal memory 121, display screen 194, camera 193, and wireless communication module 160, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (e.g., leakage current, impedance). Optionally, the power management module 141 can be located within the processor 110, or the power management module 141 and the charging management module 140 can be located in the same device.

[0370] The wireless communication function of electronic device 1000 can be realized through devices such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.

[0371] Electronic device 1000 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0372] Display screen 194 can be used to display images or videos. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a mini light-emitting diode (Mini LED), a micro light-emitting diode (Micro LED), a micro OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 1000 may include one or N displays screens 194, where N is a positive integer greater than 1.

[0373] Electronic device 1000 can achieve shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0374] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can perform algorithmic optimization of image noise, brightness, and color. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0375] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into a standard red-green-blue (RGB), YUV, or other image signal format. In some embodiments, the electronic device 1000 may include one or N cameras 193, where N is a positive integer greater than 1.

[0376] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 1000 is selecting a frequency, the DSP is used to perform Fourier transforms on the frequency energy.

[0377] Video codecs are used to compress or decompress digital video. Electronic device 1000 may support one or more video codecs. Thus, electronic device 1000 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, and MPEG 4.

[0378] An NPU (Neural Processing Unit) is a processor that borrows from the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, to rapidly process input information and continuously learn. NPUs can enable intelligent cognitive functions in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0379] Electronic device 1000 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0380] The distance sensor 180F is used to measure distance. The electronic device 1000 can measure distance via infrared or laser. In some embodiments, such as in a shooting scenario, the electronic device 1000 can utilize the distance sensor 180F to measure distance for rapid focusing.

[0381] The ambient light sensor 180L is used to sense the brightness of ambient light. The electronic device 1000 can adaptively adjust the brightness of its display screen 194 based on the sensed ambient light level. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking photos. The ambient light sensor 180L can also work in conjunction with the proximity sensor 180G to detect whether the electronic device 1000 is in a pocket, preventing accidental touches.

[0382] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 1000 can use the collected fingerprint characteristics to achieve functions such as unlocking, accessing application locks, taking photos, and answering calls.

[0383] Touch sensor 180K, also known as a touch device, can be disposed on display screen 194. The touch sensor 180K and display screen 194 together form a touchscreen, also known as a touch display. Touch sensor 180K is used to detect touch operations applied to or near it. Touch sensor 180K can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 194. In other embodiments, touch sensor 180K may also be disposed on the surface of electronic device 1000, and in a different location from display screen 194.

[0384] Button 190 includes a power button and volume buttons. Button 190 can be a mechanical button or a touch button. Electronic device 1000 can receive button input signals and implement functions related to the button input signals.

[0385] In some embodiments, the processor 110 may start the camera; obtain the zoom ratio and ambient illuminance value in the current shooting environment; and determine the corresponding shooting process based on the zoom ratio and ambient illuminance value.

[0386] In one possible implementation, the processor 110 can select a suitable image output method for the sensor based on different dynamic ranges, different zoom levels, and different ambient illuminance.

[0387] It is understood that the image processing method of this application embodiment can be applied to Figure 10 The specific implementation steps of the electronic device shown can be referred to the description of the method embodiments above, and will not be repeated here.

[0388] As can be seen from the above, the embodiments of this application provide an image processing method and an electronic device. The image output mode of the camera is determined by the zoom ratio and dynamic range of the shooting scene. When the zoom ratio is greater than or equal to a first zoom ratio and less than a second zoom ratio, and the dynamic range is high, the sensor uses DCG mode for image output. This can improve image quality in high dynamic shooting scenes, avoid ghosting problems, and help improve the user's shooting experience. Furthermore, the ambient illumination of the shooting scene can also be considered when determining the image output mode of the camera to select a suitable output mode.

[0389] This application also provides a computer program product that, when executed by a processor, implements the methods described in any of the method embodiments of this application.

[0390] The computer program product can be stored in memory and, after processes such as preprocessing, compilation, assembly, and linking, is finally converted into an executable object file that can be executed by a processor.

[0391] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the methods described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0392] The computer-readable storage medium can be volatile memory or non-volatile memory, or it can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0393] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process and technical effects of the above-described apparatus and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments, and will not be repeated here.

[0394] In the several embodiments provided in this application, the systems, apparatuses, and methods disclosed can be implemented in other ways. For example, some features of the method embodiments described above can be ignored or not performed. The apparatus embodiments described above are merely illustrative; the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components can be combined or integrated into another system. Furthermore, the coupling between units or components can be direct coupling or indirect coupling, including electrical, mechanical, or other forms of connection.

[0395] It should be understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0396] Furthermore, the terms "system" and "network" are often used interchangeably in this paper. The term "and / or" in this paper merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this paper generally indicates that the preceding and following related objects have an "or" relationship.

[0397] In summary, the above description is merely a preferred embodiment of the technical solution of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An image processing method, characterized in that, The method is applied to an electronic device, the electronic device including an image sensor, and the method includes: Start the image sensor; Obtain the current dynamic range value, zoom ratio, and ambient illuminance; When the zoom ratio is greater than or equal to the first zoom ratio and the zoom ratio is less than the second zoom ratio, and during the process of the dynamic range value increasing from the first dynamic range value to the second dynamic range value, the mode of the image sensor outputting image data is switched from the second mode to the first mode. The first dynamic range value does not meet the first condition and the second dynamic range value meets the first condition. When the dynamic range value meets the first condition, it is a high dynamic range scenario. When the dynamic range value does not meet the first condition, it is a low dynamic range scenario. When the dynamic range value meets the first condition and the ambient illuminance is greater than or equal to the first threshold, during the process of increasing the zoom ratio from the first magnification to the third magnification, the mode of the image sensor outputting image data is switched from the first mode to the third mode. When the ambient illuminance is greater than or equal to the first threshold, it is a high-illuminance scene, and when the ambient illuminance is less than the first threshold, it is a low-illuminance scene. The first mode is a dual-gain conversion DCG mode, the first multiplier is less than the second multiplier, the third multiplier is greater than or equal to the second multiplier, and the first mode, the second mode and the third mode are different from each other.

2. The method according to claim 1, characterized in that, The method further comprises: When the zoom ratio is greater than or equal to the second zoom ratio and the dynamic range value does not meet the first condition, during the process of the ambient illuminance decreasing from the first ambient illuminance value to the second ambient illuminance value, the mode of the image sensor outputting image data switches from the third mode to the second mode, the first ambient illuminance value is greater than or equal to the first threshold, and the second ambient illuminance value is less than the first threshold.

3. The method according to claim 2, characterized in that, The method further comprises: When the zoom ratio is greater than or equal to the first zoom ratio and the zoom ratio is less than the second zoom ratio, or when the zoom ratio is greater than or equal to the second zoom ratio and the ambient illuminance is less than the first threshold, during the process of the dynamic range value increasing from the third dynamic range value to the first dynamic range value, the mode of the image sensor outputting image data is the second mode, and the third dynamic range value does not meet the first condition. When the zoom ratio is greater than or equal to the first zoom ratio and the zoom ratio is less than the second zoom ratio, or when the zoom ratio is greater than or equal to the second zoom ratio and the ambient illuminance is less than the first threshold, during the process of the dynamic range value increasing from the second dynamic range value to the fourth dynamic range value, the mode of the image data output by the image sensor is the first mode, and the fourth dynamic range value satisfies the first condition.

4. The method according to claim 3, characterized in that, The method further comprises: When the zoom ratio is greater than or equal to the second zoom ratio, during the process of the ambient illuminance increasing from the third ambient illuminance value to the first ambient illuminance value, the mode in which the image sensor outputs image data is the third mode, and the third ambient illuminance value is greater than the first threshold. When the zoom ratio is greater than or equal to the first zoom ratio and less than the second zoom ratio, and the dynamic range value does not meet the first condition, during the process of the ambient illuminance decreasing from the second ambient illuminance value to the fourth ambient illuminance value, the mode in which the image sensor outputs image data is the second mode.

5. The method according to claim 2, characterized in that, The method further comprises: When the zoom ratio is greater than or equal to the first zoom ratio and the zoom ratio is less than the second zoom ratio, and during the process of the dynamic range value increasing from the first dynamic range value to the second dynamic range value, the mode of the image sensor outputting image data is switched from the second mode to the first mode according to the condition that the first dynamic range value does not meet the first condition and the second dynamic range value meets the first condition. When the dynamic range value meets the first condition and the ambient illuminance is greater than or equal to the first threshold, during the process of increasing the zoom ratio from the first ratio to the third ratio, the mode of the image sensor outputting image data is switched from the first mode to the third mode according to the condition that the first ratio is less than the second ratio and the third ratio is greater than or equal to the second ratio. When the zoom ratio is greater than or equal to the second zoom ratio and the dynamic range value does not meet the first condition, during the process of the ambient illuminance decreasing from the first ambient illuminance value to the second ambient illuminance value, the mode of the image sensor outputting image data is switched from the third mode to the second mode based on the first ambient illuminance value being greater than the first threshold and the second ambient illuminance value being less than the first threshold.

6. The method according to claim 1, characterized in that, The second multiplier is 2, the first multiplier is 1, and the third multiplier is 2.

7. The method according to any one of claims 2 to 5, characterized in that, During the process of the dynamic range value increasing from the first dynamic range value to the second dynamic range value, the zoom ratio is 1; During the process of the ambient illuminance decreasing from the first ambient illuminance value to the second ambient illuminance value, the zoom ratio is 2.

8. The method according to any one of claims 1 to 6, characterized in that, The first mode includes a high-gain conversion path and a low-gain conversion path; The high-gain conversion path outputs long exposure frames, and the low-gain conversion path outputs short exposure frames.

9. The method according to any one of claims 1 to 6, characterized in that, The method further comprises: The ambient illuminance is the illuminance index value luxindex; the larger the luxindex value, the smaller the ambient illuminance value.

10. The method according to any one of claims 1 to 6, characterized in that, The image data output by the image sensor using the first mode is an image frame resulting from the fusion of two frames.

11. The method according to any one of claims 1 to 6, characterized in that, The second mode is the binning mode.

12. The method according to any one of claims 1 to 6, characterized in that, The image sensor outputs image data in the second mode, specifically by adding the charges sensed by adjacent pixels together and then reading out the image data in a one-pixel mode.

13. The method according to any one of claims 1 to 6, characterized in that, The image data output by the image sensor in the third mode is smaller than the resolution of the image sensor.

14. An image processing method, characterized in that, The method is applied to an electronic device, the electronic device including an image sensor, and the method includes: Start the image sensor; Obtain dynamic range and zoom ratio; When the zoom ratio is 1, if the dynamic range value meets the first condition, the image data is output in the first mode, which is the dual gain conversion DCG mode. When the zoom ratio is 1, if the dynamic range value does not meet the first condition, the image data is output in the second mode. The second mode is different from the first mode. When the dynamic range value meets the first condition, it is a high dynamic range scene, and when the dynamic range value does not meet the first condition, it is a low dynamic range scene.

15. The method according to claim 14, characterized in that, The method further comprises: When the zoom ratio is 2, image data is output in a third mode when the first illuminance index value is obtained. When the second illuminance index value is obtained and the dynamic range value satisfies the first condition, image data is output in the first mode. The third mode is different from the first mode, and the ambient illuminance represented by the first illuminance index value is higher than the ambient illuminance represented by the second illuminance index value.

16. The method according to claim 15, characterized in that, The method further includes: when the zoom ratio is 2, if a third illuminance index value is obtained and the dynamic range value does not meet the first condition, the second mode is used to output image data, wherein the ambient illuminance represented by the third illuminance index value is lower than the ambient illuminance represented by the first illuminance index value.

17. The method according to any one of claims 14-16, characterized in that, The DCG mode includes a high-gain conversion path and a low-gain conversion path; The high-gain conversion path outputs long exposure frames, and the low-gain conversion path outputs short exposure frames.

18. The method according to any one of claims 14-16, characterized in that, The image data output using the first mode is an image frame resulting from the fusion of two frames.

19. The method according to any one of claims 14-16, characterized in that, The second mode is the binning mode.

20. The method according to any one of claims 14-16, characterized in that, The second mode of outputting image data specifically involves adding the charges sensed by adjacent pixels together and then reading out the image data in a pixel-by-pixel mode.

21. The method according to any one of claims 14-16, characterized in that, The image data output using the third mode is smaller than the resolution of the image sensor.

22. An electronic device, characterized in that, The device includes an image sensor, a processor, and a memory, the processor and the memory being coupled together, the memory being used to store a computer program that, when executed by the processor, causes the electronic device to perform the method of any one of claims 1 to 21.

23. A chip, characterized in that, The chip includes a processor, which, when executing instructions, performs the method as described in any one of claims 1 to 21.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the method of any one of claims 1 to 21.

25. A computer program product, characterized in that, Includes a computer program that, when run, causes a computer to perform the method as described in any one of claims 1 to 21.

Images